2025-08-14T21:24:02.9494787Z Current runner version: '2.328.0' 2025-08-14T21:24:02.9499754Z Runner name: 'i-0aaf71856f9399359' 2025-08-14T21:24:02.9500650Z Runner group name: 'default' 2025-08-14T21:24:02.9501331Z Machine name: 'ip-10-0-36-175' 2025-08-14T21:24:02.9503580Z ##[group]GITHUB_TOKEN Permissions 2025-08-14T21:24:02.9505674Z Contents: read 2025-08-14T21:24:02.9506095Z Metadata: read 2025-08-14T21:24:02.9506507Z ##[endgroup] 2025-08-14T21:24:02.9508413Z Secret source: Actions 2025-08-14T21:24:02.9509516Z Prepare workflow directory 2025-08-14T21:24:02.9916892Z Prepare all required actions 2025-08-14T21:24:02.9949143Z Getting action download info 2025-08-14T21:24:03.2659843Z Download action repository 'pytorch/test-infra@main' (SHA:83f58f391e939c10dcb8cb6d745e4cefa3b98a84) 2025-08-14T21:24:04.5795863Z Download action repository 'pytorch/pytorch@main' (SHA:3be70dc30e893b552fc0f23ca06cd8f7949b6d08) 2025-08-14T21:24:19.5335631Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-08-14T21:24:19.8746631Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-08-14T21:24:20.1264188Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-08-14T21:24:20.3295595Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-08-14T21:24:20.6280853Z Getting action download info 2025-08-14T21:24:20.7559936Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955) 2025-08-14T21:24:20.9932989Z Getting action download info 2025-08-14T21:24:21.1098376Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-08-14T21:24:21.2917988Z Getting action download info 2025-08-14T21:24:21.4235419Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-08-14T21:24:21.5828827Z Getting action download info 2025-08-14T21:24:21.7331983Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (1fc683cf17c8c673044538d10266c00f92987be2) 2025-08-14T21:24:21.7335399Z ##[group] Inputs 2025-08-14T21:24:21.7335701Z build-environment: linux-jammy-py3.9-gcc11-build 2025-08-14T21:24:21.7338097Z test-matrix: {"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]} 2025-08-14T21:24:21.7340901Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:24:21.7341492Z sync-tag: 2025-08-14T21:24:21.7342162Z timeout-minutes: 240 2025-08-14T21:24:21.7342357Z use-gha: 2025-08-14T21:24:21.7342758Z dashboard-tag: 2025-08-14T21:24:21.7343043Z s3-bucket: gha-artifacts 2025-08-14T21:24:21.7343244Z aws-role-to-assume: 2025-08-14T21:24:21.7343639Z disable-monitor: false 2025-08-14T21:24:21.7343873Z monitor-log-interval: 5 2025-08-14T21:24:21.7344110Z monitor-data-collect-interval: 1 2025-08-14T21:24:21.7344339Z ##[endgroup] 2025-08-14T21:24:21.7344779Z Complete job name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:24:21.7818201Z A job started hook has been configured by the self-hosted runner administrator 2025-08-14T21:24:21.7897226Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-08-14T21:24:21.7904328Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:24:21.7904868Z ##[endgroup] 2025-08-14T21:24:22.7617067Z Runner Type: linux.8xlarge.amx 2025-08-14T21:24:22.7617650Z Instance Type: m7i-flex.8xlarge 2025-08-14T21:24:22.7617955Z AMI Name: unknown 2025-08-14T21:24:22.7644625Z AMI ID: ami-05ffe3c48a9991133 2025-08-14T21:24:27.1658091Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-08-14T21:24:27.1658523Z with: 2025-08-14T21:24:27.1659208Z github-secret: *** 2025-08-14T21:24:27.1659744Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-08-14T21:24:27.1660324Z activate-with-label: false 2025-08-14T21:24:27.1660616Z label: with-ssh 2025-08-14T21:24:27.1660850Z remove-existing-keys: true 2025-08-14T21:24:27.1661133Z fail-silently: true 2025-08-14T21:24:27.1661396Z env: 2025-08-14T21:24:27.1661672Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:24:27.1661916Z ##[endgroup] 2025-08-14T21:24:27.2973694Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-08-14T21:24:27.2975580Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-08-14T21:24:27.3366054Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-08-14T21:24:27.3366677Z with: 2025-08-14T21:24:27.3367012Z no-sudo: true 2025-08-14T21:24:27.3367377Z submodules: recursive 2025-08-14T21:24:27.3367790Z fetch-depth: 0 2025-08-14T21:24:27.3368109Z env: 2025-08-14T21:24:27.3368482Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:24:27.3368913Z ##[endgroup] 2025-08-14T21:24:27.3575960Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:24:27.3576993Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:24:27.3588084Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:24:27.3588530Z env: 2025-08-14T21:24:27.3588791Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:24:27.3589084Z ##[endgroup] 2025-08-14T21:24:27.3672556Z ##[group]Run # Use all available CPUs for fetching 2025-08-14T21:24:27.3672889Z # Use all available CPUs for fetching 2025-08-14T21:24:27.3673149Z cd "${GITHUB_WORKSPACE}" 2025-08-14T21:24:27.3673424Z git config --global fetch.parallel 0 2025-08-14T21:24:27.3673692Z git config --global submodule.fetchJobs 0 2025-08-14T21:24:27.3673923Z  2025-08-14T21:24:27.3674166Z # Clean workspace. The default checkout action should also do this, but 2025-08-14T21:24:27.3674474Z # do it here as well just in case 2025-08-14T21:24:27.3674696Z if [[ -d .git ]]; then 2025-08-14T21:24:27.3674899Z  if [ -z "${NO_SUDO}" ]; then 2025-08-14T21:24:27.3675117Z  sudo git clean -ffdx 2025-08-14T21:24:27.3675315Z  else 2025-08-14T21:24:27.3675485Z  git clean -ffdx 2025-08-14T21:24:27.3675674Z  fi 2025-08-14T21:24:27.3675837Z fi 2025-08-14T21:24:27.3680380Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:24:27.3680785Z env: 2025-08-14T21:24:27.3681033Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:24:27.3681242Z NO_SUDO: true 2025-08-14T21:24:27.3681403Z ##[endgroup] 2025-08-14T21:24:27.3804065Z ##[group]Run actions/checkout@v4 2025-08-14T21:24:27.3804323Z with: 2025-08-14T21:24:27.3804540Z ref: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:24:27.3804804Z fetch-depth: 0 2025-08-14T21:24:27.3804998Z submodules: recursive 2025-08-14T21:24:27.3805214Z show-progress: false 2025-08-14T21:24:27.3805434Z repository: pytorch/pytorch 2025-08-14T21:24:27.3805788Z token: *** 2025-08-14T21:24:27.3805979Z ssh-strict: true 2025-08-14T21:24:27.3806170Z ssh-user: git 2025-08-14T21:24:27.3806365Z persist-credentials: true 2025-08-14T21:24:27.3806583Z clean: true 2025-08-14T21:24:27.3806801Z sparse-checkout-cone-mode: true 2025-08-14T21:24:27.3807042Z fetch-tags: false 2025-08-14T21:24:27.3807224Z lfs: false 2025-08-14T21:24:27.3807415Z set-safe-directory: true 2025-08-14T21:24:27.3807635Z env: 2025-08-14T21:24:27.3807818Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:24:27.3808024Z ##[endgroup] 2025-08-14T21:24:27.4758784Z Syncing repository: pytorch/pytorch 2025-08-14T21:24:27.4759995Z ##[group]Getting Git version info 2025-08-14T21:24:27.4760338Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-08-14T21:24:27.4760800Z [command]/usr/bin/git version 2025-08-14T21:24:27.4978674Z git version 2.47.1 2025-08-14T21:24:27.4994466Z ##[endgroup] 2025-08-14T21:24:27.5006190Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/6af5f108-712d-4d5f-8dd1-cdfc5bf51f7c/.gitconfig' 2025-08-14T21:24:27.5029484Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/6af5f108-712d-4d5f-8dd1-cdfc5bf51f7c' before making global git config changes 2025-08-14T21:24:27.5030355Z Adding repository directory to the temporary git global config as a safe directory 2025-08-14T21:24:27.5035538Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T21:24:27.5080956Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-08-14T21:24:27.5085319Z ##[group]Initializing the repository 2025-08-14T21:24:27.5089291Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T21:24:27.5140694Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-08-14T21:24:27.5141221Z hint: is subject to change. To configure the initial branch name to use in all 2025-08-14T21:24:27.5141598Z hint: of your new repositories, which will suppress this warning, call: 2025-08-14T21:24:27.5141887Z hint: 2025-08-14T21:24:27.5142124Z hint: git config --global init.defaultBranch 2025-08-14T21:24:27.5142364Z hint: 2025-08-14T21:24:27.5142609Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-08-14T21:24:27.5143008Z hint: 'development'. The just-created branch can be renamed via this command: 2025-08-14T21:24:27.5143296Z hint: 2025-08-14T21:24:27.5143464Z hint: git branch -m 2025-08-14T21:24:27.5159044Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-08-14T21:24:27.5167716Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-08-14T21:24:27.5204646Z ##[endgroup] 2025-08-14T21:24:27.5205086Z ##[group]Disabling automatic garbage collection 2025-08-14T21:24:27.5206791Z [command]/usr/bin/git config --local gc.auto 0 2025-08-14T21:24:27.5242096Z ##[endgroup] 2025-08-14T21:24:27.5242407Z ##[group]Setting up auth 2025-08-14T21:24:27.5250288Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-08-14T21:24:27.5280599Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-08-14T21:24:27.5634968Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-08-14T21:24:27.5665616Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-08-14T21:24:27.5988982Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-08-14T21:24:27.6047539Z ##[endgroup] 2025-08-14T21:24:27.6052965Z ##[group]Fetching the repository 2025-08-14T21:24:27.6057983Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-08-14T21:25:13.7944693Z From https://github.com/pytorch/pytorch 2025-08-14T21:25:13.7945091Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-08-14T21:25:13.7945764Z * [new branch] 5addvllmbuild -> origin/5addvllmbuild 2025-08-14T21:25:13.7946353Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-08-14T21:25:13.7946926Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-08-14T21:25:13.7947523Z * [new branch] JackCaoG/dynamo_make_fx_non_core_aten_ops -> origin/JackCaoG/dynamo_make_fx_non_core_aten_ops 2025-08-14T21:25:13.7948105Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-08-14T21:25:13.7948641Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-08-14T21:25:13.7949090Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-08-14T21:25:13.7949429Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-08-14T21:25:13.7949759Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-08-14T21:25:13.7950468Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-08-14T21:25:13.7950897Z * [new branch] add-missing-args-normalization -> origin/add-missing-args-normalization 2025-08-14T21:25:13.7951493Z * [new branch] add-user-guide-structure -> origin/add-user-guide-structure 2025-08-14T21:25:13.7951996Z * [new branch] addVllmPin -> origin/addVllmPin 2025-08-14T21:25:13.7952349Z * [new branch] add_windows_testing_back -> origin/add_windows_testing_back 2025-08-14T21:25:13.7952859Z * [new branch] addbuildvllm -> origin/addbuildvllm 2025-08-14T21:25:13.7953197Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-08-14T21:25:13.7953521Z * [new branch] addsimde -> origin/addsimde 2025-08-14T21:25:13.7954117Z * [new branch] addvllpinnedfile -> origin/addvllpinnedfile 2025-08-14T21:25:13.7956249Z * [new branch] adi/acl_upgrade -> origin/adi/acl_upgrade 2025-08-14T21:25:13.7956684Z * [new branch] adi/skip_slow_tests -> origin/adi/skip_slow_tests 2025-08-14T21:25:13.7957088Z * [new branch] adi/test -> origin/adi/test 2025-08-14T21:25:13.7957808Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-08-14T21:25:13.7958516Z * [new branch] adi/test_fusions -> origin/adi/test_fusions 2025-08-14T21:25:13.7959141Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-08-14T21:25:13.7959807Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-08-14T21:25:13.7960476Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-08-14T21:25:13.7962505Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-08-14T21:25:13.7963138Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-08-14T21:25:13.7964006Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-08-14T21:25:13.7964398Z * [new branch] albanD-patch-1 -> origin/albanD-patch-1 2025-08-14T21:25:13.7965048Z * [new branch] alt-disable -> origin/alt-disable 2025-08-14T21:25:13.7969348Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-08-14T21:25:13.7969960Z * [new branch] angelayi/aoti_inductor_fx -> origin/angelayi/aoti_inductor_fx 2025-08-14T21:25:13.7970543Z * [new branch] angelayi/assert_tensor_metadata_device -> origin/angelayi/assert_tensor_metadata_device 2025-08-14T21:25:13.7970981Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-08-14T21:25:13.7971328Z * [new branch] angelayi/benchmark2 -> origin/angelayi/benchmark2 2025-08-14T21:25:13.7971716Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-08-14T21:25:13.7972092Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-08-14T21:25:13.7972593Z * [new branch] angelayi/custom_op_subgraph -> origin/angelayi/custom_op_subgraph 2025-08-14T21:25:13.7973463Z * [new branch] angelayi/customop -> origin/angelayi/customop 2025-08-14T21:25:13.7973855Z * [new branch] angelayi/del_lib -> origin/angelayi/del_lib 2025-08-14T21:25:13.7974171Z * [new branch] angelayi/docs -> origin/angelayi/docs 2025-08-14T21:25:13.7974678Z * [new branch] angelayi/docs2 -> origin/angelayi/docs2 2025-08-14T21:25:13.7975009Z * [new branch] angelayi/fix_pt2 -> origin/angelayi/fix_pt2 2025-08-14T21:25:13.7975764Z * [new branch] angelayi/logging.bak -> origin/angelayi/logging.bak 2025-08-14T21:25:13.7976296Z * [new branch] angelayi/logging2 -> origin/angelayi/logging2 2025-08-14T21:25:13.7977353Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-08-14T21:25:13.7978111Z * [new branch] angelayi/pytree -> origin/angelayi/pytree 2025-08-14T21:25:13.7978610Z * [new branch] angelayi/save_error -> origin/angelayi/save_error 2025-08-14T21:25:13.7979149Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-08-14T21:25:13.7979729Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-08-14T21:25:13.7980822Z * [new branch] angelayi/tensor_nn_module_meta -> origin/angelayi/tensor_nn_module_meta 2025-08-14T21:25:13.7981336Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-08-14T21:25:13.7982141Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-08-14T21:25:13.7982671Z * [new branch] aoti_weight_sharing -> origin/aoti_weight_sharing 2025-08-14T21:25:13.7986006Z * [new branch] arsh/symint_mm_ind_decomp -> origin/arsh/symint_mm_ind_decomp 2025-08-14T21:25:13.7986451Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-08-14T21:25:13.7986877Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-08-14T21:25:13.7987263Z * [new branch] atalman-patch-1 -> origin/atalman-patch-1 2025-08-14T21:25:13.7987585Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-08-14T21:25:13.7987937Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-08-14T21:25:13.7988268Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-08-14T21:25:13.7988773Z * [new branch] atalman-patch-7 -> origin/atalman-patch-7 2025-08-14T21:25:13.7989452Z * [new branch] atalman-patch-8 -> origin/atalman-patch-8 2025-08-14T21:25:13.7995882Z * [new branch] atalman_inductor_2.3.0 -> origin/atalman_inductor_2.3.0 2025-08-14T21:25:13.7996564Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-08-14T21:25:13.7996949Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-08-14T21:25:13.7997315Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-08-14T21:25:13.7997801Z * [new branch] autoupdate-transformers-pin-via-pr -> origin/autoupdate-transformers-pin-via-pr 2025-08-14T21:25:13.7998220Z * [new branch] backupvllm -> origin/backupvllm 2025-08-14T21:25:13.7998545Z * [new branch] base/1.5 -> origin/base/1.5 2025-08-14T21:25:13.7998917Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-08-14T21:25:13.7999320Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-08-14T21:25:13.7999676Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-08-14T21:25:13.8000161Z * [new branch] benjaminglass1/mark-large-tensor-tests-serial -> origin/benjaminglass1/mark-large-tensor-tests-serial 2025-08-14T21:25:13.8002496Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-08-14T21:25:13.8002870Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-08-14T21:25:13.8003206Z * [new branch] bf/cg-log -> origin/bf/cg-log 2025-08-14T21:25:13.8003532Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-08-14T21:25:13.8003931Z * [new branch] bf/cg-skip-1-kernel -> origin/bf/cg-skip-1-kernel 2025-08-14T21:25:13.8004258Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-08-14T21:25:13.8004663Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-08-14T21:25:13.8007164Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-08-14T21:25:13.8007734Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-08-14T21:25:13.8008131Z * [new branch] bf/default-recompile-reason -> origin/bf/default-recompile-reason 2025-08-14T21:25:13.8008527Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-08-14T21:25:13.8009153Z * [new branch] bf/improve-kernel-bench -> origin/bf/improve-kernel-bench 2025-08-14T21:25:13.8009501Z * [new branch] bf/kernel-benchmark -> origin/bf/kernel-benchmark 2025-08-14T21:25:13.8009825Z * [new branch] bf/partition-doc -> origin/bf/partition-doc 2025-08-14T21:25:13.8017675Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-08-14T21:25:13.8018276Z * [new branch] bf/partition-turn-on -> origin/bf/partition-turn-on 2025-08-14T21:25:13.8018807Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-08-14T21:25:13.8019168Z * [new branch] bf/skip-asserts -> origin/bf/skip-asserts 2025-08-14T21:25:13.8019506Z * [new branch] bf16adamw -> origin/bf16adamw 2025-08-14T21:25:13.8019849Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-08-14T21:25:13.8020245Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-08-14T21:25:13.8020821Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-08-14T21:25:13.8021180Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-08-14T21:25:13.8021532Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-08-14T21:25:13.8021886Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-08-14T21:25:13.8022234Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-08-14T21:25:13.8022595Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-08-14T21:25:13.8022950Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-08-14T21:25:13.8023318Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-08-14T21:25:13.8023681Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-08-14T21:25:13.8024052Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-08-14T21:25:13.8024426Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-08-14T21:25:13.8024811Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-08-14T21:25:13.8025176Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-08-14T21:25:13.8025552Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-08-14T21:25:13.8025923Z * [new branch] bowbao/bench_updates_stage -> origin/bowbao/bench_updates_stage 2025-08-14T21:25:13.8026291Z * [new branch] bowbao/dort_rewriter -> origin/bowbao/dort_rewriter 2025-08-14T21:25:13.8026693Z * [new branch] bowbao/wip_prs -> origin/bowbao/wip_prs 2025-08-14T21:25:13.8027058Z * [new branch] bowenbao/partial_min_max_reduce -> origin/bowenbao/partial_min_max_reduce 2025-08-14T21:25:13.8027450Z * [new branch] brister/always_wrapper_ir -> origin/brister/always_wrapper_ir 2025-08-14T21:25:13.8027803Z * [new branch] brister/flatten_contig -> origin/brister/flatten_contig 2025-08-14T21:25:13.8028150Z * [new branch] brister/test_block_ptr_same -> origin/brister/test_block_ptr_same 2025-08-14T21:25:13.8028555Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-08-14T21:25:13.8028983Z * [new branch] c57382a49 -> origin/c57382a49 2025-08-14T21:25:13.8029294Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-08-14T21:25:13.8029594Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-08-14T21:25:13.8030219Z * [new branch] camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 -> origin/camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 2025-08-14T21:25:13.8030923Z * [new branch] camyll/test_precommit_hooks_lintrunner -> origin/camyll/test_precommit_hooks_lintrunner 2025-08-14T21:25:13.8031438Z * [new branch] camyllh/cherrypick-151547-for-release28 -> origin/camyllh/cherrypick-151547-for-release28 2025-08-14T21:25:13.8031910Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-08-14T21:25:13.8032921Z * [new branch] cherry-pick-149654-by-pytorch_bot_bot_ -> origin/cherry-pick-149654-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8033533Z * [new branch] cherry-pick-151939-by-pytorch_bot_bot_ -> origin/cherry-pick-151939-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8034213Z * [new branch] cherry-pick-154174-by-pytorch_bot_bot_ -> origin/cherry-pick-154174-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8035114Z * [new branch] cherry-pick-155896-by-pytorch_bot_bot_ -> origin/cherry-pick-155896-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8036178Z * [new branch] cherry-pick-156260-by-pytorch_bot_bot_ -> origin/cherry-pick-156260-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8036782Z * [new branch] cherry-pick-156719-by-pytorch_bot_bot_ -> origin/cherry-pick-156719-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8037312Z * [new branch] cherry-pick-156876-by-pytorch_bot_bot_ -> origin/cherry-pick-156876-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8040719Z * [new branch] cherry-pick-156888-by-pytorch_bot_bot_ -> origin/cherry-pick-156888-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8041299Z * [new branch] cherry-pick-157014-by-pytorch_bot_bot_ -> origin/cherry-pick-157014-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8041823Z * [new branch] cherry-pick-157179-by-pytorch_bot_bot_ -> origin/cherry-pick-157179-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8042325Z * [new branch] cherry-pick-157453-by-pytorch_bot_bot_ -> origin/cherry-pick-157453-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8042809Z * [new branch] cherry-pick-157513-by-pytorch_bot_bot_ -> origin/cherry-pick-157513-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8043898Z * [new branch] cherry-pick-157558-by-pytorch_bot_bot_ -> origin/cherry-pick-157558-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8044468Z * [new branch] cherry-pick-157598-by-pytorch_bot_bot_ -> origin/cherry-pick-157598-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8044972Z * [new branch] cherry-pick-157600-by-pytorch_bot_bot_ -> origin/cherry-pick-157600-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8045455Z * [new branch] cherry-pick-157630-by-pytorch_bot_bot_ -> origin/cherry-pick-157630-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8046434Z * [new branch] cherry-pick-157695-by-pytorch_bot_bot_ -> origin/cherry-pick-157695-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8046925Z * [new branch] cherry-pick-157732-by-pytorch_bot_bot_ -> origin/cherry-pick-157732-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8047576Z * [new branch] cherry-pick-157733-by-pytorch_bot_bot_ -> origin/cherry-pick-157733-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8048165Z * [new branch] cherry-pick-157985-by-pytorch_bot_bot_ -> origin/cherry-pick-157985-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8048811Z * [new branch] cherry-pick-157993-by-pytorch_bot_bot_ -> origin/cherry-pick-157993-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8049438Z * [new branch] cherry-pick-158064-by-pytorch_bot_bot_ -> origin/cherry-pick-158064-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8050041Z * [new branch] cherry-pick-158152-by-pytorch_bot_bot_ -> origin/cherry-pick-158152-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8050546Z * [new branch] cherry-pick-158295-by-pytorch_bot_bot_ -> origin/cherry-pick-158295-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8051039Z * [new branch] cherry-pick-158301-by-pytorch_bot_bot_ -> origin/cherry-pick-158301-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8051562Z * [new branch] cherry-pick-158537-by-pytorch_bot_bot_ -> origin/cherry-pick-158537-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8052215Z * [new branch] cherry-pick-158572-by-pytorch_bot_bot_ -> origin/cherry-pick-158572-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8052950Z * [new branch] cherry-pick-158595 -> origin/cherry-pick-158595 2025-08-14T21:25:13.8053738Z * [new branch] cherry-pick-159181-by-pytorch_bot_bot_ -> origin/cherry-pick-159181-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8054556Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8055200Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-08-14T21:25:13.8058170Z * [new branch] cherry-pick-PR-158746 -> origin/cherry-pick-PR-158746 2025-08-14T21:25:13.8063239Z * [new branch] cherrypick-e4e2701429c17078c3c475382a8b1fa4c8a8cefc -> origin/cherrypick-e4e2701429c17078c3c475382a8b1fa4c8a8cefc 2025-08-14T21:25:13.8068716Z * [new branch] chilli/flex_vllm -> origin/chilli/flex_vllm 2025-08-14T21:25:13.8070068Z * [new branch] ckluk2-compileThread-1 -> origin/ckluk2-compileThread-1 2025-08-14T21:25:13.8070485Z * [new branch] ckluk2-compileThread-2 -> origin/ckluk2-compileThread-2 2025-08-14T21:25:13.8070888Z * [new branch] ckluk2-compileThread-64 -> origin/ckluk2-compileThread-64 2025-08-14T21:25:13.8071247Z * [new branch] ckluk2-test-1 -> origin/ckluk2-test-1 2025-08-14T21:25:13.8071578Z * [new branch] cleantest1 -> origin/cleantest1 2025-08-14T21:25:13.8071916Z * [new branch] codex-testing -> origin/codex-testing 2025-08-14T21:25:13.8072467Z * [new branch] codex/create-test-for-tensor-memory-leak-in-cudagraph -> origin/codex/create-test-for-tensor-memory-leak-in-cudagraph 2025-08-14T21:25:13.8073070Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-08-14T21:25:13.8073515Z * [new branch] codex/fix-issue-160415-in-pytorch -> origin/codex/fix-issue-160415-in-pytorch 2025-08-14T21:25:13.8074048Z * [new branch] codex/fix-noqengine-quantized-engine-support -> origin/codex/fix-noqengine-quantized-engine-support 2025-08-14T21:25:13.8074606Z * [new branch] codex/fix-pin_memory-error-handling -> origin/codex/fix-pin_memory-error-handling 2025-08-14T21:25:13.8075098Z * [new branch] codex/propose-fix-for-issue-160332 -> origin/codex/propose-fix-for-issue-160332 2025-08-14T21:25:13.8075830Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-08-14T21:25:13.8076570Z * [new branch] codex/verify-torch-output-and-log-results -> origin/codex/verify-torch-output-and-log-results 2025-08-14T21:25:13.8077115Z * [new branch] compile_fsdp2_disable_stream_and_event -> origin/compile_fsdp2_disable_stream_and_event 2025-08-14T21:25:13.8077774Z * [new branch] comply-with-setuptools -> origin/comply-with-setuptools 2025-08-14T21:25:13.8078143Z * [new branch] context_test -> origin/context_test 2025-08-14T21:25:13.8078508Z * [new branch] copilot/fix-157446 -> origin/copilot/fix-157446 2025-08-14T21:25:13.8078854Z * [new branch] copilot/fix-159257 -> origin/copilot/fix-159257 2025-08-14T21:25:13.8079178Z * [new branch] copy_graph -> origin/copy_graph 2025-08-14T21:25:13.8079508Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-08-14T21:25:13.8079846Z * [new branch] csl/3_proc_sm -> origin/csl/3_proc_sm 2025-08-14T21:25:13.8080212Z * [new branch] csl/add_file_merge_conflict_csv -> origin/csl/add_file_merge_conflict_csv 2025-08-14T21:25:13.8080603Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-08-14T21:25:13.8080961Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-08-14T21:25:13.8081336Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-08-14T21:25:13.8081719Z * [new branch] csl/disable_flaky_cpp_test -> origin/csl/disable_flaky_cpp_test 2025-08-14T21:25:13.8082092Z * [new branch] csl/disable_periodic_test -> origin/csl/disable_periodic_test 2025-08-14T21:25:13.8082465Z * [new branch] csl/executorch_docker_fail -> origin/csl/executorch_docker_fail 2025-08-14T21:25:13.8082885Z * [new branch] csl/fix_check_alerts -> origin/csl/fix_check_alerts 2025-08-14T21:25:13.8083241Z * [new branch] csl/katex -> origin/csl/katex 2025-08-14T21:25:13.8083557Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-08-14T21:25:13.8083991Z * [new branch] csl/lintrunner_changed_files_removed -> origin/csl/lintrunner_changed_files_removed 2025-08-14T21:25:13.8084478Z * [new branch] csl/lintrunner_changed_files_removed_test -> origin/csl/lintrunner_changed_files_removed_test 2025-08-14T21:25:13.8084901Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-08-14T21:25:13.8085248Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-08-14T21:25:13.8085942Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-08-14T21:25:13.8086549Z * [new branch] csl/no_keep_goin_rocm -> origin/csl/no_keep_goin_rocm 2025-08-14T21:25:13.8087110Z * [new branch] csl/not_600_timeout -> origin/csl/not_600_timeout 2025-08-14T21:25:13.8087804Z * [new branch] csl/remove_unused_docker_images -> origin/csl/remove_unused_docker_images 2025-08-14T21:25:13.8089136Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-08-14T21:25:13.8089525Z * [new branch] csl/rocm_upload_artifacts_while_running -> origin/csl/rocm_upload_artifacts_while_running 2025-08-14T21:25:13.8090255Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-08-14T21:25:13.8090645Z * [new branch] csl/td_dynamo -> origin/csl/td_dynamo 2025-08-14T21:25:13.8091086Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-08-14T21:25:13.8091864Z * [new branch] csl/unused_docker -> origin/csl/unused_docker 2025-08-14T21:25:13.8092316Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-08-14T21:25:13.8093094Z * [new branch] cublasltrelax2 -> origin/cublasltrelax2 2025-08-14T21:25:13.8093741Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-08-14T21:25:13.8094364Z * [new branch] cudnnsdparefactor -> origin/cudnnsdparefactor 2025-08-14T21:25:13.8099569Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-08-14T21:25:13.8099926Z * [new branch] czhuge_muon_dev -> origin/czhuge_muon_dev 2025-08-14T21:25:13.8100252Z * [new branch] d4l3k/delete_hook -> origin/d4l3k/delete_hook 2025-08-14T21:25:13.8100586Z * [new branch] d4l3k/dist_queue -> origin/d4l3k/dist_queue 2025-08-14T21:25:13.8100916Z * [new branch] d4l3k/wait_stream -> origin/d4l3k/wait_stream 2025-08-14T21:25:13.8101287Z * [new branch] dcp-safetensor-test-fix -> origin/dcp-safetensor-test-fix 2025-08-14T21:25:13.8101631Z * [new branch] dcp_zoc -> origin/dcp_zoc 2025-08-14T21:25:13.8101958Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-08-14T21:25:13.8102803Z * [new branch] dependabot/pip/dot-ci/docker/protobuf-5.29.5 -> origin/dependabot/pip/dot-ci/docker/protobuf-5.29.5 2025-08-14T21:25:13.8103314Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-08-14T21:25:13.8103745Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-08-14T21:25:13.8105390Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-08-14T21:25:13.8106035Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-08-14T21:25:13.8106993Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-08-14T21:25:13.8107782Z * [new branch] dev/joona/cat_remove_graph -> origin/dev/joona/cat_remove_graph 2025-08-14T21:25:13.8108464Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-08-14T21:25:13.8109501Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-08-14T21:25:13.8110854Z * [new branch] dev/joona/maxpool2dwithindices_errmsg -> origin/dev/joona/maxpool2dwithindices_errmsg 2025-08-14T21:25:13.8111617Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-08-14T21:25:13.8112968Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-08-14T21:25:13.8113647Z * [new branch] dev/joona/synchronize_benchmark -> origin/dev/joona/synchronize_benchmark 2025-08-14T21:25:13.8114479Z * [new branch] dev/joona/topk_newapi -> origin/dev/joona/topk_newapi 2025-08-14T21:25:13.8115413Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-08-14T21:25:13.8116386Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-08-14T21:25:13.8117694Z * [new branch] disable -> origin/disable 2025-08-14T21:25:13.8118345Z * [new branch] divyanshk-log-api-usage-datapipes-1 -> origin/divyanshk-log-api-usage-datapipes-1 2025-08-14T21:25:13.8119017Z * [new branch] e2e-baseline -> origin/e2e-baseline 2025-08-14T21:25:13.8120994Z * [new branch] embg/test_inductor_ci_128B -> origin/embg/test_inductor_ci_128B 2025-08-14T21:25:13.8121425Z * [new branch] embg/test_inductor_ci_base -> origin/embg/test_inductor_ci_base 2025-08-14T21:25:13.8122329Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-08-14T21:25:13.8122927Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-08-14T21:25:13.8123497Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-08-14T21:25:13.8124029Z * [new branch] enable-b200-benchmark -> origin/enable-b200-benchmark 2025-08-14T21:25:13.8124577Z * [new branch] eqy-patch-1 -> origin/eqy-patch-1 2025-08-14T21:25:13.8125216Z * [new branch] eqy-patch-10 -> origin/eqy-patch-10 2025-08-14T21:25:13.8128807Z * [new branch] eqy-patch-2 -> origin/eqy-patch-2 2025-08-14T21:25:13.8129531Z * [new branch] example-convert-torch.nn -> origin/example-convert-torch.nn 2025-08-14T21:25:13.8130138Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-08-14T21:25:13.8130775Z * [new branch] exclamaforte/bump-transformer-version -> origin/exclamaforte/bump-transformer-version 2025-08-14T21:25:13.8131536Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-08-14T21:25:13.8132092Z * [new branch] exclamaforte/debug-autotuner-profile -> origin/exclamaforte/debug-autotuner-profile 2025-08-14T21:25:13.8132555Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-08-14T21:25:13.8133059Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-08-14T21:25:13.8133628Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-08-14T21:25:13.8134180Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-08-14T21:25:13.8134723Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-08-14T21:25:13.8135615Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-08-14T21:25:13.8136178Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-08-14T21:25:13.8136801Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-08-14T21:25:13.8137229Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-08-14T21:25:13.8137723Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-08-14T21:25:13.8138239Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-08-14T21:25:13.8138785Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-08-14T21:25:13.8139372Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-08-14T21:25:13.8140009Z * [new branch] exclamaforte/memory-counter -> origin/exclamaforte/memory-counter 2025-08-14T21:25:13.8140547Z * [new branch] exclamaforte/scheduler-refactor -> origin/exclamaforte/scheduler-refactor 2025-08-14T21:25:13.8141110Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-08-14T21:25:13.8141571Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-08-14T21:25:13.8142121Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-08-14T21:25:13.8142773Z * [new branch] exclamaforte/update-pandas-numpy-ci -> origin/exclamaforte/update-pandas-numpy-ci 2025-08-14T21:25:13.8144273Z * [new branch] exclamforte/gemm-model-final -> origin/exclamforte/gemm-model-final 2025-08-14T21:25:13.8145121Z * [new branch] exec -> origin/exec 2025-08-14T21:25:13.8145715Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-08-14T21:25:13.8146082Z * [new branch] export-D58091437 -> origin/export-D58091437 2025-08-14T21:25:13.8146871Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-08-14T21:25:13.8147568Z * [new branch] export-D68846308 -> origin/export-D68846308 2025-08-14T21:25:13.8148227Z * [new branch] export-D70112642 -> origin/export-D70112642 2025-08-14T21:25:13.8149946Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-08-14T21:25:13.8150398Z * [new branch] export-D72483950 -> origin/export-D72483950 2025-08-14T21:25:13.8150714Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-08-14T21:25:13.8151026Z * [new branch] export-D73287751 -> origin/export-D73287751 2025-08-14T21:25:13.8151675Z * [new branch] export-D75183591 -> origin/export-D75183591 2025-08-14T21:25:13.8152419Z * [new branch] export-D75605373 -> origin/export-D75605373 2025-08-14T21:25:13.8153103Z * [new branch] export-D75617432 -> origin/export-D75617432 2025-08-14T21:25:13.8153875Z * [new branch] export-D75659965 -> origin/export-D75659965 2025-08-14T21:25:13.8154566Z * [new branch] export-D76080931 -> origin/export-D76080931 2025-08-14T21:25:13.8155222Z * [new branch] export-D76463347 -> origin/export-D76463347 2025-08-14T21:25:13.8155890Z * [new branch] export-D76797250 -> origin/export-D76797250 2025-08-14T21:25:13.8156604Z * [new branch] export-D76885271 -> origin/export-D76885271 2025-08-14T21:25:13.8157396Z * [new branch] export-D76885620 -> origin/export-D76885620 2025-08-14T21:25:13.8158162Z * [new branch] export-D76936623 -> origin/export-D76936623 2025-08-14T21:25:13.8158687Z * [new branch] export-D76958268 -> origin/export-D76958268 2025-08-14T21:25:13.8159428Z * [new branch] export-D78047846 -> origin/export-D78047846 2025-08-14T21:25:13.8160668Z * [new branch] export-D78308105 -> origin/export-D78308105 2025-08-14T21:25:13.8160998Z * [new branch] export-D78363609 -> origin/export-D78363609 2025-08-14T21:25:13.8161735Z * [new branch] export-D78375400 -> origin/export-D78375400 2025-08-14T21:25:13.8162428Z * [new branch] export-D78431075 -> origin/export-D78431075 2025-08-14T21:25:13.8163064Z * [new branch] export-D78431305 -> origin/export-D78431305 2025-08-14T21:25:13.8164529Z * [new branch] export-D78458745 -> origin/export-D78458745 2025-08-14T21:25:13.8164861Z * [new branch] export-D78524147 -> origin/export-D78524147 2025-08-14T21:25:13.8165423Z * [new branch] export-D78580107 -> origin/export-D78580107 2025-08-14T21:25:13.8166083Z * [new branch] export-D78588406 -> origin/export-D78588406 2025-08-14T21:25:13.8166879Z * [new branch] export-D78691422 -> origin/export-D78691422 2025-08-14T21:25:13.8167550Z * [new branch] export-D78758466 -> origin/export-D78758466 2025-08-14T21:25:13.8168155Z * [new branch] export-D78822171 -> origin/export-D78822171 2025-08-14T21:25:13.8168851Z * [new branch] export-D78822351 -> origin/export-D78822351 2025-08-14T21:25:13.8169455Z * [new branch] export-D78822507 -> origin/export-D78822507 2025-08-14T21:25:13.8170109Z * [new branch] export-D78826994 -> origin/export-D78826994 2025-08-14T21:25:13.8170708Z * [new branch] export-D78894142 -> origin/export-D78894142 2025-08-14T21:25:13.8171393Z * [new branch] export-D78894324 -> origin/export-D78894324 2025-08-14T21:25:13.8172038Z * [new branch] export-D78907485 -> origin/export-D78907485 2025-08-14T21:25:13.8172670Z * [new branch] export-D78929245 -> origin/export-D78929245 2025-08-14T21:25:13.8176954Z * [new branch] export-D78934925 -> origin/export-D78934925 2025-08-14T21:25:13.8177288Z * [new branch] export-D78953203 -> origin/export-D78953203 2025-08-14T21:25:13.8177724Z * [new branch] export-D78953229 -> origin/export-D78953229 2025-08-14T21:25:13.8178056Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-08-14T21:25:13.8178371Z * [new branch] export-D78957389 -> origin/export-D78957389 2025-08-14T21:25:13.8178674Z * [new branch] export-D78957974 -> origin/export-D78957974 2025-08-14T21:25:13.8178970Z * [new branch] export-D78979812 -> origin/export-D78979812 2025-08-14T21:25:13.8179268Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-08-14T21:25:13.8179692Z * [new branch] export-D79026433 -> origin/export-D79026433 2025-08-14T21:25:13.8180123Z * [new branch] export-D79230339 -> origin/export-D79230339 2025-08-14T21:25:13.8180556Z * [new branch] export-D79319835 -> origin/export-D79319835 2025-08-14T21:25:13.8180971Z * [new branch] export-D79328456 -> origin/export-D79328456 2025-08-14T21:25:13.8181543Z * [new branch] export-D79534608 -> origin/export-D79534608 2025-08-14T21:25:13.8182624Z * [new branch] export-D79647167 -> origin/export-D79647167 2025-08-14T21:25:13.8183454Z * [new branch] export-D79751098 -> origin/export-D79751098 2025-08-14T21:25:13.8185211Z * [new branch] export-D79785974 -> origin/export-D79785974 2025-08-14T21:25:13.8186062Z * [new branch] export-D80025417 -> origin/export-D80025417 2025-08-14T21:25:13.8186476Z * [new branch] export-D80120333 -> origin/export-D80120333 2025-08-14T21:25:13.8186886Z * [new branch] export-D80214882 -> origin/export-D80214882 2025-08-14T21:25:13.8188039Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-08-14T21:25:13.8189385Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-08-14T21:25:13.8189753Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-08-14T21:25:13.8190470Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-08-14T21:25:13.8192389Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-08-14T21:25:13.8192789Z * [new branch] fca -> origin/fca 2025-08-14T21:25:13.8193330Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-08-14T21:25:13.8193696Z * [new branch] fca5 -> origin/fca5 2025-08-14T21:25:13.8195140Z * [new branch] feature/function-numa-binding -> origin/feature/function-numa-binding 2025-08-14T21:25:13.8196923Z * [new branch] fengyuan/external-proj -> origin/fengyuan/external-proj 2025-08-14T21:25:13.8197441Z * [new branch] fengyuan/out-of-tree-xpu-ops-improve-test -> origin/fengyuan/out-of-tree-xpu-ops-improve-test 2025-08-14T21:25:13.8198235Z * [new branch] fengyuan/out-of-tree-xpu-ops-remove-dtype -> origin/fengyuan/out-of-tree-xpu-ops-remove-dtype 2025-08-14T21:25:13.8198701Z * [new branch] fengyuan/test-xpu -> origin/fengyuan/test-xpu 2025-08-14T21:25:13.8199071Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-08-14T21:25:13.8199568Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-08-14T21:25:13.8200876Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-08-14T21:25:13.8201311Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-08-14T21:25:13.8201954Z * [new branch] findhao/fix-indirect-access -> origin/findhao/fix-indirect-access 2025-08-14T21:25:13.8202478Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-08-14T21:25:13.8203162Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-08-14T21:25:13.8203686Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-08-14T21:25:13.8204375Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-08-14T21:25:13.8208857Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-08-14T21:25:13.8209218Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-08-14T21:25:13.8209524Z * [new branch] fix -> origin/fix 2025-08-14T21:25:13.8209853Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-08-14T21:25:13.8210197Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-08-14T21:25:13.8210520Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-08-14T21:25:13.8210854Z * [new branch] fix-distributed-warning -> origin/fix-distributed-warning 2025-08-14T21:25:13.8211236Z * [new branch] fix-inductor-periodic-0528 -> origin/fix-inductor-periodic-0528 2025-08-14T21:25:13.8211737Z * [new branch] fix-rlease-feature-template -> origin/fix-rlease-feature-template 2025-08-14T21:25:13.8212074Z * [new branch] fix_153389 -> origin/fix_153389 2025-08-14T21:25:13.8212370Z * [new branch] fixes-triage -> origin/fixes-triage 2025-08-14T21:25:13.8212695Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-08-14T21:25:13.8213428Z * [new branch] flex-flash -> origin/flex-flash 2025-08-14T21:25:13.8214064Z * [new branch] flex-lowering -> origin/flex-lowering 2025-08-14T21:25:13.8214753Z * [new branch] flex-warning -> origin/flex-warning 2025-08-14T21:25:13.8215466Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-08-14T21:25:13.8216091Z * [new branch] flex_flash -> origin/flex_flash 2025-08-14T21:25:13.8217501Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-08-14T21:25:13.8217920Z * [new branch] fmassa/try_fix_ac_tag_propagation -> origin/fmassa/try_fix_ac_tag_propagation 2025-08-14T21:25:13.8218560Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-08-14T21:25:13.8219219Z * [new branch] fsdpv2_3d -> origin/fsdpv2_3d 2025-08-14T21:25:13.8220215Z * [new branch] fsdpv2_3d_m1 -> origin/fsdpv2_3d_m1 2025-08-14T21:25:13.8221404Z * [new branch] fx_cpp -> origin/fx_cpp 2025-08-14T21:25:13.8222107Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-08-14T21:25:13.8224187Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-08-14T21:25:13.8224841Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-08-14T21:25:13.8226015Z * [new branch] gh/CaoE/2/base -> origin/gh/CaoE/2/base 2025-08-14T21:25:13.8226660Z * [new branch] gh/CaoE/2/head -> origin/gh/CaoE/2/head 2025-08-14T21:25:13.8227075Z * [new branch] gh/CaoE/2/orig -> origin/gh/CaoE/2/orig 2025-08-14T21:25:13.8228927Z * [new branch] gh/ColinPeppler/72/base -> origin/gh/ColinPeppler/72/base 2025-08-14T21:25:13.8229344Z * [new branch] gh/ColinPeppler/72/head -> origin/gh/ColinPeppler/72/head 2025-08-14T21:25:13.8229987Z * [new branch] gh/ColinPeppler/72/orig -> origin/gh/ColinPeppler/72/orig 2025-08-14T21:25:13.8231492Z * [new branch] gh/ColinPeppler/77/base -> origin/gh/ColinPeppler/77/base 2025-08-14T21:25:13.8232478Z * [new branch] gh/ColinPeppler/77/head -> origin/gh/ColinPeppler/77/head 2025-08-14T21:25:13.8232949Z * [new branch] gh/ColinPeppler/77/orig -> origin/gh/ColinPeppler/77/orig 2025-08-14T21:25:13.8233645Z * [new branch] gh/ColinPeppler/78/base -> origin/gh/ColinPeppler/78/base 2025-08-14T21:25:13.8234214Z * [new branch] gh/ColinPeppler/78/head -> origin/gh/ColinPeppler/78/head 2025-08-14T21:25:13.8234987Z * [new branch] gh/ColinPeppler/78/orig -> origin/gh/ColinPeppler/78/orig 2025-08-14T21:25:13.8236628Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-08-14T21:25:13.8237019Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-08-14T21:25:13.8240317Z * [new branch] gh/EikanWang/80/base -> origin/gh/EikanWang/80/base 2025-08-14T21:25:13.8240905Z * [new branch] gh/EikanWang/80/head -> origin/gh/EikanWang/80/head 2025-08-14T21:25:13.8241576Z * [new branch] gh/EikanWang/80/orig -> origin/gh/EikanWang/80/orig 2025-08-14T21:25:13.8242133Z * [new branch] gh/EikanWang/81/base -> origin/gh/EikanWang/81/base 2025-08-14T21:25:13.8242467Z * [new branch] gh/EikanWang/81/head -> origin/gh/EikanWang/81/head 2025-08-14T21:25:13.8242805Z * [new branch] gh/EikanWang/81/orig -> origin/gh/EikanWang/81/orig 2025-08-14T21:25:13.8243198Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-08-14T21:25:13.8243768Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-08-14T21:25:13.8247003Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-08-14T21:25:13.8247549Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-08-14T21:25:13.8248008Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-08-14T21:25:13.8249001Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-08-14T21:25:13.8249402Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-08-14T21:25:13.8249737Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-08-14T21:25:13.8250221Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-08-14T21:25:13.8250651Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-08-14T21:25:13.8251234Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-08-14T21:25:13.8253364Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-08-14T21:25:13.8253773Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-08-14T21:25:13.8254111Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-08-14T21:25:13.8254649Z * [new branch] gh/H-Huang/183/base -> origin/gh/H-Huang/183/base 2025-08-14T21:25:13.8256160Z * [new branch] gh/H-Huang/183/head -> origin/gh/H-Huang/183/head 2025-08-14T21:25:13.8256657Z * [new branch] gh/H-Huang/183/orig -> origin/gh/H-Huang/183/orig 2025-08-14T21:25:13.8257039Z * [new branch] gh/H-Huang/187/base -> origin/gh/H-Huang/187/base 2025-08-14T21:25:13.8257408Z * [new branch] gh/H-Huang/187/head -> origin/gh/H-Huang/187/head 2025-08-14T21:25:13.8258027Z * [new branch] gh/H-Huang/187/orig -> origin/gh/H-Huang/187/orig 2025-08-14T21:25:13.8259362Z * [new branch] gh/H-Huang/192/base -> origin/gh/H-Huang/192/base 2025-08-14T21:25:13.8259695Z * [new branch] gh/H-Huang/192/head -> origin/gh/H-Huang/192/head 2025-08-14T21:25:13.8260437Z * [new branch] gh/H-Huang/192/orig -> origin/gh/H-Huang/192/orig 2025-08-14T21:25:13.8261739Z * [new branch] gh/H-Huang/195/base -> origin/gh/H-Huang/195/base 2025-08-14T21:25:13.8262140Z * [new branch] gh/H-Huang/195/head -> origin/gh/H-Huang/195/head 2025-08-14T21:25:13.8262762Z * [new branch] gh/H-Huang/195/orig -> origin/gh/H-Huang/195/orig 2025-08-14T21:25:13.8264232Z * [new branch] gh/H-Huang/196/base -> origin/gh/H-Huang/196/base 2025-08-14T21:25:13.8265078Z * [new branch] gh/H-Huang/196/head -> origin/gh/H-Huang/196/head 2025-08-14T21:25:13.8265527Z * [new branch] gh/H-Huang/196/orig -> origin/gh/H-Huang/196/orig 2025-08-14T21:25:13.8266356Z * [new branch] gh/H-Huang/197/base -> origin/gh/H-Huang/197/base 2025-08-14T21:25:13.8271260Z * [new branch] gh/H-Huang/197/head -> origin/gh/H-Huang/197/head 2025-08-14T21:25:13.8271720Z * [new branch] gh/H-Huang/197/orig -> origin/gh/H-Huang/197/orig 2025-08-14T21:25:13.8272254Z * [new branch] gh/H-Huang/198/base -> origin/gh/H-Huang/198/base 2025-08-14T21:25:13.8272599Z * [new branch] gh/H-Huang/198/head -> origin/gh/H-Huang/198/head 2025-08-14T21:25:13.8272933Z * [new branch] gh/H-Huang/198/orig -> origin/gh/H-Huang/198/orig 2025-08-14T21:25:13.8273274Z * [new branch] gh/H-Huang/199/base -> origin/gh/H-Huang/199/base 2025-08-14T21:25:13.8273811Z * [new branch] gh/H-Huang/199/head -> origin/gh/H-Huang/199/head 2025-08-14T21:25:13.8274309Z * [new branch] gh/H-Huang/199/orig -> origin/gh/H-Huang/199/orig 2025-08-14T21:25:13.8275257Z * [new branch] gh/H-Huang/200/base -> origin/gh/H-Huang/200/base 2025-08-14T21:25:13.8275667Z * [new branch] gh/H-Huang/200/head -> origin/gh/H-Huang/200/head 2025-08-14T21:25:13.8276236Z * [new branch] gh/H-Huang/200/orig -> origin/gh/H-Huang/200/orig 2025-08-14T21:25:13.8276620Z * [new branch] gh/H-Huang/201/base -> origin/gh/H-Huang/201/base 2025-08-14T21:25:13.8277179Z * [new branch] gh/H-Huang/201/head -> origin/gh/H-Huang/201/head 2025-08-14T21:25:13.8277665Z * [new branch] gh/H-Huang/201/orig -> origin/gh/H-Huang/201/orig 2025-08-14T21:25:13.8278033Z * [new branch] gh/H-Huang/202/base -> origin/gh/H-Huang/202/base 2025-08-14T21:25:13.8278691Z * [new branch] gh/H-Huang/202/head -> origin/gh/H-Huang/202/head 2025-08-14T21:25:13.8283182Z * [new branch] gh/H-Huang/202/orig -> origin/gh/H-Huang/202/orig 2025-08-14T21:25:13.8283775Z * [new branch] gh/H-Huang/203/base -> origin/gh/H-Huang/203/base 2025-08-14T21:25:13.8284263Z * [new branch] gh/H-Huang/203/head -> origin/gh/H-Huang/203/head 2025-08-14T21:25:13.8285012Z * [new branch] gh/H-Huang/203/orig -> origin/gh/H-Huang/203/orig 2025-08-14T21:25:13.8285385Z * [new branch] gh/H-Huang/204/base -> origin/gh/H-Huang/204/base 2025-08-14T21:25:13.8285792Z * [new branch] gh/H-Huang/204/head -> origin/gh/H-Huang/204/head 2025-08-14T21:25:13.8286131Z * [new branch] gh/H-Huang/204/orig -> origin/gh/H-Huang/204/orig 2025-08-14T21:25:13.8286470Z * [new branch] gh/H-Huang/205/base -> origin/gh/H-Huang/205/base 2025-08-14T21:25:13.8286803Z * [new branch] gh/H-Huang/205/head -> origin/gh/H-Huang/205/head 2025-08-14T21:25:13.8287147Z * [new branch] gh/H-Huang/205/orig -> origin/gh/H-Huang/205/orig 2025-08-14T21:25:13.8287492Z * [new branch] gh/H-Huang/206/base -> origin/gh/H-Huang/206/base 2025-08-14T21:25:13.8287987Z * [new branch] gh/H-Huang/206/head -> origin/gh/H-Huang/206/head 2025-08-14T21:25:13.8288426Z * [new branch] gh/H-Huang/206/orig -> origin/gh/H-Huang/206/orig 2025-08-14T21:25:13.8289149Z * [new branch] gh/H-Huang/207/base -> origin/gh/H-Huang/207/base 2025-08-14T21:25:13.8289781Z * [new branch] gh/H-Huang/207/head -> origin/gh/H-Huang/207/head 2025-08-14T21:25:13.8292543Z * [new branch] gh/H-Huang/207/orig -> origin/gh/H-Huang/207/orig 2025-08-14T21:25:13.8292983Z * [new branch] gh/H-Huang/208/base -> origin/gh/H-Huang/208/base 2025-08-14T21:25:13.8293309Z * [new branch] gh/H-Huang/208/head -> origin/gh/H-Huang/208/head 2025-08-14T21:25:13.8293623Z * [new branch] gh/H-Huang/208/orig -> origin/gh/H-Huang/208/orig 2025-08-14T21:25:13.8293940Z * [new branch] gh/H-Huang/209/base -> origin/gh/H-Huang/209/base 2025-08-14T21:25:13.8294252Z * [new branch] gh/H-Huang/209/head -> origin/gh/H-Huang/209/head 2025-08-14T21:25:13.8294977Z * [new branch] gh/H-Huang/209/orig -> origin/gh/H-Huang/209/orig 2025-08-14T21:25:13.8296516Z * [new branch] gh/IvanKobzarev/107/base -> origin/gh/IvanKobzarev/107/base 2025-08-14T21:25:13.8297174Z * [new branch] gh/IvanKobzarev/107/head -> origin/gh/IvanKobzarev/107/head 2025-08-14T21:25:13.8297847Z * [new branch] gh/IvanKobzarev/107/orig -> origin/gh/IvanKobzarev/107/orig 2025-08-14T21:25:13.8299402Z * [new branch] gh/IvanKobzarev/110/base -> origin/gh/IvanKobzarev/110/base 2025-08-14T21:25:13.8299773Z * [new branch] gh/IvanKobzarev/110/head -> origin/gh/IvanKobzarev/110/head 2025-08-14T21:25:13.8300139Z * [new branch] gh/IvanKobzarev/110/orig -> origin/gh/IvanKobzarev/110/orig 2025-08-14T21:25:13.8301668Z * [new branch] gh/IvanKobzarev/111/base -> origin/gh/IvanKobzarev/111/base 2025-08-14T21:25:13.8302035Z * [new branch] gh/IvanKobzarev/111/head -> origin/gh/IvanKobzarev/111/head 2025-08-14T21:25:13.8302402Z * [new branch] gh/IvanKobzarev/111/orig -> origin/gh/IvanKobzarev/111/orig 2025-08-14T21:25:13.8303436Z * [new branch] gh/IvanKobzarev/112/base -> origin/gh/IvanKobzarev/112/base 2025-08-14T21:25:13.8304006Z * [new branch] gh/IvanKobzarev/112/head -> origin/gh/IvanKobzarev/112/head 2025-08-14T21:25:13.8304700Z * [new branch] gh/IvanKobzarev/112/orig -> origin/gh/IvanKobzarev/112/orig 2025-08-14T21:25:13.8308611Z * [new branch] gh/IvanKobzarev/115/base -> origin/gh/IvanKobzarev/115/base 2025-08-14T21:25:13.8309214Z * [new branch] gh/IvanKobzarev/115/head -> origin/gh/IvanKobzarev/115/head 2025-08-14T21:25:13.8309596Z * [new branch] gh/IvanKobzarev/115/orig -> origin/gh/IvanKobzarev/115/orig 2025-08-14T21:25:13.8309972Z * [new branch] gh/IvanKobzarev/116/base -> origin/gh/IvanKobzarev/116/base 2025-08-14T21:25:13.8310543Z * [new branch] gh/IvanKobzarev/116/head -> origin/gh/IvanKobzarev/116/head 2025-08-14T21:25:13.8310932Z * [new branch] gh/IvanKobzarev/116/orig -> origin/gh/IvanKobzarev/116/orig 2025-08-14T21:25:13.8311290Z * [new branch] gh/IvanKobzarev/118/base -> origin/gh/IvanKobzarev/118/base 2025-08-14T21:25:13.8311878Z * [new branch] gh/IvanKobzarev/118/head -> origin/gh/IvanKobzarev/118/head 2025-08-14T21:25:13.8312280Z * [new branch] gh/IvanKobzarev/118/orig -> origin/gh/IvanKobzarev/118/orig 2025-08-14T21:25:13.8313551Z * [new branch] gh/IvanKobzarev/124/base -> origin/gh/IvanKobzarev/124/base 2025-08-14T21:25:13.8313969Z * [new branch] gh/IvanKobzarev/124/head -> origin/gh/IvanKobzarev/124/head 2025-08-14T21:25:13.8314755Z * [new branch] gh/IvanKobzarev/124/orig -> origin/gh/IvanKobzarev/124/orig 2025-08-14T21:25:13.8316139Z * [new branch] gh/IvanKobzarev/126/base -> origin/gh/IvanKobzarev/126/base 2025-08-14T21:25:13.8316574Z * [new branch] gh/IvanKobzarev/126/head -> origin/gh/IvanKobzarev/126/head 2025-08-14T21:25:13.8317320Z * [new branch] gh/IvanKobzarev/126/orig -> origin/gh/IvanKobzarev/126/orig 2025-08-14T21:25:13.8318604Z * [new branch] gh/IvanKobzarev/127/base -> origin/gh/IvanKobzarev/127/base 2025-08-14T21:25:13.8319101Z * [new branch] gh/IvanKobzarev/127/head -> origin/gh/IvanKobzarev/127/head 2025-08-14T21:25:13.8319771Z * [new branch] gh/IvanKobzarev/127/orig -> origin/gh/IvanKobzarev/127/orig 2025-08-14T21:25:13.8322751Z * [new branch] gh/IvanKobzarev/128/base -> origin/gh/IvanKobzarev/128/base 2025-08-14T21:25:13.8323134Z * [new branch] gh/IvanKobzarev/128/head -> origin/gh/IvanKobzarev/128/head 2025-08-14T21:25:13.8323545Z * [new branch] gh/IvanKobzarev/128/orig -> origin/gh/IvanKobzarev/128/orig 2025-08-14T21:25:13.8324126Z * [new branch] gh/IvanKobzarev/129/base -> origin/gh/IvanKobzarev/129/base 2025-08-14T21:25:13.8328604Z * [new branch] gh/IvanKobzarev/129/head -> origin/gh/IvanKobzarev/129/head 2025-08-14T21:25:13.8333093Z * [new branch] gh/IvanKobzarev/129/orig -> origin/gh/IvanKobzarev/129/orig 2025-08-14T21:25:13.8333535Z * [new branch] gh/IvanKobzarev/130/base -> origin/gh/IvanKobzarev/130/base 2025-08-14T21:25:13.8333907Z * [new branch] gh/IvanKobzarev/130/head -> origin/gh/IvanKobzarev/130/head 2025-08-14T21:25:13.8334276Z * [new branch] gh/IvanKobzarev/130/orig -> origin/gh/IvanKobzarev/130/orig 2025-08-14T21:25:13.8334635Z * [new branch] gh/IvanKobzarev/131/base -> origin/gh/IvanKobzarev/131/base 2025-08-14T21:25:13.8334998Z * [new branch] gh/IvanKobzarev/131/head -> origin/gh/IvanKobzarev/131/head 2025-08-14T21:25:13.8335384Z * [new branch] gh/IvanKobzarev/131/orig -> origin/gh/IvanKobzarev/131/orig 2025-08-14T21:25:13.8335764Z * [new branch] gh/IvanKobzarev/132/base -> origin/gh/IvanKobzarev/132/base 2025-08-14T21:25:13.8336111Z * [new branch] gh/IvanKobzarev/132/head -> origin/gh/IvanKobzarev/132/head 2025-08-14T21:25:13.8336468Z * [new branch] gh/IvanKobzarev/132/orig -> origin/gh/IvanKobzarev/132/orig 2025-08-14T21:25:13.8336831Z * [new branch] gh/IvanKobzarev/133/base -> origin/gh/IvanKobzarev/133/base 2025-08-14T21:25:13.8337202Z * [new branch] gh/IvanKobzarev/133/head -> origin/gh/IvanKobzarev/133/head 2025-08-14T21:25:13.8337530Z * [new branch] gh/IvanKobzarev/133/orig -> origin/gh/IvanKobzarev/133/orig 2025-08-14T21:25:13.8337871Z * [new branch] gh/IvanKobzarev/134/base -> origin/gh/IvanKobzarev/134/base 2025-08-14T21:25:13.8338572Z * [new branch] gh/IvanKobzarev/134/head -> origin/gh/IvanKobzarev/134/head 2025-08-14T21:25:13.8338960Z * [new branch] gh/IvanKobzarev/134/orig -> origin/gh/IvanKobzarev/134/orig 2025-08-14T21:25:13.8339324Z * [new branch] gh/IvanKobzarev/135/base -> origin/gh/IvanKobzarev/135/base 2025-08-14T21:25:13.8339712Z * [new branch] gh/IvanKobzarev/135/head -> origin/gh/IvanKobzarev/135/head 2025-08-14T21:25:13.8340273Z * [new branch] gh/IvanKobzarev/135/orig -> origin/gh/IvanKobzarev/135/orig 2025-08-14T21:25:13.8342924Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-08-14T21:25:13.8343563Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-08-14T21:25:13.8344098Z * [new branch] gh/NikhilAPatel/16/base -> origin/gh/NikhilAPatel/16/base 2025-08-14T21:25:13.8344627Z * [new branch] gh/NikhilAPatel/16/head -> origin/gh/NikhilAPatel/16/head 2025-08-14T21:25:13.8349130Z * [new branch] gh/NikhilAPatel/16/orig -> origin/gh/NikhilAPatel/16/orig 2025-08-14T21:25:13.8349778Z * [new branch] gh/NikhilAPatel/18/base -> origin/gh/NikhilAPatel/18/base 2025-08-14T21:25:13.8350305Z * [new branch] gh/NikhilAPatel/18/head -> origin/gh/NikhilAPatel/18/head 2025-08-14T21:25:13.8350848Z * [new branch] gh/NikhilAPatel/18/orig -> origin/gh/NikhilAPatel/18/orig 2025-08-14T21:25:13.8351744Z * [new branch] gh/NikhilAPatel/19/base -> origin/gh/NikhilAPatel/19/base 2025-08-14T21:25:13.8352165Z * [new branch] gh/NikhilAPatel/19/head -> origin/gh/NikhilAPatel/19/head 2025-08-14T21:25:13.8352542Z * [new branch] gh/NikhilAPatel/19/orig -> origin/gh/NikhilAPatel/19/orig 2025-08-14T21:25:13.8352924Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-08-14T21:25:13.8353319Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-08-14T21:25:13.8353840Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-08-14T21:25:13.8354206Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-08-14T21:25:13.8354570Z * [new branch] gh/NikhilAPatel/8/base -> origin/gh/NikhilAPatel/8/base 2025-08-14T21:25:13.8354927Z * [new branch] gh/NikhilAPatel/8/head -> origin/gh/NikhilAPatel/8/head 2025-08-14T21:25:13.8355281Z * [new branch] gh/NikhilAPatel/8/orig -> origin/gh/NikhilAPatel/8/orig 2025-08-14T21:25:13.8355643Z * [new branch] gh/NikhilAPatel/9/base -> origin/gh/NikhilAPatel/9/base 2025-08-14T21:25:13.8356010Z * [new branch] gh/NikhilAPatel/9/head -> origin/gh/NikhilAPatel/9/head 2025-08-14T21:25:13.8356774Z * [new branch] gh/NikhilAPatel/9/orig -> origin/gh/NikhilAPatel/9/orig 2025-08-14T21:25:13.8359853Z * [new branch] gh/PaliC/1/base -> origin/gh/PaliC/1/base 2025-08-14T21:25:13.8360419Z * [new branch] gh/PaliC/1/head -> origin/gh/PaliC/1/head 2025-08-14T21:25:13.8360860Z * [new branch] gh/PaliC/1/orig -> origin/gh/PaliC/1/orig 2025-08-14T21:25:13.8361313Z * [new branch] gh/PaliC/12/base -> origin/gh/PaliC/12/base 2025-08-14T21:25:13.8361756Z * [new branch] gh/PaliC/12/head -> origin/gh/PaliC/12/head 2025-08-14T21:25:13.8362070Z * [new branch] gh/PaliC/12/orig -> origin/gh/PaliC/12/orig 2025-08-14T21:25:13.8362717Z * [new branch] gh/PaliC/13/base -> origin/gh/PaliC/13/base 2025-08-14T21:25:13.8363513Z * [new branch] gh/PaliC/13/head -> origin/gh/PaliC/13/head 2025-08-14T21:25:13.8363911Z * [new branch] gh/PaliC/13/orig -> origin/gh/PaliC/13/orig 2025-08-14T21:25:13.8370136Z * [new branch] gh/PaliC/14/base -> origin/gh/PaliC/14/base 2025-08-14T21:25:13.8372237Z * [new branch] gh/PaliC/14/head -> origin/gh/PaliC/14/head 2025-08-14T21:25:13.8372625Z * [new branch] gh/PaliC/14/orig -> origin/gh/PaliC/14/orig 2025-08-14T21:25:13.8372943Z * [new branch] gh/PaliC/15/base -> origin/gh/PaliC/15/base 2025-08-14T21:25:13.8373261Z * [new branch] gh/PaliC/15/head -> origin/gh/PaliC/15/head 2025-08-14T21:25:13.8373574Z * [new branch] gh/PaliC/15/orig -> origin/gh/PaliC/15/orig 2025-08-14T21:25:13.8373880Z * [new branch] gh/PaliC/16/base -> origin/gh/PaliC/16/base 2025-08-14T21:25:13.8374180Z * [new branch] gh/PaliC/16/head -> origin/gh/PaliC/16/head 2025-08-14T21:25:13.8374483Z * [new branch] gh/PaliC/16/orig -> origin/gh/PaliC/16/orig 2025-08-14T21:25:13.8374826Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-08-14T21:25:13.8375134Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-08-14T21:25:13.8375441Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-08-14T21:25:13.8375748Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-08-14T21:25:13.8376047Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-08-14T21:25:13.8376341Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-08-14T21:25:13.8376643Z * [new branch] gh/PaliC/19/base -> origin/gh/PaliC/19/base 2025-08-14T21:25:13.8376960Z * [new branch] gh/PaliC/19/head -> origin/gh/PaliC/19/head 2025-08-14T21:25:13.8377268Z * [new branch] gh/PaliC/19/orig -> origin/gh/PaliC/19/orig 2025-08-14T21:25:13.8377837Z * [new branch] gh/PaliC/2/base -> origin/gh/PaliC/2/base 2025-08-14T21:25:13.8378713Z * [new branch] gh/PaliC/2/head -> origin/gh/PaliC/2/head 2025-08-14T21:25:13.8379111Z * [new branch] gh/PaliC/2/orig -> origin/gh/PaliC/2/orig 2025-08-14T21:25:13.8382900Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-08-14T21:25:13.8383481Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-08-14T21:25:13.8383957Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-08-14T21:25:13.8384441Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-08-14T21:25:13.8384881Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-08-14T21:25:13.8385325Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-08-14T21:25:13.8385780Z * [new branch] gh/PaliC/22/base -> origin/gh/PaliC/22/base 2025-08-14T21:25:13.8386222Z * [new branch] gh/PaliC/22/head -> origin/gh/PaliC/22/head 2025-08-14T21:25:13.8386651Z * [new branch] gh/PaliC/22/orig -> origin/gh/PaliC/22/orig 2025-08-14T21:25:13.8386961Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-08-14T21:25:13.8387275Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-08-14T21:25:13.8387737Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-08-14T21:25:13.8388816Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-08-14T21:25:13.8389366Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-08-14T21:25:13.8390026Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-08-14T21:25:13.8391791Z * [new branch] gh/PaulZhang12/17/base -> origin/gh/PaulZhang12/17/base 2025-08-14T21:25:13.8392186Z * [new branch] gh/PaulZhang12/17/head -> origin/gh/PaulZhang12/17/head 2025-08-14T21:25:13.8392876Z * [new branch] gh/PaulZhang12/18/base -> origin/gh/PaulZhang12/18/base 2025-08-14T21:25:13.8394070Z * [new branch] gh/PaulZhang12/18/head -> origin/gh/PaulZhang12/18/head 2025-08-14T21:25:13.8394706Z * [new branch] gh/PaulZhang12/18/orig -> origin/gh/PaulZhang12/18/orig 2025-08-14T21:25:13.8395656Z * [new branch] gh/PaulZhang12/19/base -> origin/gh/PaulZhang12/19/base 2025-08-14T21:25:13.8396394Z * [new branch] gh/PaulZhang12/19/head -> origin/gh/PaulZhang12/19/head 2025-08-14T21:25:13.8397083Z * [new branch] gh/PaulZhang12/19/orig -> origin/gh/PaulZhang12/19/orig 2025-08-14T21:25:13.8401134Z * [new branch] gh/PaulZhang12/20/base -> origin/gh/PaulZhang12/20/base 2025-08-14T21:25:13.8401579Z * [new branch] gh/PaulZhang12/20/head -> origin/gh/PaulZhang12/20/head 2025-08-14T21:25:13.8401948Z * [new branch] gh/PaulZhang12/20/orig -> origin/gh/PaulZhang12/20/orig 2025-08-14T21:25:13.8402311Z * [new branch] gh/PaulZhang12/21/base -> origin/gh/PaulZhang12/21/base 2025-08-14T21:25:13.8402664Z * [new branch] gh/PaulZhang12/21/head -> origin/gh/PaulZhang12/21/head 2025-08-14T21:25:13.8403018Z * [new branch] gh/PaulZhang12/21/orig -> origin/gh/PaulZhang12/21/orig 2025-08-14T21:25:13.8403364Z * [new branch] gh/PaulZhang12/22/base -> origin/gh/PaulZhang12/22/base 2025-08-14T21:25:13.8403719Z * [new branch] gh/PaulZhang12/22/head -> origin/gh/PaulZhang12/22/head 2025-08-14T21:25:13.8404298Z * [new branch] gh/PaulZhang12/22/orig -> origin/gh/PaulZhang12/22/orig 2025-08-14T21:25:13.8405476Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-08-14T21:25:13.8405860Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-08-14T21:25:13.8410290Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-08-14T21:25:13.8411025Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-08-14T21:25:13.8411542Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-08-14T21:25:13.8412033Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-08-14T21:25:13.8412394Z * [new branch] gh/Sidharth123-cpu/42/base -> origin/gh/Sidharth123-cpu/42/base 2025-08-14T21:25:13.8412751Z * [new branch] gh/Sidharth123-cpu/42/head -> origin/gh/Sidharth123-cpu/42/head 2025-08-14T21:25:13.8413107Z * [new branch] gh/Sidharth123-cpu/42/orig -> origin/gh/Sidharth123-cpu/42/orig 2025-08-14T21:25:13.8413820Z * [new branch] gh/Sidharth123-cpu/43/base -> origin/gh/Sidharth123-cpu/43/base 2025-08-14T21:25:13.8414511Z * [new branch] gh/Sidharth123-cpu/43/head -> origin/gh/Sidharth123-cpu/43/head 2025-08-14T21:25:13.8415222Z * [new branch] gh/Sidharth123-cpu/43/orig -> origin/gh/Sidharth123-cpu/43/orig 2025-08-14T21:25:13.8416449Z * [new branch] gh/Sidharth123-cpu/44/base -> origin/gh/Sidharth123-cpu/44/base 2025-08-14T21:25:13.8416821Z * [new branch] gh/Sidharth123-cpu/44/head -> origin/gh/Sidharth123-cpu/44/head 2025-08-14T21:25:13.8417325Z * [new branch] gh/Sidharth123-cpu/44/orig -> origin/gh/Sidharth123-cpu/44/orig 2025-08-14T21:25:13.8418916Z * [new branch] gh/Sidharth123-cpu/45/base -> origin/gh/Sidharth123-cpu/45/base 2025-08-14T21:25:13.8419532Z * [new branch] gh/Sidharth123-cpu/45/head -> origin/gh/Sidharth123-cpu/45/head 2025-08-14T21:25:13.8420246Z * [new branch] gh/Sidharth123-cpu/45/orig -> origin/gh/Sidharth123-cpu/45/orig 2025-08-14T21:25:13.8420789Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-08-14T21:25:13.8421482Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-08-14T21:25:13.8422958Z * [new branch] gh/StrongerXi/103/base -> origin/gh/StrongerXi/103/base 2025-08-14T21:25:13.8423440Z * [new branch] gh/StrongerXi/103/head -> origin/gh/StrongerXi/103/head 2025-08-14T21:25:13.8423910Z * [new branch] gh/StrongerXi/103/orig -> origin/gh/StrongerXi/103/orig 2025-08-14T21:25:13.8424502Z * [new branch] gh/StrongerXi/133/base -> origin/gh/StrongerXi/133/base 2025-08-14T21:25:13.8425210Z * [new branch] gh/StrongerXi/133/head -> origin/gh/StrongerXi/133/head 2025-08-14T21:25:13.8425856Z * [new branch] gh/StrongerXi/133/orig -> origin/gh/StrongerXi/133/orig 2025-08-14T21:25:13.8429081Z * [new branch] gh/StrongerXi/134/base -> origin/gh/StrongerXi/134/base 2025-08-14T21:25:13.8429704Z * [new branch] gh/StrongerXi/134/head -> origin/gh/StrongerXi/134/head 2025-08-14T21:25:13.8430083Z * [new branch] gh/StrongerXi/134/orig -> origin/gh/StrongerXi/134/orig 2025-08-14T21:25:13.8430440Z * [new branch] gh/StrongerXi/135/base -> origin/gh/StrongerXi/135/base 2025-08-14T21:25:13.8430799Z * [new branch] gh/StrongerXi/135/head -> origin/gh/StrongerXi/135/head 2025-08-14T21:25:13.8431157Z * [new branch] gh/StrongerXi/135/orig -> origin/gh/StrongerXi/135/orig 2025-08-14T21:25:13.8432814Z * [new branch] gh/StrongerXi/136/base -> origin/gh/StrongerXi/136/base 2025-08-14T21:25:13.8433178Z * [new branch] gh/StrongerXi/136/head -> origin/gh/StrongerXi/136/head 2025-08-14T21:25:13.8433526Z * [new branch] gh/StrongerXi/136/orig -> origin/gh/StrongerXi/136/orig 2025-08-14T21:25:13.8433892Z * [new branch] gh/StrongerXi/137/base -> origin/gh/StrongerXi/137/base 2025-08-14T21:25:13.8434440Z * [new branch] gh/StrongerXi/137/head -> origin/gh/StrongerXi/137/head 2025-08-14T21:25:13.8434800Z * [new branch] gh/StrongerXi/137/orig -> origin/gh/StrongerXi/137/orig 2025-08-14T21:25:13.8436137Z * [new branch] gh/StrongerXi/138/base -> origin/gh/StrongerXi/138/base 2025-08-14T21:25:13.8436524Z * [new branch] gh/StrongerXi/138/head -> origin/gh/StrongerXi/138/head 2025-08-14T21:25:13.8440933Z * [new branch] gh/StrongerXi/138/orig -> origin/gh/StrongerXi/138/orig 2025-08-14T21:25:13.8441596Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-08-14T21:25:13.8442127Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-08-14T21:25:13.8443003Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-08-14T21:25:13.8443437Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-08-14T21:25:13.8443808Z * [new branch] gh/XilunWu/131/base -> origin/gh/XilunWu/131/base 2025-08-14T21:25:13.8448922Z * [new branch] gh/XilunWu/131/head -> origin/gh/XilunWu/131/head 2025-08-14T21:25:13.8449496Z * [new branch] gh/XilunWu/131/orig -> origin/gh/XilunWu/131/orig 2025-08-14T21:25:13.8449964Z * [new branch] gh/XilunWu/133/base -> origin/gh/XilunWu/133/base 2025-08-14T21:25:13.8450431Z * [new branch] gh/XilunWu/133/head -> origin/gh/XilunWu/133/head 2025-08-14T21:25:13.8451355Z * [new branch] gh/XilunWu/133/orig -> origin/gh/XilunWu/133/orig 2025-08-14T21:25:13.8451743Z * [new branch] gh/XilunWu/136/base -> origin/gh/XilunWu/136/base 2025-08-14T21:25:13.8452240Z * [new branch] gh/XilunWu/136/head -> origin/gh/XilunWu/136/head 2025-08-14T21:25:13.8452582Z * [new branch] gh/XilunWu/136/orig -> origin/gh/XilunWu/136/orig 2025-08-14T21:25:13.8452896Z * [new branch] gh/XilunWu/139/base -> origin/gh/XilunWu/139/base 2025-08-14T21:25:13.8453204Z * [new branch] gh/XilunWu/139/head -> origin/gh/XilunWu/139/head 2025-08-14T21:25:13.8453563Z * [new branch] gh/XilunWu/139/orig -> origin/gh/XilunWu/139/orig 2025-08-14T21:25:13.8453880Z * [new branch] gh/XilunWu/143/base -> origin/gh/XilunWu/143/base 2025-08-14T21:25:13.8454192Z * [new branch] gh/XilunWu/143/head -> origin/gh/XilunWu/143/head 2025-08-14T21:25:13.8454511Z * [new branch] gh/XilunWu/143/orig -> origin/gh/XilunWu/143/orig 2025-08-14T21:25:13.8454836Z * [new branch] gh/XilunWu/144/base -> origin/gh/XilunWu/144/base 2025-08-14T21:25:13.8455175Z * [new branch] gh/XilunWu/144/head -> origin/gh/XilunWu/144/head 2025-08-14T21:25:13.8455516Z * [new branch] gh/XilunWu/144/orig -> origin/gh/XilunWu/144/orig 2025-08-14T21:25:13.8455864Z * [new branch] gh/XilunWu/145/base -> origin/gh/XilunWu/145/base 2025-08-14T21:25:13.8456213Z * [new branch] gh/XilunWu/145/head -> origin/gh/XilunWu/145/head 2025-08-14T21:25:13.8456560Z * [new branch] gh/XilunWu/145/orig -> origin/gh/XilunWu/145/orig 2025-08-14T21:25:13.8456894Z * [new branch] gh/XilunWu/146/base -> origin/gh/XilunWu/146/base 2025-08-14T21:25:13.8457222Z * [new branch] gh/XilunWu/146/head -> origin/gh/XilunWu/146/head 2025-08-14T21:25:13.8457599Z * [new branch] gh/XilunWu/146/orig -> origin/gh/XilunWu/146/orig 2025-08-14T21:25:13.8458704Z * [new branch] gh/XilunWu/147/base -> origin/gh/XilunWu/147/base 2025-08-14T21:25:13.8459042Z * [new branch] gh/XilunWu/147/head -> origin/gh/XilunWu/147/head 2025-08-14T21:25:13.8459683Z * [new branch] gh/XilunWu/147/orig -> origin/gh/XilunWu/147/orig 2025-08-14T21:25:13.8461400Z * [new branch] gh/XilunWu/148/base -> origin/gh/XilunWu/148/base 2025-08-14T21:25:13.8461819Z * [new branch] gh/XilunWu/148/head -> origin/gh/XilunWu/148/head 2025-08-14T21:25:13.8462177Z * [new branch] gh/XilunWu/148/orig -> origin/gh/XilunWu/148/orig 2025-08-14T21:25:13.8467643Z * [new branch] gh/XilunWu/149/base -> origin/gh/XilunWu/149/base 2025-08-14T21:25:13.8471850Z * [new branch] gh/XilunWu/149/head -> origin/gh/XilunWu/149/head 2025-08-14T21:25:13.8472262Z * [new branch] gh/XilunWu/149/orig -> origin/gh/XilunWu/149/orig 2025-08-14T21:25:13.8472636Z * [new branch] gh/XilunWu/150/base -> origin/gh/XilunWu/150/base 2025-08-14T21:25:13.8472988Z * [new branch] gh/XilunWu/150/head -> origin/gh/XilunWu/150/head 2025-08-14T21:25:13.8473334Z * [new branch] gh/XilunWu/150/orig -> origin/gh/XilunWu/150/orig 2025-08-14T21:25:13.8473674Z * [new branch] gh/XilunWu/151/base -> origin/gh/XilunWu/151/base 2025-08-14T21:25:13.8474023Z * [new branch] gh/XilunWu/151/head -> origin/gh/XilunWu/151/head 2025-08-14T21:25:13.8474393Z * [new branch] gh/XilunWu/151/orig -> origin/gh/XilunWu/151/orig 2025-08-14T21:25:13.8474731Z * [new branch] gh/XilunWu/152/base -> origin/gh/XilunWu/152/base 2025-08-14T21:25:13.8475073Z * [new branch] gh/XilunWu/152/head -> origin/gh/XilunWu/152/head 2025-08-14T21:25:13.8475407Z * [new branch] gh/XilunWu/152/orig -> origin/gh/XilunWu/152/orig 2025-08-14T21:25:13.8475930Z * [new branch] gh/XilunWu/153/base -> origin/gh/XilunWu/153/base 2025-08-14T21:25:13.8476624Z * [new branch] gh/XilunWu/153/head -> origin/gh/XilunWu/153/head 2025-08-14T21:25:13.8476969Z * [new branch] gh/XilunWu/153/orig -> origin/gh/XilunWu/153/orig 2025-08-14T21:25:13.8477310Z * [new branch] gh/XilunWu/154/base -> origin/gh/XilunWu/154/base 2025-08-14T21:25:13.8477648Z * [new branch] gh/XilunWu/154/head -> origin/gh/XilunWu/154/head 2025-08-14T21:25:13.8477988Z * [new branch] gh/XilunWu/154/orig -> origin/gh/XilunWu/154/orig 2025-08-14T21:25:13.8478534Z * [new branch] gh/XilunWu/156/base -> origin/gh/XilunWu/156/base 2025-08-14T21:25:13.8479053Z * [new branch] gh/XilunWu/156/head -> origin/gh/XilunWu/156/head 2025-08-14T21:25:13.8479523Z * [new branch] gh/XilunWu/156/orig -> origin/gh/XilunWu/156/orig 2025-08-14T21:25:13.8480025Z * [new branch] gh/XilunWu/157/base -> origin/gh/XilunWu/157/base 2025-08-14T21:25:13.8480518Z * [new branch] gh/XilunWu/157/head -> origin/gh/XilunWu/157/head 2025-08-14T21:25:13.8481374Z * [new branch] gh/XilunWu/157/orig -> origin/gh/XilunWu/157/orig 2025-08-14T21:25:13.8482217Z * [new branch] gh/XilunWu/158/base -> origin/gh/XilunWu/158/base 2025-08-14T21:25:13.8482763Z * [new branch] gh/XilunWu/158/head -> origin/gh/XilunWu/158/head 2025-08-14T21:25:13.8483337Z * [new branch] gh/XilunWu/158/orig -> origin/gh/XilunWu/158/orig 2025-08-14T21:25:13.8486930Z * [new branch] gh/XilunWu/159/base -> origin/gh/XilunWu/159/base 2025-08-14T21:25:13.8487341Z * [new branch] gh/XilunWu/159/head -> origin/gh/XilunWu/159/head 2025-08-14T21:25:13.8487690Z * [new branch] gh/XilunWu/159/orig -> origin/gh/XilunWu/159/orig 2025-08-14T21:25:13.8488069Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-08-14T21:25:13.8488598Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-08-14T21:25:13.8488946Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-08-14T21:25:13.8490104Z * [new branch] gh/XilunWu/161/base -> origin/gh/XilunWu/161/base 2025-08-14T21:25:13.8490527Z * [new branch] gh/XilunWu/161/head -> origin/gh/XilunWu/161/head 2025-08-14T21:25:13.8491093Z * [new branch] gh/XilunWu/161/orig -> origin/gh/XilunWu/161/orig 2025-08-14T21:25:13.8492377Z * [new branch] gh/XilunWu/162/base -> origin/gh/XilunWu/162/base 2025-08-14T21:25:13.8492966Z * [new branch] gh/XilunWu/162/head -> origin/gh/XilunWu/162/head 2025-08-14T21:25:13.8493567Z * [new branch] gh/XilunWu/162/orig -> origin/gh/XilunWu/162/orig 2025-08-14T21:25:13.8494891Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-08-14T21:25:13.8495260Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-08-14T21:25:13.8496424Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-08-14T21:25:13.8497033Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-08-14T21:25:13.8497633Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-08-14T21:25:13.8498312Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-08-14T21:25:13.8499597Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-08-14T21:25:13.8499975Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-08-14T21:25:13.8500741Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-08-14T21:25:13.8502081Z * [new branch] gh/XuehaiPan/189/base -> origin/gh/XuehaiPan/189/base 2025-08-14T21:25:13.8502459Z * [new branch] gh/XuehaiPan/189/head -> origin/gh/XuehaiPan/189/head 2025-08-14T21:25:13.8503129Z * [new branch] gh/XuehaiPan/189/orig -> origin/gh/XuehaiPan/189/orig 2025-08-14T21:25:13.8503941Z * [new branch] gh/XuehaiPan/227/base -> origin/gh/XuehaiPan/227/base 2025-08-14T21:25:13.8504621Z * [new branch] gh/XuehaiPan/227/head -> origin/gh/XuehaiPan/227/head 2025-08-14T21:25:13.8505266Z * [new branch] gh/XuehaiPan/227/orig -> origin/gh/XuehaiPan/227/orig 2025-08-14T21:25:13.8506573Z * [new branch] gh/XuehaiPan/231/base -> origin/gh/XuehaiPan/231/base 2025-08-14T21:25:13.8507495Z * [new branch] gh/XuehaiPan/231/head -> origin/gh/XuehaiPan/231/head 2025-08-14T21:25:13.8507945Z * [new branch] gh/XuehaiPan/231/orig -> origin/gh/XuehaiPan/231/orig 2025-08-14T21:25:13.8508630Z * [new branch] gh/XuehaiPan/232/base -> origin/gh/XuehaiPan/232/base 2025-08-14T21:25:13.8509246Z * [new branch] gh/XuehaiPan/232/head -> origin/gh/XuehaiPan/232/head 2025-08-14T21:25:13.8510245Z * [new branch] gh/XuehaiPan/232/orig -> origin/gh/XuehaiPan/232/orig 2025-08-14T21:25:13.8511210Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-08-14T21:25:13.8511754Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-08-14T21:25:13.8512588Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-08-14T21:25:13.8513379Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-08-14T21:25:13.8514113Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-08-14T21:25:13.8514745Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-08-14T21:25:13.8515923Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-08-14T21:25:13.8517383Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-08-14T21:25:13.8517809Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-08-14T21:25:13.8518182Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-08-14T21:25:13.8518774Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-08-14T21:25:13.8519260Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-08-14T21:25:13.8520478Z * [new branch] gh/XuehaiPan/257/base -> origin/gh/XuehaiPan/257/base 2025-08-14T21:25:13.8520843Z * [new branch] gh/XuehaiPan/257/head -> origin/gh/XuehaiPan/257/head 2025-08-14T21:25:13.8521512Z * [new branch] gh/XuehaiPan/257/orig -> origin/gh/XuehaiPan/257/orig 2025-08-14T21:25:13.8522637Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-08-14T21:25:13.8522984Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-08-14T21:25:13.8523711Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-08-14T21:25:13.8525088Z * [new branch] gh/XuehaiPan/283/base -> origin/gh/XuehaiPan/283/base 2025-08-14T21:25:13.8525590Z * [new branch] gh/XuehaiPan/283/head -> origin/gh/XuehaiPan/283/head 2025-08-14T21:25:13.8525945Z * [new branch] gh/XuehaiPan/283/orig -> origin/gh/XuehaiPan/283/orig 2025-08-14T21:25:13.8527777Z * [new branch] gh/XuehaiPan/290/base -> origin/gh/XuehaiPan/290/base 2025-08-14T21:25:13.8528471Z * [new branch] gh/XuehaiPan/290/head -> origin/gh/XuehaiPan/290/head 2025-08-14T21:25:13.8528900Z * [new branch] gh/XuehaiPan/290/orig -> origin/gh/XuehaiPan/290/orig 2025-08-14T21:25:13.8529290Z * [new branch] gh/XuehaiPan/328/base -> origin/gh/XuehaiPan/328/base 2025-08-14T21:25:13.8529676Z * [new branch] gh/XuehaiPan/328/head -> origin/gh/XuehaiPan/328/head 2025-08-14T21:25:13.8530438Z * [new branch] gh/XuehaiPan/328/orig -> origin/gh/XuehaiPan/328/orig 2025-08-14T21:25:13.8531546Z * [new branch] gh/XuehaiPan/339/base -> origin/gh/XuehaiPan/339/base 2025-08-14T21:25:13.8531904Z * [new branch] gh/XuehaiPan/339/head -> origin/gh/XuehaiPan/339/head 2025-08-14T21:25:13.8532593Z * [new branch] gh/XuehaiPan/339/orig -> origin/gh/XuehaiPan/339/orig 2025-08-14T21:25:13.8536706Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-08-14T21:25:13.8537132Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-08-14T21:25:13.8537480Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-08-14T21:25:13.8537817Z * [new branch] gh/XuehaiPan/344/base -> origin/gh/XuehaiPan/344/base 2025-08-14T21:25:13.8538140Z * [new branch] gh/XuehaiPan/344/head -> origin/gh/XuehaiPan/344/head 2025-08-14T21:25:13.8538475Z * [new branch] gh/XuehaiPan/344/orig -> origin/gh/XuehaiPan/344/orig 2025-08-14T21:25:13.8539009Z * [new branch] gh/XuehaiPan/345/base -> origin/gh/XuehaiPan/345/base 2025-08-14T21:25:13.8539727Z * [new branch] gh/XuehaiPan/345/head -> origin/gh/XuehaiPan/345/head 2025-08-14T21:25:13.8540177Z * [new branch] gh/XuehaiPan/345/orig -> origin/gh/XuehaiPan/345/orig 2025-08-14T21:25:13.8540576Z * [new branch] gh/XuehaiPan/346/base -> origin/gh/XuehaiPan/346/base 2025-08-14T21:25:13.8541062Z * [new branch] gh/XuehaiPan/346/head -> origin/gh/XuehaiPan/346/head 2025-08-14T21:25:13.8541955Z * [new branch] gh/XuehaiPan/346/orig -> origin/gh/XuehaiPan/346/orig 2025-08-14T21:25:13.8542501Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-08-14T21:25:13.8543154Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-08-14T21:25:13.8543825Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-08-14T21:25:13.8545641Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-08-14T21:25:13.8546055Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-08-14T21:25:13.8546466Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-08-14T21:25:13.8547454Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-08-14T21:25:13.8548372Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-08-14T21:25:13.8549030Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-08-14T21:25:13.8549805Z * [new branch] gh/XuehaiPan/352/base -> origin/gh/XuehaiPan/352/base 2025-08-14T21:25:13.8550255Z * [new branch] gh/XuehaiPan/352/head -> origin/gh/XuehaiPan/352/head 2025-08-14T21:25:13.8551156Z * [new branch] gh/XuehaiPan/352/orig -> origin/gh/XuehaiPan/352/orig 2025-08-14T21:25:13.8552231Z * [new branch] gh/XuehaiPan/356/base -> origin/gh/XuehaiPan/356/base 2025-08-14T21:25:13.8552599Z * [new branch] gh/XuehaiPan/356/head -> origin/gh/XuehaiPan/356/head 2025-08-14T21:25:13.8553284Z * [new branch] gh/XuehaiPan/356/orig -> origin/gh/XuehaiPan/356/orig 2025-08-14T21:25:13.8554357Z * [new branch] gh/XuehaiPan/357/base -> origin/gh/XuehaiPan/357/base 2025-08-14T21:25:13.8554769Z * [new branch] gh/XuehaiPan/357/head -> origin/gh/XuehaiPan/357/head 2025-08-14T21:25:13.8555543Z * [new branch] gh/XuehaiPan/357/orig -> origin/gh/XuehaiPan/357/orig 2025-08-14T21:25:13.8556668Z * [new branch] gh/XuehaiPan/358/base -> origin/gh/XuehaiPan/358/base 2025-08-14T21:25:13.8557142Z * [new branch] gh/XuehaiPan/358/head -> origin/gh/XuehaiPan/358/head 2025-08-14T21:25:13.8557696Z * [new branch] gh/XuehaiPan/358/orig -> origin/gh/XuehaiPan/358/orig 2025-08-14T21:25:13.8561645Z * [new branch] gh/XuehaiPan/359/base -> origin/gh/XuehaiPan/359/base 2025-08-14T21:25:13.8562107Z * [new branch] gh/XuehaiPan/359/head -> origin/gh/XuehaiPan/359/head 2025-08-14T21:25:13.8562486Z * [new branch] gh/XuehaiPan/359/orig -> origin/gh/XuehaiPan/359/orig 2025-08-14T21:25:13.8562871Z * [new branch] gh/XuehaiPan/360/base -> origin/gh/XuehaiPan/360/base 2025-08-14T21:25:13.8563242Z * [new branch] gh/XuehaiPan/360/head -> origin/gh/XuehaiPan/360/head 2025-08-14T21:25:13.8563600Z * [new branch] gh/XuehaiPan/360/orig -> origin/gh/XuehaiPan/360/orig 2025-08-14T21:25:13.8563963Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-08-14T21:25:13.8564297Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-08-14T21:25:13.8564963Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-08-14T21:25:13.8569515Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-08-14T21:25:13.8570089Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-08-14T21:25:13.8570588Z * [new branch] gh/XuehaiPan/368/base -> origin/gh/XuehaiPan/368/base 2025-08-14T21:25:13.8571102Z * [new branch] gh/XuehaiPan/368/head -> origin/gh/XuehaiPan/368/head 2025-08-14T21:25:13.8571436Z * [new branch] gh/XuehaiPan/368/orig -> origin/gh/XuehaiPan/368/orig 2025-08-14T21:25:13.8571773Z * [new branch] gh/XuehaiPan/369/base -> origin/gh/XuehaiPan/369/base 2025-08-14T21:25:13.8572105Z * [new branch] gh/XuehaiPan/369/head -> origin/gh/XuehaiPan/369/head 2025-08-14T21:25:13.8572438Z * [new branch] gh/XuehaiPan/369/orig -> origin/gh/XuehaiPan/369/orig 2025-08-14T21:25:13.8572771Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-08-14T21:25:13.8573104Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-08-14T21:25:13.8573616Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-08-14T21:25:13.8574002Z * [new branch] gh/XuehaiPan/371/base -> origin/gh/XuehaiPan/371/base 2025-08-14T21:25:13.8579374Z * [new branch] gh/XuehaiPan/371/head -> origin/gh/XuehaiPan/371/head 2025-08-14T21:25:13.8582971Z * [new branch] gh/XuehaiPan/371/orig -> origin/gh/XuehaiPan/371/orig 2025-08-14T21:25:13.8583368Z * [new branch] gh/XuehaiPan/372/base -> origin/gh/XuehaiPan/372/base 2025-08-14T21:25:13.8583709Z * [new branch] gh/XuehaiPan/372/head -> origin/gh/XuehaiPan/372/head 2025-08-14T21:25:13.8584077Z * [new branch] gh/XuehaiPan/372/orig -> origin/gh/XuehaiPan/372/orig 2025-08-14T21:25:13.8584437Z * [new branch] gh/XuehaiPan/373/base -> origin/gh/XuehaiPan/373/base 2025-08-14T21:25:13.8584806Z * [new branch] gh/XuehaiPan/373/head -> origin/gh/XuehaiPan/373/head 2025-08-14T21:25:13.8585168Z * [new branch] gh/XuehaiPan/373/orig -> origin/gh/XuehaiPan/373/orig 2025-08-14T21:25:13.8585679Z * [new branch] gh/XuehaiPan/374/base -> origin/gh/XuehaiPan/374/base 2025-08-14T21:25:13.8586050Z * [new branch] gh/XuehaiPan/374/head -> origin/gh/XuehaiPan/374/head 2025-08-14T21:25:13.8586412Z * [new branch] gh/XuehaiPan/374/orig -> origin/gh/XuehaiPan/374/orig 2025-08-14T21:25:13.8586753Z * [new branch] gh/XuehaiPan/375/base -> origin/gh/XuehaiPan/375/base 2025-08-14T21:25:13.8587074Z * [new branch] gh/XuehaiPan/375/head -> origin/gh/XuehaiPan/375/head 2025-08-14T21:25:13.8587394Z * [new branch] gh/XuehaiPan/375/orig -> origin/gh/XuehaiPan/375/orig 2025-08-14T21:25:13.8587723Z * [new branch] gh/XuehaiPan/376/base -> origin/gh/XuehaiPan/376/base 2025-08-14T21:25:13.8588075Z * [new branch] gh/XuehaiPan/376/head -> origin/gh/XuehaiPan/376/head 2025-08-14T21:25:13.8588428Z * [new branch] gh/XuehaiPan/376/orig -> origin/gh/XuehaiPan/376/orig 2025-08-14T21:25:13.8588766Z * [new branch] gh/XuehaiPan/377/base -> origin/gh/XuehaiPan/377/base 2025-08-14T21:25:13.8589112Z * [new branch] gh/XuehaiPan/377/head -> origin/gh/XuehaiPan/377/head 2025-08-14T21:25:13.8589464Z * [new branch] gh/XuehaiPan/377/orig -> origin/gh/XuehaiPan/377/orig 2025-08-14T21:25:13.8589822Z * [new branch] gh/XuehaiPan/378/base -> origin/gh/XuehaiPan/378/base 2025-08-14T21:25:13.8590434Z * [new branch] gh/XuehaiPan/378/head -> origin/gh/XuehaiPan/378/head 2025-08-14T21:25:13.8591051Z * [new branch] gh/XuehaiPan/378/orig -> origin/gh/XuehaiPan/378/orig 2025-08-14T21:25:13.8592194Z * [new branch] gh/XuehaiPan/379/base -> origin/gh/XuehaiPan/379/base 2025-08-14T21:25:13.8592569Z * [new branch] gh/XuehaiPan/379/head -> origin/gh/XuehaiPan/379/head 2025-08-14T21:25:13.8593322Z * [new branch] gh/XuehaiPan/379/orig -> origin/gh/XuehaiPan/379/orig 2025-08-14T21:25:13.8594712Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-08-14T21:25:13.8595095Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-08-14T21:25:13.8595814Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-08-14T21:25:13.8597161Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-08-14T21:25:13.8597858Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-08-14T21:25:13.8598463Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-08-14T21:25:13.8599061Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-08-14T21:25:13.8603008Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-08-14T21:25:13.8606676Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-08-14T21:25:13.8607072Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-08-14T21:25:13.8607443Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-08-14T21:25:13.8607809Z * [new branch] gh/ZhiweiYan-96/64/base -> origin/gh/ZhiweiYan-96/64/base 2025-08-14T21:25:13.8608172Z * [new branch] gh/ZhiweiYan-96/64/head -> origin/gh/ZhiweiYan-96/64/head 2025-08-14T21:25:13.8608526Z * [new branch] gh/ZhiweiYan-96/64/orig -> origin/gh/ZhiweiYan-96/64/orig 2025-08-14T21:25:13.8609112Z * [new branch] gh/ZhiweiYan-96/65/base -> origin/gh/ZhiweiYan-96/65/base 2025-08-14T21:25:13.8609482Z * [new branch] gh/ZhiweiYan-96/65/head -> origin/gh/ZhiweiYan-96/65/head 2025-08-14T21:25:13.8610037Z * [new branch] gh/ZhiweiYan-96/65/orig -> origin/gh/ZhiweiYan-96/65/orig 2025-08-14T21:25:13.8610411Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-08-14T21:25:13.8610775Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-08-14T21:25:13.8611139Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-08-14T21:25:13.8611506Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-08-14T21:25:13.8611851Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-08-14T21:25:13.8612349Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-08-14T21:25:13.8612701Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-08-14T21:25:13.8613958Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-08-14T21:25:13.8614335Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-08-14T21:25:13.8618872Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-08-14T21:25:13.8619301Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-08-14T21:25:13.8619670Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-08-14T21:25:13.8620006Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-08-14T21:25:13.8620342Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-08-14T21:25:13.8620827Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-08-14T21:25:13.8621554Z * [new branch] gh/alexbrauckmann/paddedtensor_init -> origin/gh/alexbrauckmann/paddedtensor_init 2025-08-14T21:25:13.8622083Z * [new branch] gh/alexbrauckmann/paddedtensor_meta_init -> origin/gh/alexbrauckmann/paddedtensor_meta_init 2025-08-14T21:25:13.8622734Z * [new branch] gh/alexsamardzic/7/base -> origin/gh/alexsamardzic/7/base 2025-08-14T21:25:13.8623109Z * [new branch] gh/alexsamardzic/7/head -> origin/gh/alexsamardzic/7/head 2025-08-14T21:25:13.8623467Z * [new branch] gh/alexsamardzic/7/orig -> origin/gh/alexsamardzic/7/orig 2025-08-14T21:25:13.8623994Z * [new branch] gh/alexsamardzic/8/base -> origin/gh/alexsamardzic/8/base 2025-08-14T21:25:13.8624370Z * [new branch] gh/alexsamardzic/8/head -> origin/gh/alexsamardzic/8/head 2025-08-14T21:25:13.8624951Z * [new branch] gh/alexsamardzic/8/orig -> origin/gh/alexsamardzic/8/orig 2025-08-14T21:25:13.8626934Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-08-14T21:25:13.8627542Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-08-14T21:25:13.8628048Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-08-14T21:25:13.8628795Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-08-14T21:25:13.8629678Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-08-14T21:25:13.8630270Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-08-14T21:25:13.8631685Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-08-14T21:25:13.8632598Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-08-14T21:25:13.8633173Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-08-14T21:25:13.8634358Z * [new branch] gh/andyanwang/1/base -> origin/gh/andyanwang/1/base 2025-08-14T21:25:13.8635347Z * [new branch] gh/andyanwang/1/head -> origin/gh/andyanwang/1/head 2025-08-14T21:25:13.8635808Z * [new branch] gh/andyanwang/1/orig -> origin/gh/andyanwang/1/orig 2025-08-14T21:25:13.8636557Z * [new branch] gh/andyanwang/13/base -> origin/gh/andyanwang/13/base 2025-08-14T21:25:13.8637303Z * [new branch] gh/andyanwang/13/head -> origin/gh/andyanwang/13/head 2025-08-14T21:25:13.8638024Z * [new branch] gh/andyanwang/13/orig -> origin/gh/andyanwang/13/orig 2025-08-14T21:25:13.8641796Z * [new branch] gh/andyanwang/2/base -> origin/gh/andyanwang/2/base 2025-08-14T21:25:13.8642207Z * [new branch] gh/andyanwang/2/head -> origin/gh/andyanwang/2/head 2025-08-14T21:25:13.8642545Z * [new branch] gh/andyanwang/2/orig -> origin/gh/andyanwang/2/orig 2025-08-14T21:25:13.8642884Z * [new branch] gh/andyanwang/28/base -> origin/gh/andyanwang/28/base 2025-08-14T21:25:13.8643257Z * [new branch] gh/andyanwang/28/head -> origin/gh/andyanwang/28/head 2025-08-14T21:25:13.8644126Z * [new branch] gh/andyanwang/28/orig -> origin/gh/andyanwang/28/orig 2025-08-14T21:25:13.8645580Z * [new branch] gh/andyanwang/3/base -> origin/gh/andyanwang/3/base 2025-08-14T21:25:13.8646107Z * [new branch] gh/andyanwang/3/head -> origin/gh/andyanwang/3/head 2025-08-14T21:25:13.8646612Z * [new branch] gh/andyanwang/3/orig -> origin/gh/andyanwang/3/orig 2025-08-14T21:25:13.8647118Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-08-14T21:25:13.8648055Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-08-14T21:25:13.8648487Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-08-14T21:25:13.8648870Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-08-14T21:25:13.8649654Z * [new branch] gh/andyanwang/32/base -> origin/gh/andyanwang/32/base 2025-08-14T21:25:13.8650284Z * [new branch] gh/andyanwang/32/head -> origin/gh/andyanwang/32/head 2025-08-14T21:25:13.8651488Z * [new branch] gh/andyanwang/32/orig -> origin/gh/andyanwang/32/orig 2025-08-14T21:25:13.8652022Z * [new branch] gh/andyanwang/33/base -> origin/gh/andyanwang/33/base 2025-08-14T21:25:13.8652677Z * [new branch] gh/andyanwang/33/head -> origin/gh/andyanwang/33/head 2025-08-14T21:25:13.8654291Z * [new branch] gh/andyanwang/33/orig -> origin/gh/andyanwang/33/orig 2025-08-14T21:25:13.8654898Z * [new branch] gh/andyanwang/34/base -> origin/gh/andyanwang/34/base 2025-08-14T21:25:13.8660117Z * [new branch] gh/andyanwang/34/head -> origin/gh/andyanwang/34/head 2025-08-14T21:25:13.8660687Z * [new branch] gh/andyanwang/34/orig -> origin/gh/andyanwang/34/orig 2025-08-14T21:25:13.8661193Z * [new branch] gh/andyanwang/35/base -> origin/gh/andyanwang/35/base 2025-08-14T21:25:13.8661656Z * [new branch] gh/andyanwang/35/head -> origin/gh/andyanwang/35/head 2025-08-14T21:25:13.8662543Z * [new branch] gh/andyanwang/35/orig -> origin/gh/andyanwang/35/orig 2025-08-14T21:25:13.8662974Z * [new branch] gh/andyanwang/36/base -> origin/gh/andyanwang/36/base 2025-08-14T21:25:13.8663330Z * [new branch] gh/andyanwang/36/head -> origin/gh/andyanwang/36/head 2025-08-14T21:25:13.8663690Z * [new branch] gh/andyanwang/36/orig -> origin/gh/andyanwang/36/orig 2025-08-14T21:25:13.8666667Z * [new branch] gh/andyanwang/37/base -> origin/gh/andyanwang/37/base 2025-08-14T21:25:13.8667042Z * [new branch] gh/andyanwang/37/head -> origin/gh/andyanwang/37/head 2025-08-14T21:25:13.8667577Z * [new branch] gh/andyanwang/37/orig -> origin/gh/andyanwang/37/orig 2025-08-14T21:25:13.8667942Z * [new branch] gh/andyanwang/38/base -> origin/gh/andyanwang/38/base 2025-08-14T21:25:13.8668290Z * [new branch] gh/andyanwang/38/head -> origin/gh/andyanwang/38/head 2025-08-14T21:25:13.8668635Z * [new branch] gh/andyanwang/38/orig -> origin/gh/andyanwang/38/orig 2025-08-14T21:25:13.8668976Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-08-14T21:25:13.8669330Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-08-14T21:25:13.8669684Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-08-14T21:25:13.8671123Z * [new branch] gh/andyanwang/4/base -> origin/gh/andyanwang/4/base 2025-08-14T21:25:13.8671497Z * [new branch] gh/andyanwang/4/head -> origin/gh/andyanwang/4/head 2025-08-14T21:25:13.8671857Z * [new branch] gh/andyanwang/4/orig -> origin/gh/andyanwang/4/orig 2025-08-14T21:25:13.8672211Z * [new branch] gh/andyanwang/40/base -> origin/gh/andyanwang/40/base 2025-08-14T21:25:13.8672563Z * [new branch] gh/andyanwang/40/head -> origin/gh/andyanwang/40/head 2025-08-14T21:25:13.8672918Z * [new branch] gh/andyanwang/40/orig -> origin/gh/andyanwang/40/orig 2025-08-14T21:25:13.8673771Z * [new branch] gh/angelayi/102/base -> origin/gh/angelayi/102/base 2025-08-14T21:25:13.8674126Z * [new branch] gh/angelayi/102/head -> origin/gh/angelayi/102/head 2025-08-14T21:25:13.8674480Z * [new branch] gh/angelayi/102/orig -> origin/gh/angelayi/102/orig 2025-08-14T21:25:13.8675180Z * [new branch] gh/angelayi/103/base -> origin/gh/angelayi/103/base 2025-08-14T21:25:13.8675846Z * [new branch] gh/angelayi/103/head -> origin/gh/angelayi/103/head 2025-08-14T21:25:13.8676538Z * [new branch] gh/angelayi/103/orig -> origin/gh/angelayi/103/orig 2025-08-14T21:25:13.8677866Z * [new branch] gh/angelayi/104/base -> origin/gh/angelayi/104/base 2025-08-14T21:25:13.8678368Z * [new branch] gh/angelayi/104/head -> origin/gh/angelayi/104/head 2025-08-14T21:25:13.8679498Z * [new branch] gh/angelayi/104/orig -> origin/gh/angelayi/104/orig 2025-08-14T21:25:13.8679869Z * [new branch] gh/angelayi/105/base -> origin/gh/angelayi/105/base 2025-08-14T21:25:13.8680564Z * [new branch] gh/angelayi/105/head -> origin/gh/angelayi/105/head 2025-08-14T21:25:13.8681090Z * [new branch] gh/angelayi/105/orig -> origin/gh/angelayi/105/orig 2025-08-14T21:25:13.8682356Z * [new branch] gh/angelayi/106/base -> origin/gh/angelayi/106/base 2025-08-14T21:25:13.8682855Z * [new branch] gh/angelayi/106/head -> origin/gh/angelayi/106/head 2025-08-14T21:25:13.8683380Z * [new branch] gh/angelayi/106/orig -> origin/gh/angelayi/106/orig 2025-08-14T21:25:13.8684592Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-08-14T21:25:13.8685143Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-08-14T21:25:13.8686070Z * [new branch] gh/angelayi/108/base -> origin/gh/angelayi/108/base 2025-08-14T21:25:13.8686664Z * [new branch] gh/angelayi/108/head -> origin/gh/angelayi/108/head 2025-08-14T21:25:13.8687359Z * [new branch] gh/angelayi/108/orig -> origin/gh/angelayi/108/orig 2025-08-14T21:25:13.8690404Z * [new branch] gh/angelayi/109/base -> origin/gh/angelayi/109/base 2025-08-14T21:25:13.8690733Z * [new branch] gh/angelayi/109/head -> origin/gh/angelayi/109/head 2025-08-14T21:25:13.8691135Z * [new branch] gh/angelayi/109/orig -> origin/gh/angelayi/109/orig 2025-08-14T21:25:13.8691469Z * [new branch] gh/angelayi/110/base -> origin/gh/angelayi/110/base 2025-08-14T21:25:13.8691826Z * [new branch] gh/angelayi/110/head -> origin/gh/angelayi/110/head 2025-08-14T21:25:13.8692149Z * [new branch] gh/angelayi/110/orig -> origin/gh/angelayi/110/orig 2025-08-14T21:25:13.8692856Z * [new branch] gh/angelayi/97/base -> origin/gh/angelayi/97/base 2025-08-14T21:25:13.8693479Z * [new branch] gh/angelayi/97/head -> origin/gh/angelayi/97/head 2025-08-14T21:25:13.8694101Z * [new branch] gh/angelayi/97/orig -> origin/gh/angelayi/97/orig 2025-08-14T21:25:13.8697728Z * [new branch] gh/ani300/1/base -> origin/gh/ani300/1/base 2025-08-14T21:25:13.8698517Z * [new branch] gh/ani300/1/head -> origin/gh/ani300/1/head 2025-08-14T21:25:13.8698982Z * [new branch] gh/ani300/1/orig -> origin/gh/ani300/1/orig 2025-08-14T21:25:13.8699384Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-08-14T21:25:13.8699818Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-08-14T21:25:13.8700214Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-08-14T21:25:13.8700618Z * [new branch] gh/anijain2305/766/base -> origin/gh/anijain2305/766/base 2025-08-14T21:25:13.8701020Z * [new branch] gh/anijain2305/766/head -> origin/gh/anijain2305/766/head 2025-08-14T21:25:13.8701397Z * [new branch] gh/anijain2305/766/orig -> origin/gh/anijain2305/766/orig 2025-08-14T21:25:13.8702244Z * [new branch] gh/anijain2305/790/base -> origin/gh/anijain2305/790/base 2025-08-14T21:25:13.8702655Z * [new branch] gh/anijain2305/790/head -> origin/gh/anijain2305/790/head 2025-08-14T21:25:13.8703624Z * [new branch] gh/anijain2305/790/orig -> origin/gh/anijain2305/790/orig 2025-08-14T21:25:13.8704442Z * [new branch] gh/anijain2305/792/base -> origin/gh/anijain2305/792/base 2025-08-14T21:25:13.8705841Z * [new branch] gh/anijain2305/792/head -> origin/gh/anijain2305/792/head 2025-08-14T21:25:13.8706218Z * [new branch] gh/anijain2305/792/orig -> origin/gh/anijain2305/792/orig 2025-08-14T21:25:13.8707514Z * [new branch] gh/anijain2305/803/base -> origin/gh/anijain2305/803/base 2025-08-14T21:25:13.8707884Z * [new branch] gh/anijain2305/803/head -> origin/gh/anijain2305/803/head 2025-08-14T21:25:13.8708465Z * [new branch] gh/anijain2305/803/orig -> origin/gh/anijain2305/803/orig 2025-08-14T21:25:13.8709803Z * [new branch] gh/anijain2305/804/base -> origin/gh/anijain2305/804/base 2025-08-14T21:25:13.8710341Z * [new branch] gh/anijain2305/804/head -> origin/gh/anijain2305/804/head 2025-08-14T21:25:13.8711029Z * [new branch] gh/anijain2305/804/orig -> origin/gh/anijain2305/804/orig 2025-08-14T21:25:13.8712211Z * [new branch] gh/anijain2305/805/base -> origin/gh/anijain2305/805/base 2025-08-14T21:25:13.8712589Z * [new branch] gh/anijain2305/805/head -> origin/gh/anijain2305/805/head 2025-08-14T21:25:13.8713270Z * [new branch] gh/anijain2305/805/orig -> origin/gh/anijain2305/805/orig 2025-08-14T21:25:13.8714562Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-08-14T21:25:13.8714940Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-08-14T21:25:13.8715687Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-08-14T21:25:13.8720241Z * [new branch] gh/anijain2305/811/base -> origin/gh/anijain2305/811/base 2025-08-14T21:25:13.8720624Z * [new branch] gh/anijain2305/811/head -> origin/gh/anijain2305/811/head 2025-08-14T21:25:13.8720997Z * [new branch] gh/anijain2305/811/orig -> origin/gh/anijain2305/811/orig 2025-08-14T21:25:13.8721378Z * [new branch] gh/anijain2305/812/base -> origin/gh/anijain2305/812/base 2025-08-14T21:25:13.8721747Z * [new branch] gh/anijain2305/812/head -> origin/gh/anijain2305/812/head 2025-08-14T21:25:13.8722128Z * [new branch] gh/anijain2305/812/orig -> origin/gh/anijain2305/812/orig 2025-08-14T21:25:13.8725246Z * [new branch] gh/anijain2305/813/base -> origin/gh/anijain2305/813/base 2025-08-14T21:25:13.8725832Z * [new branch] gh/anijain2305/813/head -> origin/gh/anijain2305/813/head 2025-08-14T21:25:13.8726300Z * [new branch] gh/anijain2305/813/orig -> origin/gh/anijain2305/813/orig 2025-08-14T21:25:13.8730709Z * [new branch] gh/anijain2305/814/base -> origin/gh/anijain2305/814/base 2025-08-14T21:25:13.8736541Z * [new branch] gh/anijain2305/814/head -> origin/gh/anijain2305/814/head 2025-08-14T21:25:13.8738314Z * [new branch] gh/anijain2305/814/orig -> origin/gh/anijain2305/814/orig 2025-08-14T21:25:13.8738679Z * [new branch] gh/anijain2305/815/base -> origin/gh/anijain2305/815/base 2025-08-14T21:25:13.8739016Z * [new branch] gh/anijain2305/815/head -> origin/gh/anijain2305/815/head 2025-08-14T21:25:13.8739339Z * [new branch] gh/anijain2305/815/orig -> origin/gh/anijain2305/815/orig 2025-08-14T21:25:13.8739668Z * [new branch] gh/anijain2305/816/base -> origin/gh/anijain2305/816/base 2025-08-14T21:25:13.8739988Z * [new branch] gh/anijain2305/816/head -> origin/gh/anijain2305/816/head 2025-08-14T21:25:13.8740329Z * [new branch] gh/anijain2305/817/base -> origin/gh/anijain2305/817/base 2025-08-14T21:25:13.8740865Z * [new branch] gh/anijain2305/817/head -> origin/gh/anijain2305/817/head 2025-08-14T21:25:13.8741192Z * [new branch] gh/anijain2305/817/orig -> origin/gh/anijain2305/817/orig 2025-08-14T21:25:13.8741516Z * [new branch] gh/anijain2305/818/base -> origin/gh/anijain2305/818/base 2025-08-14T21:25:13.8741651Z * [new branch] gh/anijain2305/818/head -> origin/gh/anijain2305/818/head 2025-08-14T21:25:13.8741790Z * [new branch] gh/anijain2305/818/orig -> origin/gh/anijain2305/818/orig 2025-08-14T21:25:13.8741922Z * [new branch] gh/anijain2305/819/base -> origin/gh/anijain2305/819/base 2025-08-14T21:25:13.8742058Z * [new branch] gh/anijain2305/819/head -> origin/gh/anijain2305/819/head 2025-08-14T21:25:13.8742205Z * [new branch] gh/anijain2305/819/orig -> origin/gh/anijain2305/819/orig 2025-08-14T21:25:13.8742348Z * [new branch] gh/anijain2305/820/base -> origin/gh/anijain2305/820/base 2025-08-14T21:25:13.8742494Z * [new branch] gh/anijain2305/820/head -> origin/gh/anijain2305/820/head 2025-08-14T21:25:13.8742629Z * [new branch] gh/anijain2305/820/orig -> origin/gh/anijain2305/820/orig 2025-08-14T21:25:13.8742791Z * [new branch] gh/anijain2305/821/base -> origin/gh/anijain2305/821/base 2025-08-14T21:25:13.8743093Z * [new branch] gh/anijain2305/821/head -> origin/gh/anijain2305/821/head 2025-08-14T21:25:13.8743233Z * [new branch] gh/anijain2305/821/orig -> origin/gh/anijain2305/821/orig 2025-08-14T21:25:13.8743378Z * [new branch] gh/anijain2305/822/base -> origin/gh/anijain2305/822/base 2025-08-14T21:25:13.8743521Z * [new branch] gh/anijain2305/822/head -> origin/gh/anijain2305/822/head 2025-08-14T21:25:13.8743710Z * [new branch] gh/anijain2305/822/orig -> origin/gh/anijain2305/822/orig 2025-08-14T21:25:13.8743861Z * [new branch] gh/anijain2305/823/base -> origin/gh/anijain2305/823/base 2025-08-14T21:25:13.8746319Z * [new branch] gh/anijain2305/823/head -> origin/gh/anijain2305/823/head 2025-08-14T21:25:13.8746487Z * [new branch] gh/anijain2305/823/orig -> origin/gh/anijain2305/823/orig 2025-08-14T21:25:13.8746643Z * [new branch] gh/anijain2305/824/base -> origin/gh/anijain2305/824/base 2025-08-14T21:25:13.8746801Z * [new branch] gh/anijain2305/824/head -> origin/gh/anijain2305/824/head 2025-08-14T21:25:13.8749815Z * [new branch] gh/anijain2305/824/orig -> origin/gh/anijain2305/824/orig 2025-08-14T21:25:13.8749959Z * [new branch] gh/anijain2305/825/base -> origin/gh/anijain2305/825/base 2025-08-14T21:25:13.8750108Z * [new branch] gh/anijain2305/825/head -> origin/gh/anijain2305/825/head 2025-08-14T21:25:13.8750256Z * [new branch] gh/anijain2305/825/orig -> origin/gh/anijain2305/825/orig 2025-08-14T21:25:13.8750845Z * [new branch] gh/anijain2305/826/base -> origin/gh/anijain2305/826/base 2025-08-14T21:25:13.8751715Z * [new branch] gh/anijain2305/826/head -> origin/gh/anijain2305/826/head 2025-08-14T21:25:13.8752143Z * [new branch] gh/anijain2305/826/orig -> origin/gh/anijain2305/826/orig 2025-08-14T21:25:13.8753142Z * [new branch] gh/anijain2305/827/base -> origin/gh/anijain2305/827/base 2025-08-14T21:25:13.8753436Z * [new branch] gh/anijain2305/827/head -> origin/gh/anijain2305/827/head 2025-08-14T21:25:13.8755156Z * [new branch] gh/anijain2305/827/orig -> origin/gh/anijain2305/827/orig 2025-08-14T21:25:13.8755351Z * [new branch] gh/anijain2305/828/base -> origin/gh/anijain2305/828/base 2025-08-14T21:25:13.8755943Z * [new branch] gh/anijain2305/828/head -> origin/gh/anijain2305/828/head 2025-08-14T21:25:13.8757527Z * [new branch] gh/anijain2305/828/orig -> origin/gh/anijain2305/828/orig 2025-08-14T21:25:13.8757682Z * [new branch] gh/anijain2305/829/base -> origin/gh/anijain2305/829/base 2025-08-14T21:25:13.8758430Z * [new branch] gh/anijain2305/829/head -> origin/gh/anijain2305/829/head 2025-08-14T21:25:13.8759132Z * [new branch] gh/anijain2305/829/orig -> origin/gh/anijain2305/829/orig 2025-08-14T21:25:13.8764650Z * [new branch] gh/anijain2305/830/base -> origin/gh/anijain2305/830/base 2025-08-14T21:25:13.8764865Z * [new branch] gh/anijain2305/830/head -> origin/gh/anijain2305/830/head 2025-08-14T21:25:13.8765003Z * [new branch] gh/anijain2305/830/orig -> origin/gh/anijain2305/830/orig 2025-08-14T21:25:13.8765133Z * [new branch] gh/anijain2305/831/base -> origin/gh/anijain2305/831/base 2025-08-14T21:25:13.8765294Z * [new branch] gh/anijain2305/831/head -> origin/gh/anijain2305/831/head 2025-08-14T21:25:13.8765436Z * [new branch] gh/anijain2305/831/orig -> origin/gh/anijain2305/831/orig 2025-08-14T21:25:13.8765579Z * [new branch] gh/anijain2305/832/base -> origin/gh/anijain2305/832/base 2025-08-14T21:25:13.8765716Z * [new branch] gh/anijain2305/832/head -> origin/gh/anijain2305/832/head 2025-08-14T21:25:13.8765849Z * [new branch] gh/anijain2305/832/orig -> origin/gh/anijain2305/832/orig 2025-08-14T21:25:13.8770797Z * [new branch] gh/anijain2305/833/base -> origin/gh/anijain2305/833/base 2025-08-14T21:25:13.8770979Z * [new branch] gh/anijain2305/833/head -> origin/gh/anijain2305/833/head 2025-08-14T21:25:13.8771124Z * [new branch] gh/anijain2305/833/orig -> origin/gh/anijain2305/833/orig 2025-08-14T21:25:13.8771425Z * [new branch] gh/anijain2305/834/base -> origin/gh/anijain2305/834/base 2025-08-14T21:25:13.8771569Z * [new branch] gh/anijain2305/834/head -> origin/gh/anijain2305/834/head 2025-08-14T21:25:13.8771711Z * [new branch] gh/anijain2305/834/orig -> origin/gh/anijain2305/834/orig 2025-08-14T21:25:13.8771841Z * [new branch] gh/anijain2305/835/base -> origin/gh/anijain2305/835/base 2025-08-14T21:25:13.8771982Z * [new branch] gh/anijain2305/835/head -> origin/gh/anijain2305/835/head 2025-08-14T21:25:13.8772120Z * [new branch] gh/anijain2305/835/orig -> origin/gh/anijain2305/835/orig 2025-08-14T21:25:13.8776172Z * [new branch] gh/anijain2305/836/base -> origin/gh/anijain2305/836/base 2025-08-14T21:25:13.8776365Z * [new branch] gh/anijain2305/836/head -> origin/gh/anijain2305/836/head 2025-08-14T21:25:13.8776509Z * [new branch] gh/anijain2305/836/orig -> origin/gh/anijain2305/836/orig 2025-08-14T21:25:13.8776664Z * [new branch] gh/anijain2305/837/base -> origin/gh/anijain2305/837/base 2025-08-14T21:25:13.8776809Z * [new branch] gh/anijain2305/837/head -> origin/gh/anijain2305/837/head 2025-08-14T21:25:13.8776967Z * [new branch] gh/anijain2305/837/orig -> origin/gh/anijain2305/837/orig 2025-08-14T21:25:13.8777524Z * [new branch] gh/anijain2305/838/base -> origin/gh/anijain2305/838/base 2025-08-14T21:25:13.8778060Z * [new branch] gh/anijain2305/838/head -> origin/gh/anijain2305/838/head 2025-08-14T21:25:13.8778760Z * [new branch] gh/anijain2305/838/orig -> origin/gh/anijain2305/838/orig 2025-08-14T21:25:13.8780024Z * [new branch] gh/anijain2305/839/base -> origin/gh/anijain2305/839/base 2025-08-14T21:25:13.8780181Z * [new branch] gh/anijain2305/839/head -> origin/gh/anijain2305/839/head 2025-08-14T21:25:13.8780781Z * [new branch] gh/anijain2305/839/orig -> origin/gh/anijain2305/839/orig 2025-08-14T21:25:13.8781947Z * [new branch] gh/anijain2305/840/base -> origin/gh/anijain2305/840/base 2025-08-14T21:25:13.8782531Z * [new branch] gh/anijain2305/840/head -> origin/gh/anijain2305/840/head 2025-08-14T21:25:13.8783460Z * [new branch] gh/anijain2305/840/orig -> origin/gh/anijain2305/840/orig 2025-08-14T21:25:13.8784360Z * [new branch] gh/anijain2305/841/base -> origin/gh/anijain2305/841/base 2025-08-14T21:25:13.8784782Z * [new branch] gh/anijain2305/841/head -> origin/gh/anijain2305/841/head 2025-08-14T21:25:13.8785719Z * [new branch] gh/anijain2305/841/orig -> origin/gh/anijain2305/841/orig 2025-08-14T21:25:13.8786554Z * [new branch] gh/anijain2305/842/base -> origin/gh/anijain2305/842/base 2025-08-14T21:25:13.8786917Z * [new branch] gh/anijain2305/842/head -> origin/gh/anijain2305/842/head 2025-08-14T21:25:13.8787796Z * [new branch] gh/anijain2305/842/orig -> origin/gh/anijain2305/842/orig 2025-08-14T21:25:13.8788656Z * [new branch] gh/anijain2305/843/base -> origin/gh/anijain2305/843/base 2025-08-14T21:25:13.8788939Z * [new branch] gh/anijain2305/843/head -> origin/gh/anijain2305/843/head 2025-08-14T21:25:13.8789864Z * [new branch] gh/anijain2305/843/orig -> origin/gh/anijain2305/843/orig 2025-08-14T21:25:13.8790819Z * [new branch] gh/anijain2305/844/base -> origin/gh/anijain2305/844/base 2025-08-14T21:25:13.8791065Z * [new branch] gh/anijain2305/844/head -> origin/gh/anijain2305/844/head 2025-08-14T21:25:13.8792129Z * [new branch] gh/anijain2305/844/orig -> origin/gh/anijain2305/844/orig 2025-08-14T21:25:13.8793145Z * [new branch] gh/anijain2305/845/base -> origin/gh/anijain2305/845/base 2025-08-14T21:25:13.8793352Z * [new branch] gh/anijain2305/845/head -> origin/gh/anijain2305/845/head 2025-08-14T21:25:13.8794495Z * [new branch] gh/anijain2305/845/orig -> origin/gh/anijain2305/845/orig 2025-08-14T21:25:13.8795493Z * [new branch] gh/anijain2305/846/base -> origin/gh/anijain2305/846/base 2025-08-14T21:25:13.8795818Z * [new branch] gh/anijain2305/846/head -> origin/gh/anijain2305/846/head 2025-08-14T21:25:13.8796639Z * [new branch] gh/anijain2305/846/orig -> origin/gh/anijain2305/846/orig 2025-08-14T21:25:13.8798012Z * [new branch] gh/anijain2305/847/base -> origin/gh/anijain2305/847/base 2025-08-14T21:25:13.8798375Z * [new branch] gh/anijain2305/847/head -> origin/gh/anijain2305/847/head 2025-08-14T21:25:13.8799314Z * [new branch] gh/anijain2305/847/orig -> origin/gh/anijain2305/847/orig 2025-08-14T21:25:13.8802545Z * [new branch] gh/anijain2305/848/base -> origin/gh/anijain2305/848/base 2025-08-14T21:25:13.8802706Z * [new branch] gh/anijain2305/848/head -> origin/gh/anijain2305/848/head 2025-08-14T21:25:13.8802855Z * [new branch] gh/anijain2305/848/orig -> origin/gh/anijain2305/848/orig 2025-08-14T21:25:13.8803006Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-08-14T21:25:13.8803159Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-08-14T21:25:13.8804016Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-08-14T21:25:13.8805225Z * [new branch] gh/ankitageorge/10/base -> origin/gh/ankitageorge/10/base 2025-08-14T21:25:13.8805642Z * [new branch] gh/ankitageorge/10/head -> origin/gh/ankitageorge/10/head 2025-08-14T21:25:13.8806732Z * [new branch] gh/ankitageorge/10/orig -> origin/gh/ankitageorge/10/orig 2025-08-14T21:25:13.8807257Z * [new branch] gh/ankitageorge/12/base -> origin/gh/ankitageorge/12/base 2025-08-14T21:25:13.8808231Z * [new branch] gh/ankitageorge/12/head -> origin/gh/ankitageorge/12/head 2025-08-14T21:25:13.8808428Z * [new branch] gh/ankitageorge/12/orig -> origin/gh/ankitageorge/12/orig 2025-08-14T21:25:13.8811721Z * [new branch] gh/ankitageorge/13/base -> origin/gh/ankitageorge/13/base 2025-08-14T21:25:13.8816648Z * [new branch] gh/ankitageorge/13/head -> origin/gh/ankitageorge/13/head 2025-08-14T21:25:13.8816827Z * [new branch] gh/ankitageorge/13/orig -> origin/gh/ankitageorge/13/orig 2025-08-14T21:25:13.8816976Z * [new branch] gh/ankitageorge/14/base -> origin/gh/ankitageorge/14/base 2025-08-14T21:25:13.8817114Z * [new branch] gh/ankitageorge/14/head -> origin/gh/ankitageorge/14/head 2025-08-14T21:25:13.8817262Z * [new branch] gh/ankitageorge/14/orig -> origin/gh/ankitageorge/14/orig 2025-08-14T21:25:13.8817445Z * [new branch] gh/ankitageorge/15/base -> origin/gh/ankitageorge/15/base 2025-08-14T21:25:13.8818331Z * [new branch] gh/ankitageorge/15/head -> origin/gh/ankitageorge/15/head 2025-08-14T21:25:13.8818908Z * [new branch] gh/ankitageorge/15/orig -> origin/gh/ankitageorge/15/orig 2025-08-14T21:25:13.8823587Z * [new branch] gh/ankitageorge/16/base -> origin/gh/ankitageorge/16/base 2025-08-14T21:25:13.8823779Z * [new branch] gh/ankitageorge/16/head -> origin/gh/ankitageorge/16/head 2025-08-14T21:25:13.8823921Z * [new branch] gh/ankitageorge/16/orig -> origin/gh/ankitageorge/16/orig 2025-08-14T21:25:13.8824055Z * [new branch] gh/ankitageorge/17/base -> origin/gh/ankitageorge/17/base 2025-08-14T21:25:13.8824201Z * [new branch] gh/ankitageorge/17/head -> origin/gh/ankitageorge/17/head 2025-08-14T21:25:13.8824337Z * [new branch] gh/ankitageorge/17/orig -> origin/gh/ankitageorge/17/orig 2025-08-14T21:25:13.8825154Z * [new branch] gh/ankitageorge/18/base -> origin/gh/ankitageorge/18/base 2025-08-14T21:25:13.8825675Z * [new branch] gh/ankitageorge/18/head -> origin/gh/ankitageorge/18/head 2025-08-14T21:25:13.8826296Z * [new branch] gh/ankitageorge/18/orig -> origin/gh/ankitageorge/18/orig 2025-08-14T21:25:13.8832498Z * [new branch] gh/ankitageorge/19/base -> origin/gh/ankitageorge/19/base 2025-08-14T21:25:13.8832683Z * [new branch] gh/ankitageorge/19/head -> origin/gh/ankitageorge/19/head 2025-08-14T21:25:13.8832837Z * [new branch] gh/ankitageorge/19/orig -> origin/gh/ankitageorge/19/orig 2025-08-14T21:25:13.8832978Z * [new branch] gh/ankitageorge/20/base -> origin/gh/ankitageorge/20/base 2025-08-14T21:25:13.8833137Z * [new branch] gh/ankitageorge/20/head -> origin/gh/ankitageorge/20/head 2025-08-14T21:25:13.8833302Z * [new branch] gh/ankitageorge/20/orig -> origin/gh/ankitageorge/20/orig 2025-08-14T21:25:13.8833454Z * [new branch] gh/ankitageorge/21/base -> origin/gh/ankitageorge/21/base 2025-08-14T21:25:13.8833610Z * [new branch] gh/ankitageorge/21/head -> origin/gh/ankitageorge/21/head 2025-08-14T21:25:13.8833757Z * [new branch] gh/ankitageorge/21/orig -> origin/gh/ankitageorge/21/orig 2025-08-14T21:25:13.8837648Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-08-14T21:25:13.8837951Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-08-14T21:25:13.8840932Z * [new branch] gh/anshul-si/10/base -> origin/gh/anshul-si/10/base 2025-08-14T21:25:13.8841248Z * [new branch] gh/anshul-si/10/head -> origin/gh/anshul-si/10/head 2025-08-14T21:25:13.8845815Z * [new branch] gh/anshul-si/10/orig -> origin/gh/anshul-si/10/orig 2025-08-14T21:25:13.8852326Z * [new branch] gh/anshul-si/11/base -> origin/gh/anshul-si/11/base 2025-08-14T21:25:13.8852892Z * [new branch] gh/anshul-si/11/head -> origin/gh/anshul-si/11/head 2025-08-14T21:25:13.8853160Z * [new branch] gh/anshul-si/11/orig -> origin/gh/anshul-si/11/orig 2025-08-14T21:25:13.8853418Z * [new branch] gh/anshul-si/12/base -> origin/gh/anshul-si/12/base 2025-08-14T21:25:13.8854070Z * [new branch] gh/anshul-si/12/head -> origin/gh/anshul-si/12/head 2025-08-14T21:25:13.8854247Z * [new branch] gh/anshul-si/12/orig -> origin/gh/anshul-si/12/orig 2025-08-14T21:25:13.8854394Z * [new branch] gh/anshul-si/13/base -> origin/gh/anshul-si/13/base 2025-08-14T21:25:13.8854529Z * [new branch] gh/anshul-si/13/head -> origin/gh/anshul-si/13/head 2025-08-14T21:25:13.8854667Z * [new branch] gh/anshul-si/13/orig -> origin/gh/anshul-si/13/orig 2025-08-14T21:25:13.8854832Z * [new branch] gh/anshul-si/14/base -> origin/gh/anshul-si/14/base 2025-08-14T21:25:13.8854976Z * [new branch] gh/anshul-si/14/head -> origin/gh/anshul-si/14/head 2025-08-14T21:25:13.8855121Z * [new branch] gh/anshul-si/14/orig -> origin/gh/anshul-si/14/orig 2025-08-14T21:25:13.8855260Z * [new branch] gh/anshul-si/15/base -> origin/gh/anshul-si/15/base 2025-08-14T21:25:13.8855395Z * [new branch] gh/anshul-si/15/head -> origin/gh/anshul-si/15/head 2025-08-14T21:25:13.8861153Z * [new branch] gh/anshul-si/15/orig -> origin/gh/anshul-si/15/orig 2025-08-14T21:25:13.8865683Z * [new branch] gh/anshul-si/16/base -> origin/gh/anshul-si/16/base 2025-08-14T21:25:13.8865875Z * [new branch] gh/anshul-si/16/head -> origin/gh/anshul-si/16/head 2025-08-14T21:25:13.8866203Z * [new branch] gh/anshul-si/16/orig -> origin/gh/anshul-si/16/orig 2025-08-14T21:25:13.8866372Z * [new branch] gh/anshul-si/17/base -> origin/gh/anshul-si/17/base 2025-08-14T21:25:13.8866522Z * [new branch] gh/anshul-si/17/head -> origin/gh/anshul-si/17/head 2025-08-14T21:25:13.8866668Z * [new branch] gh/anshul-si/17/orig -> origin/gh/anshul-si/17/orig 2025-08-14T21:25:13.8866868Z * [new branch] gh/anshul-si/18/base -> origin/gh/anshul-si/18/base 2025-08-14T21:25:13.8867017Z * [new branch] gh/anshul-si/18/head -> origin/gh/anshul-si/18/head 2025-08-14T21:25:13.8867179Z * [new branch] gh/anshul-si/18/orig -> origin/gh/anshul-si/18/orig 2025-08-14T21:25:13.8867342Z * [new branch] gh/anshul-si/19/base -> origin/gh/anshul-si/19/base 2025-08-14T21:25:13.8867523Z * [new branch] gh/anshul-si/19/head -> origin/gh/anshul-si/19/head 2025-08-14T21:25:13.8867679Z * [new branch] gh/anshul-si/19/orig -> origin/gh/anshul-si/19/orig 2025-08-14T21:25:13.8867831Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-08-14T21:25:13.8867988Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-08-14T21:25:13.8868138Z * [new branch] gh/anshul-si/20/base -> origin/gh/anshul-si/20/base 2025-08-14T21:25:13.8868277Z * [new branch] gh/anshul-si/20/head -> origin/gh/anshul-si/20/head 2025-08-14T21:25:13.8868421Z * [new branch] gh/anshul-si/20/orig -> origin/gh/anshul-si/20/orig 2025-08-14T21:25:13.8868559Z * [new branch] gh/anshul-si/21/base -> origin/gh/anshul-si/21/base 2025-08-14T21:25:13.8868703Z * [new branch] gh/anshul-si/21/head -> origin/gh/anshul-si/21/head 2025-08-14T21:25:13.8868854Z * [new branch] gh/anshul-si/21/orig -> origin/gh/anshul-si/21/orig 2025-08-14T21:25:13.8869000Z * [new branch] gh/anshul-si/22/base -> origin/gh/anshul-si/22/base 2025-08-14T21:25:13.8869188Z * [new branch] gh/anshul-si/22/head -> origin/gh/anshul-si/22/head 2025-08-14T21:25:13.8869333Z * [new branch] gh/anshul-si/22/orig -> origin/gh/anshul-si/22/orig 2025-08-14T21:25:13.8869478Z * [new branch] gh/anshul-si/23/base -> origin/gh/anshul-si/23/base 2025-08-14T21:25:13.8869624Z * [new branch] gh/anshul-si/23/head -> origin/gh/anshul-si/23/head 2025-08-14T21:25:13.8869770Z * [new branch] gh/anshul-si/23/orig -> origin/gh/anshul-si/23/orig 2025-08-14T21:25:13.8869917Z * [new branch] gh/anshul-si/24/base -> origin/gh/anshul-si/24/base 2025-08-14T21:25:13.8870067Z * [new branch] gh/anshul-si/24/head -> origin/gh/anshul-si/24/head 2025-08-14T21:25:13.8870212Z * [new branch] gh/anshul-si/24/orig -> origin/gh/anshul-si/24/orig 2025-08-14T21:25:13.8870407Z * [new branch] gh/anshul-si/25/base -> origin/gh/anshul-si/25/base 2025-08-14T21:25:13.8874956Z * [new branch] gh/anshul-si/25/head -> origin/gh/anshul-si/25/head 2025-08-14T21:25:13.8875154Z * [new branch] gh/anshul-si/25/orig -> origin/gh/anshul-si/25/orig 2025-08-14T21:25:13.8875308Z * [new branch] gh/anshul-si/26/base -> origin/gh/anshul-si/26/base 2025-08-14T21:25:13.8875461Z * [new branch] gh/anshul-si/26/head -> origin/gh/anshul-si/26/head 2025-08-14T21:25:13.8875620Z * [new branch] gh/anshul-si/26/orig -> origin/gh/anshul-si/26/orig 2025-08-14T21:25:13.8875769Z * [new branch] gh/anshul-si/27/base -> origin/gh/anshul-si/27/base 2025-08-14T21:25:13.8875958Z * [new branch] gh/anshul-si/27/head -> origin/gh/anshul-si/27/head 2025-08-14T21:25:13.8876776Z * [new branch] gh/anshul-si/27/orig -> origin/gh/anshul-si/27/orig 2025-08-14T21:25:13.8882586Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-08-14T21:25:13.8882963Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-08-14T21:25:13.8883222Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-08-14T21:25:13.8883411Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-08-14T21:25:13.8884100Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-08-14T21:25:13.8884279Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-08-14T21:25:13.8884427Z * [new branch] gh/anshul-si/6/base -> origin/gh/anshul-si/6/base 2025-08-14T21:25:13.8884561Z * [new branch] gh/anshul-si/6/head -> origin/gh/anshul-si/6/head 2025-08-14T21:25:13.8884720Z * [new branch] gh/anshul-si/6/orig -> origin/gh/anshul-si/6/orig 2025-08-14T21:25:13.8885203Z * [new branch] gh/anshul-si/7/base -> origin/gh/anshul-si/7/base 2025-08-14T21:25:13.8886413Z * [new branch] gh/anshul-si/7/head -> origin/gh/anshul-si/7/head 2025-08-14T21:25:13.8886894Z * [new branch] gh/anshul-si/7/orig -> origin/gh/anshul-si/7/orig 2025-08-14T21:25:13.8888206Z * [new branch] gh/anshul-si/8/base -> origin/gh/anshul-si/8/base 2025-08-14T21:25:13.8888367Z * [new branch] gh/anshul-si/8/head -> origin/gh/anshul-si/8/head 2025-08-14T21:25:13.8889608Z * [new branch] gh/anshul-si/8/orig -> origin/gh/anshul-si/8/orig 2025-08-14T21:25:13.8890196Z * [new branch] gh/anshul-si/9/base -> origin/gh/anshul-si/9/base 2025-08-14T21:25:13.8891424Z * [new branch] gh/anshul-si/9/head -> origin/gh/anshul-si/9/head 2025-08-14T21:25:13.8891775Z * [new branch] gh/anshul-si/9/orig -> origin/gh/anshul-si/9/orig 2025-08-14T21:25:13.8893039Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-08-14T21:25:13.8893317Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-08-14T21:25:13.8894653Z * [new branch] gh/aorenste/235/base -> origin/gh/aorenste/235/base 2025-08-14T21:25:13.8894994Z * [new branch] gh/aorenste/235/head -> origin/gh/aorenste/235/head 2025-08-14T21:25:13.8896247Z * [new branch] gh/aorenste/235/orig -> origin/gh/aorenste/235/orig 2025-08-14T21:25:13.8897987Z * [new branch] gh/aorenste/236/base -> origin/gh/aorenste/236/base 2025-08-14T21:25:13.8898183Z * [new branch] gh/aorenste/236/head -> origin/gh/aorenste/236/head 2025-08-14T21:25:13.8898624Z * [new branch] gh/aorenste/236/orig -> origin/gh/aorenste/236/orig 2025-08-14T21:25:13.8899909Z * [new branch] gh/aorenste/237/base -> origin/gh/aorenste/237/base 2025-08-14T21:25:13.8900447Z * [new branch] gh/aorenste/237/head -> origin/gh/aorenste/237/head 2025-08-14T21:25:13.8901020Z * [new branch] gh/aorenste/237/orig -> origin/gh/aorenste/237/orig 2025-08-14T21:25:13.8902295Z * [new branch] gh/aorenste/238/base -> origin/gh/aorenste/238/base 2025-08-14T21:25:13.8902453Z * [new branch] gh/aorenste/238/head -> origin/gh/aorenste/238/head 2025-08-14T21:25:13.8903587Z * [new branch] gh/aorenste/238/orig -> origin/gh/aorenste/238/orig 2025-08-14T21:25:13.8904809Z * [new branch] gh/bdhirsh/650/base -> origin/gh/bdhirsh/650/base 2025-08-14T21:25:13.8905427Z * [new branch] gh/bdhirsh/650/head -> origin/gh/bdhirsh/650/head 2025-08-14T21:25:13.8906286Z * [new branch] gh/bdhirsh/650/orig -> origin/gh/bdhirsh/650/orig 2025-08-14T21:25:13.8908002Z * [new branch] gh/bdhirsh/656/base -> origin/gh/bdhirsh/656/base 2025-08-14T21:25:13.8908193Z * [new branch] gh/bdhirsh/656/head -> origin/gh/bdhirsh/656/head 2025-08-14T21:25:13.8909039Z * [new branch] gh/bdhirsh/657/base -> origin/gh/bdhirsh/657/base 2025-08-14T21:25:13.8909937Z * [new branch] gh/bdhirsh/657/head -> origin/gh/bdhirsh/657/head 2025-08-14T21:25:13.8910492Z * [new branch] gh/bdhirsh/659/base -> origin/gh/bdhirsh/659/base 2025-08-14T21:25:13.8912114Z * [new branch] gh/bdhirsh/659/head -> origin/gh/bdhirsh/659/head 2025-08-14T21:25:13.8912307Z * [new branch] gh/bdhirsh/659/orig -> origin/gh/bdhirsh/659/orig 2025-08-14T21:25:13.8912867Z * [new branch] gh/bdhirsh/663/base -> origin/gh/bdhirsh/663/base 2025-08-14T21:25:13.8913495Z * [new branch] gh/bdhirsh/663/head -> origin/gh/bdhirsh/663/head 2025-08-14T21:25:13.8914240Z * [new branch] gh/bdhirsh/663/orig -> origin/gh/bdhirsh/663/orig 2025-08-14T21:25:13.8915523Z * [new branch] gh/bdhirsh/665/base -> origin/gh/bdhirsh/665/base 2025-08-14T21:25:13.8915911Z * [new branch] gh/bdhirsh/665/head -> origin/gh/bdhirsh/665/head 2025-08-14T21:25:13.8916829Z * [new branch] gh/bdhirsh/665/orig -> origin/gh/bdhirsh/665/orig 2025-08-14T21:25:13.8918100Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-08-14T21:25:13.8918415Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-08-14T21:25:13.8919497Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-08-14T21:25:13.8920604Z * [new branch] gh/benjaminglass1/79/base -> origin/gh/benjaminglass1/79/base 2025-08-14T21:25:13.8920986Z * [new branch] gh/benjaminglass1/79/head -> origin/gh/benjaminglass1/79/head 2025-08-14T21:25:13.8921765Z * [new branch] gh/benjaminglass1/79/orig -> origin/gh/benjaminglass1/79/orig 2025-08-14T21:25:13.8923196Z * [new branch] gh/benjaminglass1/86/base -> origin/gh/benjaminglass1/86/base 2025-08-14T21:25:13.8923409Z * [new branch] gh/benjaminglass1/86/head -> origin/gh/benjaminglass1/86/head 2025-08-14T21:25:13.8923801Z * [new branch] gh/benjaminglass1/86/orig -> origin/gh/benjaminglass1/86/orig 2025-08-14T21:25:13.8925868Z * [new branch] gh/benjaminglass1/89/base -> origin/gh/benjaminglass1/89/base 2025-08-14T21:25:13.8926089Z * [new branch] gh/benjaminglass1/89/head -> origin/gh/benjaminglass1/89/head 2025-08-14T21:25:13.8926258Z * [new branch] gh/benjaminglass1/89/orig -> origin/gh/benjaminglass1/89/orig 2025-08-14T21:25:13.8931358Z * [new branch] gh/benjaminglass1/91/base -> origin/gh/benjaminglass1/91/base 2025-08-14T21:25:13.8935604Z * [new branch] gh/benjaminglass1/91/head -> origin/gh/benjaminglass1/91/head 2025-08-14T21:25:13.8940096Z * [new branch] gh/benjaminglass1/91/orig -> origin/gh/benjaminglass1/91/orig 2025-08-14T21:25:13.8944407Z * [new branch] gh/benjaminglass1/93/base -> origin/gh/benjaminglass1/93/base 2025-08-14T21:25:13.8946380Z * [new branch] gh/benjaminglass1/93/head -> origin/gh/benjaminglass1/93/head 2025-08-14T21:25:13.8949930Z * [new branch] gh/benjaminglass1/93/orig -> origin/gh/benjaminglass1/93/orig 2025-08-14T21:25:13.8950125Z * [new branch] gh/benjaminglass1/94/base -> origin/gh/benjaminglass1/94/base 2025-08-14T21:25:13.8950477Z * [new branch] gh/benjaminglass1/94/head -> origin/gh/benjaminglass1/94/head 2025-08-14T21:25:13.8950734Z * [new branch] gh/benjaminglass1/94/orig -> origin/gh/benjaminglass1/94/orig 2025-08-14T21:25:13.8951247Z * [new branch] gh/benjaminglass1/95/base -> origin/gh/benjaminglass1/95/base 2025-08-14T21:25:13.8951446Z * [new branch] gh/benjaminglass1/95/head -> origin/gh/benjaminglass1/95/head 2025-08-14T21:25:13.8951634Z * [new branch] gh/benjaminglass1/95/orig -> origin/gh/benjaminglass1/95/orig 2025-08-14T21:25:13.8951807Z * [new branch] gh/benjaminglass1/96/base -> origin/gh/benjaminglass1/96/base 2025-08-14T21:25:13.8951994Z * [new branch] gh/benjaminglass1/96/head -> origin/gh/benjaminglass1/96/head 2025-08-14T21:25:13.8952159Z * [new branch] gh/benjaminglass1/96/orig -> origin/gh/benjaminglass1/96/orig 2025-08-14T21:25:13.8952322Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-08-14T21:25:13.8952478Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-08-14T21:25:13.8952658Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-08-14T21:25:13.8952820Z * [new branch] gh/benjaminglass1/98/base -> origin/gh/benjaminglass1/98/base 2025-08-14T21:25:13.8952987Z * [new branch] gh/benjaminglass1/98/head -> origin/gh/benjaminglass1/98/head 2025-08-14T21:25:13.8953155Z * [new branch] gh/benjaminglass1/98/orig -> origin/gh/benjaminglass1/98/orig 2025-08-14T21:25:13.8953316Z * [new branch] gh/bobrenjc93/478/base -> origin/gh/bobrenjc93/478/base 2025-08-14T21:25:13.8953475Z * [new branch] gh/bobrenjc93/478/head -> origin/gh/bobrenjc93/478/head 2025-08-14T21:25:13.8953631Z * [new branch] gh/bobrenjc93/478/orig -> origin/gh/bobrenjc93/478/orig 2025-08-14T21:25:13.8953788Z * [new branch] gh/bobrenjc93/514/base -> origin/gh/bobrenjc93/514/base 2025-08-14T21:25:13.8953941Z * [new branch] gh/bobrenjc93/514/head -> origin/gh/bobrenjc93/514/head 2025-08-14T21:25:13.8954096Z * [new branch] gh/bobrenjc93/514/orig -> origin/gh/bobrenjc93/514/orig 2025-08-14T21:25:13.8954304Z * [new branch] gh/bobrenjc93/521/base -> origin/gh/bobrenjc93/521/base 2025-08-14T21:25:13.8954456Z * [new branch] gh/bobrenjc93/521/head -> origin/gh/bobrenjc93/521/head 2025-08-14T21:25:13.8954613Z * [new branch] gh/bobrenjc93/521/orig -> origin/gh/bobrenjc93/521/orig 2025-08-14T21:25:13.8954768Z * [new branch] gh/bobrenjc93/522/base -> origin/gh/bobrenjc93/522/base 2025-08-14T21:25:13.8954921Z * [new branch] gh/bobrenjc93/522/head -> origin/gh/bobrenjc93/522/head 2025-08-14T21:25:13.8955076Z * [new branch] gh/bobrenjc93/522/orig -> origin/gh/bobrenjc93/522/orig 2025-08-14T21:25:13.8955231Z * [new branch] gh/bobrenjc93/525/base -> origin/gh/bobrenjc93/525/base 2025-08-14T21:25:13.8955387Z * [new branch] gh/bobrenjc93/525/head -> origin/gh/bobrenjc93/525/head 2025-08-14T21:25:13.8955540Z * [new branch] gh/bobrenjc93/525/orig -> origin/gh/bobrenjc93/525/orig 2025-08-14T21:25:13.8955696Z * [new branch] gh/bobrenjc93/526/base -> origin/gh/bobrenjc93/526/base 2025-08-14T21:25:13.8955848Z * [new branch] gh/bobrenjc93/526/head -> origin/gh/bobrenjc93/526/head 2025-08-14T21:25:13.8955993Z * [new branch] gh/bobrenjc93/526/orig -> origin/gh/bobrenjc93/526/orig 2025-08-14T21:25:13.8956413Z * [new branch] gh/bobrenjc93/527/base -> origin/gh/bobrenjc93/527/base 2025-08-14T21:25:13.8956569Z * [new branch] gh/bobrenjc93/527/head -> origin/gh/bobrenjc93/527/head 2025-08-14T21:25:13.8956715Z * [new branch] gh/bobrenjc93/527/orig -> origin/gh/bobrenjc93/527/orig 2025-08-14T21:25:13.8957377Z * [new branch] gh/bobrenjc93/528/base -> origin/gh/bobrenjc93/528/base 2025-08-14T21:25:13.8958618Z * [new branch] gh/bobrenjc93/528/head -> origin/gh/bobrenjc93/528/head 2025-08-14T21:25:13.8958808Z * [new branch] gh/bobrenjc93/528/orig -> origin/gh/bobrenjc93/528/orig 2025-08-14T21:25:13.8959863Z * [new branch] gh/bobrenjc93/529/base -> origin/gh/bobrenjc93/529/base 2025-08-14T21:25:13.8960202Z * [new branch] gh/bobrenjc93/529/head -> origin/gh/bobrenjc93/529/head 2025-08-14T21:25:13.8963463Z * [new branch] gh/bobrenjc93/529/orig -> origin/gh/bobrenjc93/529/orig 2025-08-14T21:25:13.8963671Z * [new branch] gh/bobrenjc93/534/base -> origin/gh/bobrenjc93/534/base 2025-08-14T21:25:13.8963826Z * [new branch] gh/bobrenjc93/534/head -> origin/gh/bobrenjc93/534/head 2025-08-14T21:25:13.8963975Z * [new branch] gh/bobrenjc93/534/orig -> origin/gh/bobrenjc93/534/orig 2025-08-14T21:25:13.8964377Z * [new branch] gh/bobrenjc93/535/base -> origin/gh/bobrenjc93/535/base 2025-08-14T21:25:13.8965014Z * [new branch] gh/bobrenjc93/535/head -> origin/gh/bobrenjc93/535/head 2025-08-14T21:25:13.8965553Z * [new branch] gh/bobrenjc93/535/orig -> origin/gh/bobrenjc93/535/orig 2025-08-14T21:25:13.8970317Z * [new branch] gh/bobrenjc93/536/base -> origin/gh/bobrenjc93/536/base 2025-08-14T21:25:13.8970510Z * [new branch] gh/bobrenjc93/536/head -> origin/gh/bobrenjc93/536/head 2025-08-14T21:25:13.8970656Z * [new branch] gh/bobrenjc93/536/orig -> origin/gh/bobrenjc93/536/orig 2025-08-14T21:25:13.8970794Z * [new branch] gh/bobrenjc93/537/base -> origin/gh/bobrenjc93/537/base 2025-08-14T21:25:13.8970938Z * [new branch] gh/bobrenjc93/537/head -> origin/gh/bobrenjc93/537/head 2025-08-14T21:25:13.8971077Z * [new branch] gh/bobrenjc93/537/orig -> origin/gh/bobrenjc93/537/orig 2025-08-14T21:25:13.8971406Z * [new branch] gh/bobrenjc93/538/base -> origin/gh/bobrenjc93/538/base 2025-08-14T21:25:13.8971836Z * [new branch] gh/bobrenjc93/538/head -> origin/gh/bobrenjc93/538/head 2025-08-14T21:25:13.8972262Z * [new branch] gh/bobrenjc93/538/orig -> origin/gh/bobrenjc93/538/orig 2025-08-14T21:25:13.8977882Z * [new branch] gh/bobrenjc93/539/base -> origin/gh/bobrenjc93/539/base 2025-08-14T21:25:13.8978231Z * [new branch] gh/bobrenjc93/539/head -> origin/gh/bobrenjc93/539/head 2025-08-14T21:25:13.8978480Z * [new branch] gh/bobrenjc93/539/orig -> origin/gh/bobrenjc93/539/orig 2025-08-14T21:25:13.8978655Z * [new branch] gh/bobrenjc93/540/base -> origin/gh/bobrenjc93/540/base 2025-08-14T21:25:13.8978936Z * [new branch] gh/bobrenjc93/540/head -> origin/gh/bobrenjc93/540/head 2025-08-14T21:25:13.8979104Z * [new branch] gh/bobrenjc93/540/orig -> origin/gh/bobrenjc93/540/orig 2025-08-14T21:25:13.8984267Z * [new branch] gh/bobrenjc93/541/base -> origin/gh/bobrenjc93/541/base 2025-08-14T21:25:13.8984631Z * [new branch] gh/bobrenjc93/541/head -> origin/gh/bobrenjc93/541/head 2025-08-14T21:25:13.8984881Z * [new branch] gh/bobrenjc93/541/orig -> origin/gh/bobrenjc93/541/orig 2025-08-14T21:25:13.8985138Z * [new branch] gh/bobrenjc93/542/base -> origin/gh/bobrenjc93/542/base 2025-08-14T21:25:13.8985401Z * [new branch] gh/bobrenjc93/542/head -> origin/gh/bobrenjc93/542/head 2025-08-14T21:25:13.8985579Z * [new branch] gh/bobrenjc93/542/orig -> origin/gh/bobrenjc93/542/orig 2025-08-14T21:25:13.8985730Z * [new branch] gh/bobrenjc93/543/base -> origin/gh/bobrenjc93/543/base 2025-08-14T21:25:13.8986342Z * [new branch] gh/bobrenjc93/543/head -> origin/gh/bobrenjc93/543/head 2025-08-14T21:25:13.8986494Z * [new branch] gh/bobrenjc93/543/orig -> origin/gh/bobrenjc93/543/orig 2025-08-14T21:25:13.8986778Z * [new branch] gh/bobrenjc93/544/base -> origin/gh/bobrenjc93/544/base 2025-08-14T21:25:13.8986948Z * [new branch] gh/bobrenjc93/544/head -> origin/gh/bobrenjc93/544/head 2025-08-14T21:25:13.8987095Z * [new branch] gh/bobrenjc93/544/orig -> origin/gh/bobrenjc93/544/orig 2025-08-14T21:25:13.8987242Z * [new branch] gh/bobrenjc93/545/base -> origin/gh/bobrenjc93/545/base 2025-08-14T21:25:13.8987647Z * [new branch] gh/bobrenjc93/545/head -> origin/gh/bobrenjc93/545/head 2025-08-14T21:25:13.8992846Z * [new branch] gh/bobrenjc93/545/orig -> origin/gh/bobrenjc93/545/orig 2025-08-14T21:25:13.8993048Z * [new branch] gh/bobrenjc93/546/base -> origin/gh/bobrenjc93/546/base 2025-08-14T21:25:13.8993205Z * [new branch] gh/bobrenjc93/546/head -> origin/gh/bobrenjc93/546/head 2025-08-14T21:25:13.8993378Z * [new branch] gh/bobrenjc93/546/orig -> origin/gh/bobrenjc93/546/orig 2025-08-14T21:25:13.8993553Z * [new branch] gh/bobrenjc93/547/base -> origin/gh/bobrenjc93/547/base 2025-08-14T21:25:13.8993716Z * [new branch] gh/bobrenjc93/547/head -> origin/gh/bobrenjc93/547/head 2025-08-14T21:25:13.8993883Z * [new branch] gh/bobrenjc93/547/orig -> origin/gh/bobrenjc93/547/orig 2025-08-14T21:25:13.8994049Z * [new branch] gh/bobrenjc93/548/base -> origin/gh/bobrenjc93/548/base 2025-08-14T21:25:13.8994240Z * [new branch] gh/bobrenjc93/548/head -> origin/gh/bobrenjc93/548/head 2025-08-14T21:25:13.8995284Z * [new branch] gh/bobrenjc93/548/orig -> origin/gh/bobrenjc93/548/orig 2025-08-14T21:25:13.8997217Z * [new branch] gh/bobrenjc93/549/base -> origin/gh/bobrenjc93/549/base 2025-08-14T21:25:13.8997495Z * [new branch] gh/bobrenjc93/549/head -> origin/gh/bobrenjc93/549/head 2025-08-14T21:25:13.8997732Z * [new branch] gh/bobrenjc93/549/orig -> origin/gh/bobrenjc93/549/orig 2025-08-14T21:25:13.9000832Z * [new branch] gh/briancoutinho/2/base -> origin/gh/briancoutinho/2/base 2025-08-14T21:25:13.9005902Z * [new branch] gh/briancoutinho/2/head -> origin/gh/briancoutinho/2/head 2025-08-14T21:25:13.9006090Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-08-14T21:25:13.9006228Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-08-14T21:25:13.9006352Z * [new branch] gh/c00w/38/base -> origin/gh/c00w/38/base 2025-08-14T21:25:13.9006475Z * [new branch] gh/c00w/38/head -> origin/gh/c00w/38/head 2025-08-14T21:25:13.9006604Z * [new branch] gh/c00w/38/orig -> origin/gh/c00w/38/orig 2025-08-14T21:25:13.9006726Z * [new branch] gh/c00w/48/base -> origin/gh/c00w/48/base 2025-08-14T21:25:13.9006864Z * [new branch] gh/c00w/48/head -> origin/gh/c00w/48/head 2025-08-14T21:25:13.9006999Z * [new branch] gh/c00w/48/orig -> origin/gh/c00w/48/orig 2025-08-14T21:25:13.9007295Z * [new branch] gh/c00w/50/base -> origin/gh/c00w/50/base 2025-08-14T21:25:13.9016122Z * [new branch] gh/c00w/50/head -> origin/gh/c00w/50/head 2025-08-14T21:25:13.9018140Z * [new branch] gh/c00w/50/orig -> origin/gh/c00w/50/orig 2025-08-14T21:25:13.9018652Z * [new branch] gh/c00w/51/base -> origin/gh/c00w/51/base 2025-08-14T21:25:13.9022179Z * [new branch] gh/c00w/51/head -> origin/gh/c00w/51/head 2025-08-14T21:25:13.9025022Z * [new branch] gh/c00w/51/orig -> origin/gh/c00w/51/orig 2025-08-14T21:25:13.9025158Z * [new branch] gh/c00w/52/base -> origin/gh/c00w/52/base 2025-08-14T21:25:13.9025515Z * [new branch] gh/c00w/52/head -> origin/gh/c00w/52/head 2025-08-14T21:25:13.9025680Z * [new branch] gh/c00w/52/orig -> origin/gh/c00w/52/orig 2025-08-14T21:25:13.9025809Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-08-14T21:25:13.9025940Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-08-14T21:25:13.9026069Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-08-14T21:25:13.9026197Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-08-14T21:25:13.9026333Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-08-14T21:25:13.9026459Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-08-14T21:25:13.9026637Z * [new branch] gh/chenmillie/1/base -> origin/gh/chenmillie/1/base 2025-08-14T21:25:13.9026797Z * [new branch] gh/chenmillie/1/head -> origin/gh/chenmillie/1/head 2025-08-14T21:25:13.9026953Z * [new branch] gh/chenmillie/1/orig -> origin/gh/chenmillie/1/orig 2025-08-14T21:25:13.9027106Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-08-14T21:25:13.9027248Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-08-14T21:25:13.9027386Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-08-14T21:25:13.9027560Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-08-14T21:25:13.9027709Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-08-14T21:25:13.9027876Z * [new branch] gh/coconutruben/11/base -> origin/gh/coconutruben/11/base 2025-08-14T21:25:13.9028033Z * [new branch] gh/coconutruben/11/head -> origin/gh/coconutruben/11/head 2025-08-14T21:25:13.9028191Z * [new branch] gh/coconutruben/11/orig -> origin/gh/coconutruben/11/orig 2025-08-14T21:25:13.9032795Z * [new branch] gh/coconutruben/12/base -> origin/gh/coconutruben/12/base 2025-08-14T21:25:13.9033180Z * [new branch] gh/coconutruben/12/head -> origin/gh/coconutruben/12/head 2025-08-14T21:25:13.9033361Z * [new branch] gh/coconutruben/12/orig -> origin/gh/coconutruben/12/orig 2025-08-14T21:25:13.9033519Z * [new branch] gh/coconutruben/13/base -> origin/gh/coconutruben/13/base 2025-08-14T21:25:13.9033677Z * [new branch] gh/coconutruben/13/head -> origin/gh/coconutruben/13/head 2025-08-14T21:25:13.9033834Z * [new branch] gh/coconutruben/13/orig -> origin/gh/coconutruben/13/orig 2025-08-14T21:25:13.9033988Z * [new branch] gh/coconutruben/14/base -> origin/gh/coconutruben/14/base 2025-08-14T21:25:13.9034186Z * [new branch] gh/coconutruben/14/head -> origin/gh/coconutruben/14/head 2025-08-14T21:25:13.9034362Z * [new branch] gh/coconutruben/14/orig -> origin/gh/coconutruben/14/orig 2025-08-14T21:25:13.9034902Z * [new branch] gh/coconutruben/15/base -> origin/gh/coconutruben/15/base 2025-08-14T21:25:13.9035990Z * [new branch] gh/coconutruben/15/head -> origin/gh/coconutruben/15/head 2025-08-14T21:25:13.9036631Z * [new branch] gh/coconutruben/15/orig -> origin/gh/coconutruben/15/orig 2025-08-14T21:25:13.9045276Z * [new branch] gh/coconutruben/16/base -> origin/gh/coconutruben/16/base 2025-08-14T21:25:13.9049716Z * [new branch] gh/coconutruben/16/head -> origin/gh/coconutruben/16/head 2025-08-14T21:25:13.9049950Z * [new branch] gh/coconutruben/16/orig -> origin/gh/coconutruben/16/orig 2025-08-14T21:25:13.9050558Z * [new branch] gh/coconutruben/17/base -> origin/gh/coconutruben/17/base 2025-08-14T21:25:13.9050899Z * [new branch] gh/coconutruben/17/head -> origin/gh/coconutruben/17/head 2025-08-14T21:25:13.9051061Z * [new branch] gh/coconutruben/17/orig -> origin/gh/coconutruben/17/orig 2025-08-14T21:25:13.9051226Z * [new branch] gh/coconutruben/18/base -> origin/gh/coconutruben/18/base 2025-08-14T21:25:13.9051382Z * [new branch] gh/coconutruben/18/head -> origin/gh/coconutruben/18/head 2025-08-14T21:25:13.9051542Z * [new branch] gh/coconutruben/18/orig -> origin/gh/coconutruben/18/orig 2025-08-14T21:25:13.9051695Z * [new branch] gh/coconutruben/19/base -> origin/gh/coconutruben/19/base 2025-08-14T21:25:13.9051849Z * [new branch] gh/coconutruben/19/head -> origin/gh/coconutruben/19/head 2025-08-14T21:25:13.9052043Z * [new branch] gh/coconutruben/19/orig -> origin/gh/coconutruben/19/orig 2025-08-14T21:25:13.9052238Z * [new branch] gh/coconutruben/20/base -> origin/gh/coconutruben/20/base 2025-08-14T21:25:13.9052417Z * [new branch] gh/coconutruben/20/head -> origin/gh/coconutruben/20/head 2025-08-14T21:25:13.9052588Z * [new branch] gh/coconutruben/20/orig -> origin/gh/coconutruben/20/orig 2025-08-14T21:25:13.9052745Z * [new branch] gh/coconutruben/21/base -> origin/gh/coconutruben/21/base 2025-08-14T21:25:13.9052927Z * [new branch] gh/coconutruben/21/head -> origin/gh/coconutruben/21/head 2025-08-14T21:25:13.9053084Z * [new branch] gh/coconutruben/21/orig -> origin/gh/coconutruben/21/orig 2025-08-14T21:25:13.9053253Z * [new branch] gh/coconutruben/22/base -> origin/gh/coconutruben/22/base 2025-08-14T21:25:13.9053406Z * [new branch] gh/coconutruben/22/head -> origin/gh/coconutruben/22/head 2025-08-14T21:25:13.9053714Z * [new branch] gh/coconutruben/22/orig -> origin/gh/coconutruben/22/orig 2025-08-14T21:25:13.9057879Z * [new branch] gh/coconutruben/23/base -> origin/gh/coconutruben/23/base 2025-08-14T21:25:13.9058227Z * [new branch] gh/coconutruben/23/head -> origin/gh/coconutruben/23/head 2025-08-14T21:25:13.9058386Z * [new branch] gh/coconutruben/23/orig -> origin/gh/coconutruben/23/orig 2025-08-14T21:25:13.9058545Z * [new branch] gh/coconutruben/24/base -> origin/gh/coconutruben/24/base 2025-08-14T21:25:13.9061560Z * [new branch] gh/coconutruben/24/head -> origin/gh/coconutruben/24/head 2025-08-14T21:25:13.9061881Z * [new branch] gh/coconutruben/24/orig -> origin/gh/coconutruben/24/orig 2025-08-14T21:25:13.9066550Z * [new branch] gh/coconutruben/25/base -> origin/gh/coconutruben/25/base 2025-08-14T21:25:13.9071599Z * [new branch] gh/coconutruben/25/head -> origin/gh/coconutruben/25/head 2025-08-14T21:25:13.9071966Z * [new branch] gh/coconutruben/25/orig -> origin/gh/coconutruben/25/orig 2025-08-14T21:25:13.9072244Z * [new branch] gh/coconutruben/26/base -> origin/gh/coconutruben/26/base 2025-08-14T21:25:13.9072482Z * [new branch] gh/coconutruben/26/head -> origin/gh/coconutruben/26/head 2025-08-14T21:25:13.9072670Z * [new branch] gh/coconutruben/26/orig -> origin/gh/coconutruben/26/orig 2025-08-14T21:25:13.9072834Z * [new branch] gh/coconutruben/27/base -> origin/gh/coconutruben/27/base 2025-08-14T21:25:13.9072989Z * [new branch] gh/coconutruben/27/head -> origin/gh/coconutruben/27/head 2025-08-14T21:25:13.9073149Z * [new branch] gh/coconutruben/27/orig -> origin/gh/coconutruben/27/orig 2025-08-14T21:25:13.9073335Z * [new branch] gh/codingwithsurya/10/base -> origin/gh/codingwithsurya/10/base 2025-08-14T21:25:13.9073500Z * [new branch] gh/codingwithsurya/10/head -> origin/gh/codingwithsurya/10/head 2025-08-14T21:25:13.9074035Z * [new branch] gh/codingwithsurya/10/orig -> origin/gh/codingwithsurya/10/orig 2025-08-14T21:25:13.9074223Z * [new branch] gh/codingwithsurya/11/base -> origin/gh/codingwithsurya/11/base 2025-08-14T21:25:13.9074401Z * [new branch] gh/codingwithsurya/11/head -> origin/gh/codingwithsurya/11/head 2025-08-14T21:25:13.9074564Z * [new branch] gh/codingwithsurya/11/orig -> origin/gh/codingwithsurya/11/orig 2025-08-14T21:25:13.9074738Z * [new branch] gh/codingwithsurya/12/base -> origin/gh/codingwithsurya/12/base 2025-08-14T21:25:13.9075778Z * [new branch] gh/codingwithsurya/12/head -> origin/gh/codingwithsurya/12/head 2025-08-14T21:25:13.9076463Z * [new branch] gh/codingwithsurya/12/orig -> origin/gh/codingwithsurya/12/orig 2025-08-14T21:25:13.9080715Z * [new branch] gh/codingwithsurya/13/base -> origin/gh/codingwithsurya/13/base 2025-08-14T21:25:13.9081039Z * [new branch] gh/codingwithsurya/13/head -> origin/gh/codingwithsurya/13/head 2025-08-14T21:25:13.9081300Z * [new branch] gh/codingwithsurya/13/orig -> origin/gh/codingwithsurya/13/orig 2025-08-14T21:25:13.9081496Z * [new branch] gh/codingwithsurya/14/base -> origin/gh/codingwithsurya/14/base 2025-08-14T21:25:13.9081757Z * [new branch] gh/codingwithsurya/14/head -> origin/gh/codingwithsurya/14/head 2025-08-14T21:25:13.9082101Z * [new branch] gh/codingwithsurya/14/orig -> origin/gh/codingwithsurya/14/orig 2025-08-14T21:25:13.9082572Z * [new branch] gh/codingwithsurya/15/base -> origin/gh/codingwithsurya/15/base 2025-08-14T21:25:13.9087911Z * [new branch] gh/codingwithsurya/15/head -> origin/gh/codingwithsurya/15/head 2025-08-14T21:25:13.9088260Z * [new branch] gh/codingwithsurya/15/orig -> origin/gh/codingwithsurya/15/orig 2025-08-14T21:25:13.9088523Z * [new branch] gh/codingwithsurya/16/base -> origin/gh/codingwithsurya/16/base 2025-08-14T21:25:13.9089004Z * [new branch] gh/codingwithsurya/16/head -> origin/gh/codingwithsurya/16/head 2025-08-14T21:25:13.9089253Z * [new branch] gh/codingwithsurya/16/orig -> origin/gh/codingwithsurya/16/orig 2025-08-14T21:25:13.9089480Z * [new branch] gh/codingwithsurya/17/base -> origin/gh/codingwithsurya/17/base 2025-08-14T21:25:13.9089717Z * [new branch] gh/codingwithsurya/17/head -> origin/gh/codingwithsurya/17/head 2025-08-14T21:25:13.9089957Z * [new branch] gh/codingwithsurya/17/orig -> origin/gh/codingwithsurya/17/orig 2025-08-14T21:25:13.9090130Z * [new branch] gh/codingwithsurya/18/base -> origin/gh/codingwithsurya/18/base 2025-08-14T21:25:13.9091146Z * [new branch] gh/codingwithsurya/18/head -> origin/gh/codingwithsurya/18/head 2025-08-14T21:25:13.9091500Z * [new branch] gh/codingwithsurya/18/orig -> origin/gh/codingwithsurya/18/orig 2025-08-14T21:25:13.9094743Z * [new branch] gh/codingwithsurya/19/base -> origin/gh/codingwithsurya/19/base 2025-08-14T21:25:13.9095109Z * [new branch] gh/codingwithsurya/19/head -> origin/gh/codingwithsurya/19/head 2025-08-14T21:25:13.9095422Z * [new branch] gh/codingwithsurya/19/orig -> origin/gh/codingwithsurya/19/orig 2025-08-14T21:25:13.9095722Z * [new branch] gh/codingwithsurya/20/base -> origin/gh/codingwithsurya/20/base 2025-08-14T21:25:13.9095914Z * [new branch] gh/codingwithsurya/20/head -> origin/gh/codingwithsurya/20/head 2025-08-14T21:25:13.9096930Z * [new branch] gh/codingwithsurya/20/orig -> origin/gh/codingwithsurya/20/orig 2025-08-14T21:25:13.9100126Z * [new branch] gh/codingwithsurya/21/base -> origin/gh/codingwithsurya/21/base 2025-08-14T21:25:13.9100436Z * [new branch] gh/codingwithsurya/21/head -> origin/gh/codingwithsurya/21/head 2025-08-14T21:25:13.9100801Z * [new branch] gh/codingwithsurya/21/orig -> origin/gh/codingwithsurya/21/orig 2025-08-14T21:25:13.9101119Z * [new branch] gh/codingwithsurya/8/base -> origin/gh/codingwithsurya/8/base 2025-08-14T21:25:13.9101362Z * [new branch] gh/codingwithsurya/8/head -> origin/gh/codingwithsurya/8/head 2025-08-14T21:25:13.9101967Z * [new branch] gh/codingwithsurya/8/orig -> origin/gh/codingwithsurya/8/orig 2025-08-14T21:25:13.9102454Z * [new branch] gh/codingwithsurya/9/base -> origin/gh/codingwithsurya/9/base 2025-08-14T21:25:13.9103794Z * [new branch] gh/codingwithsurya/9/head -> origin/gh/codingwithsurya/9/head 2025-08-14T21:25:13.9104170Z * [new branch] gh/codingwithsurya/9/orig -> origin/gh/codingwithsurya/9/orig 2025-08-14T21:25:13.9108090Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-08-14T21:25:13.9108426Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-08-14T21:25:13.9108811Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-08-14T21:25:13.9109445Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-08-14T21:25:13.9109639Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-08-14T21:25:13.9109822Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-08-14T21:25:13.9109984Z * [new branch] gh/colinchan15/4/base -> origin/gh/colinchan15/4/base 2025-08-14T21:25:13.9110196Z * [new branch] gh/colinchan15/4/head -> origin/gh/colinchan15/4/head 2025-08-14T21:25:13.9111471Z * [new branch] gh/colinchan15/5/base -> origin/gh/colinchan15/5/base 2025-08-14T21:25:13.9112144Z * [new branch] gh/colinchan15/5/head -> origin/gh/colinchan15/5/head 2025-08-14T21:25:13.9112859Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-08-14T21:25:13.9113436Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-08-14T21:25:13.9114936Z * [new branch] gh/davidberard98/351/base -> origin/gh/davidberard98/351/base 2025-08-14T21:25:13.9115238Z * [new branch] gh/davidberard98/351/head -> origin/gh/davidberard98/351/head 2025-08-14T21:25:13.9116553Z * [new branch] gh/davidberard98/351/orig -> origin/gh/davidberard98/351/orig 2025-08-14T21:25:13.9117105Z * [new branch] gh/davidberard98/353/base -> origin/gh/davidberard98/353/base 2025-08-14T21:25:13.9117669Z * [new branch] gh/davidberard98/353/head -> origin/gh/davidberard98/353/head 2025-08-14T21:25:13.9118913Z * [new branch] gh/davidberard98/353/orig -> origin/gh/davidberard98/353/orig 2025-08-14T21:25:13.9119190Z * [new branch] gh/davidberard98/356/base -> origin/gh/davidberard98/356/base 2025-08-14T21:25:13.9120462Z * [new branch] gh/davidberard98/356/head -> origin/gh/davidberard98/356/head 2025-08-14T21:25:13.9120623Z * [new branch] gh/davidberard98/356/orig -> origin/gh/davidberard98/356/orig 2025-08-14T21:25:13.9123022Z * [new branch] gh/davidberard98/382/base -> origin/gh/davidberard98/382/base 2025-08-14T21:25:13.9123215Z * [new branch] gh/davidberard98/382/head -> origin/gh/davidberard98/382/head 2025-08-14T21:25:13.9123380Z * [new branch] gh/davidberard98/382/orig -> origin/gh/davidberard98/382/orig 2025-08-14T21:25:13.9129149Z * [new branch] gh/davidberard98/386/base -> origin/gh/davidberard98/386/base 2025-08-14T21:25:13.9129357Z * [new branch] gh/davidberard98/386/head -> origin/gh/davidberard98/386/head 2025-08-14T21:25:13.9129516Z * [new branch] gh/davidberard98/386/orig -> origin/gh/davidberard98/386/orig 2025-08-14T21:25:13.9129893Z * [new branch] gh/davidberard98/389/base -> origin/gh/davidberard98/389/base 2025-08-14T21:25:13.9130084Z * [new branch] gh/davidberard98/389/head -> origin/gh/davidberard98/389/head 2025-08-14T21:25:13.9130231Z * [new branch] gh/davidberard98/389/orig -> origin/gh/davidberard98/389/orig 2025-08-14T21:25:13.9130383Z * [new branch] gh/davidberard98/390/base -> origin/gh/davidberard98/390/base 2025-08-14T21:25:13.9130527Z * [new branch] gh/davidberard98/390/head -> origin/gh/davidberard98/390/head 2025-08-14T21:25:13.9130674Z * [new branch] gh/davidberard98/390/orig -> origin/gh/davidberard98/390/orig 2025-08-14T21:25:13.9131140Z * [new branch] gh/davidberard98/391/base -> origin/gh/davidberard98/391/base 2025-08-14T21:25:13.9131770Z * [new branch] gh/davidberard98/391/head -> origin/gh/davidberard98/391/head 2025-08-14T21:25:13.9132403Z * [new branch] gh/davidberard98/391/orig -> origin/gh/davidberard98/391/orig 2025-08-14T21:25:13.9137439Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-08-14T21:25:13.9137660Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-08-14T21:25:13.9137826Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-08-14T21:25:13.9138009Z * [new branch] gh/davidberard98/393/base -> origin/gh/davidberard98/393/base 2025-08-14T21:25:13.9138178Z * [new branch] gh/davidberard98/393/head -> origin/gh/davidberard98/393/head 2025-08-14T21:25:13.9138334Z * [new branch] gh/davidberard98/393/orig -> origin/gh/davidberard98/393/orig 2025-08-14T21:25:13.9138672Z * [new branch] gh/davidberard98/394/base -> origin/gh/davidberard98/394/base 2025-08-14T21:25:13.9138962Z * [new branch] gh/davidberard98/394/head -> origin/gh/davidberard98/394/head 2025-08-14T21:25:13.9140430Z * [new branch] gh/davidberard98/394/orig -> origin/gh/davidberard98/394/orig 2025-08-14T21:25:13.9140867Z * [new branch] gh/davidberard98/395/base -> origin/gh/davidberard98/395/base 2025-08-14T21:25:13.9142925Z * [new branch] gh/davidberard98/395/head -> origin/gh/davidberard98/395/head 2025-08-14T21:25:13.9143244Z * [new branch] gh/davidberard98/395/orig -> origin/gh/davidberard98/395/orig 2025-08-14T21:25:13.9143439Z * [new branch] gh/davidberard98/396/base -> origin/gh/davidberard98/396/base 2025-08-14T21:25:13.9144809Z * [new branch] gh/davidberard98/396/head -> origin/gh/davidberard98/396/head 2025-08-14T21:25:13.9145015Z * [new branch] gh/davidberard98/396/orig -> origin/gh/davidberard98/396/orig 2025-08-14T21:25:13.9146300Z * [new branch] gh/davidberard98/397/base -> origin/gh/davidberard98/397/base 2025-08-14T21:25:13.9146548Z * [new branch] gh/davidberard98/397/head -> origin/gh/davidberard98/397/head 2025-08-14T21:25:13.9150363Z * [new branch] gh/davidberard98/397/orig -> origin/gh/davidberard98/397/orig 2025-08-14T21:25:13.9150557Z * [new branch] gh/davidberard98/398/base -> origin/gh/davidberard98/398/base 2025-08-14T21:25:13.9150722Z * [new branch] gh/davidberard98/398/head -> origin/gh/davidberard98/398/head 2025-08-14T21:25:13.9150871Z * [new branch] gh/davidberard98/398/orig -> origin/gh/davidberard98/398/orig 2025-08-14T21:25:13.9151353Z * [new branch] gh/desertfire/570/base -> origin/gh/desertfire/570/base 2025-08-14T21:25:13.9152026Z * [new branch] gh/desertfire/570/head -> origin/gh/desertfire/570/head 2025-08-14T21:25:13.9152977Z * [new branch] gh/desertfire/570/orig -> origin/gh/desertfire/570/orig 2025-08-14T21:25:13.9153514Z * [new branch] gh/desertfire/572/base -> origin/gh/desertfire/572/base 2025-08-14T21:25:13.9154865Z * [new branch] gh/desertfire/572/head -> origin/gh/desertfire/572/head 2025-08-14T21:25:13.9155054Z * [new branch] gh/desertfire/572/orig -> origin/gh/desertfire/572/orig 2025-08-14T21:25:13.9156853Z * [new branch] gh/desertfire/589/base -> origin/gh/desertfire/589/base 2025-08-14T21:25:13.9157022Z * [new branch] gh/desertfire/589/head -> origin/gh/desertfire/589/head 2025-08-14T21:25:13.9157963Z * [new branch] gh/desertfire/589/orig -> origin/gh/desertfire/589/orig 2025-08-14T21:25:13.9162391Z * [new branch] gh/desertfire/590/base -> origin/gh/desertfire/590/base 2025-08-14T21:25:13.9162592Z * [new branch] gh/desertfire/590/head -> origin/gh/desertfire/590/head 2025-08-14T21:25:13.9162752Z * [new branch] gh/desertfire/590/orig -> origin/gh/desertfire/590/orig 2025-08-14T21:25:13.9162927Z * [new branch] gh/desertfire/591/base -> origin/gh/desertfire/591/base 2025-08-14T21:25:13.9163083Z * [new branch] gh/desertfire/591/head -> origin/gh/desertfire/591/head 2025-08-14T21:25:13.9163245Z * [new branch] gh/desertfire/591/orig -> origin/gh/desertfire/591/orig 2025-08-14T21:25:13.9163404Z * [new branch] gh/desertfire/592/base -> origin/gh/desertfire/592/base 2025-08-14T21:25:13.9163754Z * [new branch] gh/desertfire/592/head -> origin/gh/desertfire/592/head 2025-08-14T21:25:13.9164750Z * [new branch] gh/desertfire/592/orig -> origin/gh/desertfire/592/orig 2025-08-14T21:25:13.9165233Z * [new branch] gh/desertfire/593/base -> origin/gh/desertfire/593/base 2025-08-14T21:25:13.9169204Z * [new branch] gh/desertfire/593/head -> origin/gh/desertfire/593/head 2025-08-14T21:25:13.9169394Z * [new branch] gh/desertfire/593/orig -> origin/gh/desertfire/593/orig 2025-08-14T21:25:13.9169553Z * [new branch] gh/desertfire/594/base -> origin/gh/desertfire/594/base 2025-08-14T21:25:13.9169867Z * [new branch] gh/desertfire/594/head -> origin/gh/desertfire/594/head 2025-08-14T21:25:13.9170021Z * [new branch] gh/desertfire/594/orig -> origin/gh/desertfire/594/orig 2025-08-14T21:25:13.9170364Z * [new branch] gh/desertfire/595/base -> origin/gh/desertfire/595/base 2025-08-14T21:25:13.9170530Z * [new branch] gh/desertfire/595/head -> origin/gh/desertfire/595/head 2025-08-14T21:25:13.9172056Z * [new branch] gh/desertfire/595/orig -> origin/gh/desertfire/595/orig 2025-08-14T21:25:13.9172313Z * [new branch] gh/desertfire/596/base -> origin/gh/desertfire/596/base 2025-08-14T21:25:13.9172882Z * [new branch] gh/desertfire/596/head -> origin/gh/desertfire/596/head 2025-08-14T21:25:13.9174260Z * [new branch] gh/desertfire/596/orig -> origin/gh/desertfire/596/orig 2025-08-14T21:25:13.9174596Z * [new branch] gh/desertfire/597/base -> origin/gh/desertfire/597/base 2025-08-14T21:25:13.9177491Z * [new branch] gh/desertfire/597/head -> origin/gh/desertfire/597/head 2025-08-14T21:25:13.9177857Z * [new branch] gh/desertfire/597/orig -> origin/gh/desertfire/597/orig 2025-08-14T21:25:13.9178114Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-08-14T21:25:13.9178351Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-08-14T21:25:13.9178973Z * [new branch] gh/dharakk/4/base -> origin/gh/dharakk/4/base 2025-08-14T21:25:13.9179902Z * [new branch] gh/dharakk/4/head -> origin/gh/dharakk/4/head 2025-08-14T21:25:13.9180390Z * [new branch] gh/dharakk/4/orig -> origin/gh/dharakk/4/orig 2025-08-14T21:25:13.9183128Z * [new branch] gh/drisspg/140/base -> origin/gh/drisspg/140/base 2025-08-14T21:25:13.9183460Z * [new branch] gh/drisspg/140/head -> origin/gh/drisspg/140/head 2025-08-14T21:25:13.9184173Z * [new branch] gh/drisspg/140/orig -> origin/gh/drisspg/140/orig 2025-08-14T21:25:13.9184520Z * [new branch] gh/drisspg/149/base -> origin/gh/drisspg/149/base 2025-08-14T21:25:13.9184669Z * [new branch] gh/drisspg/149/head -> origin/gh/drisspg/149/head 2025-08-14T21:25:13.9189219Z * [new branch] gh/drisspg/149/orig -> origin/gh/drisspg/149/orig 2025-08-14T21:25:13.9189873Z * [new branch] gh/drisspg/150/base -> origin/gh/drisspg/150/base 2025-08-14T21:25:13.9190053Z * [new branch] gh/drisspg/150/head -> origin/gh/drisspg/150/head 2025-08-14T21:25:13.9190194Z * [new branch] gh/drisspg/150/orig -> origin/gh/drisspg/150/orig 2025-08-14T21:25:13.9190352Z * [new branch] gh/drisspg/151/base -> origin/gh/drisspg/151/base 2025-08-14T21:25:13.9190508Z * [new branch] gh/drisspg/151/head -> origin/gh/drisspg/151/head 2025-08-14T21:25:13.9190662Z * [new branch] gh/drisspg/151/orig -> origin/gh/drisspg/151/orig 2025-08-14T21:25:13.9190807Z * [new branch] gh/drisspg/158/base -> origin/gh/drisspg/158/base 2025-08-14T21:25:13.9191418Z * [new branch] gh/drisspg/158/head -> origin/gh/drisspg/158/head 2025-08-14T21:25:13.9192480Z * [new branch] gh/drisspg/158/orig -> origin/gh/drisspg/158/orig 2025-08-14T21:25:13.9192882Z * [new branch] gh/drisspg/159/base -> origin/gh/drisspg/159/base 2025-08-14T21:25:13.9193825Z * [new branch] gh/drisspg/159/head -> origin/gh/drisspg/159/head 2025-08-14T21:25:13.9194175Z * [new branch] gh/drisspg/159/orig -> origin/gh/drisspg/159/orig 2025-08-14T21:25:13.9197358Z * [new branch] gh/drisspg/166/base -> origin/gh/drisspg/166/base 2025-08-14T21:25:13.9197708Z * [new branch] gh/drisspg/166/head -> origin/gh/drisspg/166/head 2025-08-14T21:25:13.9197917Z * [new branch] gh/drisspg/166/orig -> origin/gh/drisspg/166/orig 2025-08-14T21:25:13.9198069Z * [new branch] gh/drisspg/168/base -> origin/gh/drisspg/168/base 2025-08-14T21:25:13.9198261Z * [new branch] gh/drisspg/168/head -> origin/gh/drisspg/168/head 2025-08-14T21:25:13.9200700Z * [new branch] gh/drisspg/168/orig -> origin/gh/drisspg/168/orig 2025-08-14T21:25:13.9201061Z * [new branch] gh/drisspg/169/base -> origin/gh/drisspg/169/base 2025-08-14T21:25:13.9201357Z * [new branch] gh/drisspg/169/head -> origin/gh/drisspg/169/head 2025-08-14T21:25:13.9201633Z * [new branch] gh/drisspg/169/orig -> origin/gh/drisspg/169/orig 2025-08-14T21:25:13.9201911Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-08-14T21:25:13.9202450Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-08-14T21:25:13.9203327Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-08-14T21:25:13.9207272Z * [new branch] gh/drisspg/171/base -> origin/gh/drisspg/171/base 2025-08-14T21:25:13.9207591Z * [new branch] gh/drisspg/171/head -> origin/gh/drisspg/171/head 2025-08-14T21:25:13.9207822Z * [new branch] gh/drisspg/171/orig -> origin/gh/drisspg/171/orig 2025-08-14T21:25:13.9208035Z * [new branch] gh/drisspg/172/base -> origin/gh/drisspg/172/base 2025-08-14T21:25:13.9208247Z * [new branch] gh/drisspg/172/head -> origin/gh/drisspg/172/head 2025-08-14T21:25:13.9208465Z * [new branch] gh/drisspg/172/orig -> origin/gh/drisspg/172/orig 2025-08-14T21:25:13.9208944Z * [new branch] gh/drisspg/173/base -> origin/gh/drisspg/173/base 2025-08-14T21:25:13.9209484Z * [new branch] gh/drisspg/173/head -> origin/gh/drisspg/173/head 2025-08-14T21:25:13.9209681Z * [new branch] gh/drisspg/173/orig -> origin/gh/drisspg/173/orig 2025-08-14T21:25:13.9213927Z * [new branch] gh/drisspg/174/base -> origin/gh/drisspg/174/base 2025-08-14T21:25:13.9214255Z * [new branch] gh/drisspg/174/head -> origin/gh/drisspg/174/head 2025-08-14T21:25:13.9214503Z * [new branch] gh/drisspg/174/orig -> origin/gh/drisspg/174/orig 2025-08-14T21:25:13.9214677Z * [new branch] gh/drisspg/175/base -> origin/gh/drisspg/175/base 2025-08-14T21:25:13.9214897Z * [new branch] gh/drisspg/175/head -> origin/gh/drisspg/175/head 2025-08-14T21:25:13.9215143Z * [new branch] gh/drisspg/175/orig -> origin/gh/drisspg/175/orig 2025-08-14T21:25:13.9215849Z * [new branch] gh/drisspg/176/base -> origin/gh/drisspg/176/base 2025-08-14T21:25:13.9216082Z * [new branch] gh/drisspg/176/head -> origin/gh/drisspg/176/head 2025-08-14T21:25:13.9216998Z * [new branch] gh/drisspg/176/orig -> origin/gh/drisspg/176/orig 2025-08-14T21:25:13.9218245Z * [new branch] gh/drisspg/177/base -> origin/gh/drisspg/177/base 2025-08-14T21:25:13.9218395Z * [new branch] gh/drisspg/177/head -> origin/gh/drisspg/177/head 2025-08-14T21:25:13.9221567Z * [new branch] gh/drisspg/177/orig -> origin/gh/drisspg/177/orig 2025-08-14T21:25:13.9221742Z * [new branch] gh/drisspg/178/base -> origin/gh/drisspg/178/base 2025-08-14T21:25:13.9221883Z * [new branch] gh/drisspg/178/head -> origin/gh/drisspg/178/head 2025-08-14T21:25:13.9222037Z * [new branch] gh/drisspg/178/orig -> origin/gh/drisspg/178/orig 2025-08-14T21:25:13.9222454Z * [new branch] gh/drisspg/179/base -> origin/gh/drisspg/179/base 2025-08-14T21:25:13.9222911Z * [new branch] gh/drisspg/179/head -> origin/gh/drisspg/179/head 2025-08-14T21:25:13.9223650Z * [new branch] gh/drisspg/179/orig -> origin/gh/drisspg/179/orig 2025-08-14T21:25:13.9224958Z * [new branch] gh/drisspg/180/base -> origin/gh/drisspg/180/base 2025-08-14T21:25:13.9225535Z * [new branch] gh/drisspg/180/head -> origin/gh/drisspg/180/head 2025-08-14T21:25:13.9225765Z * [new branch] gh/drisspg/180/orig -> origin/gh/drisspg/180/orig 2025-08-14T21:25:13.9228268Z * [new branch] gh/drisspg/181/base -> origin/gh/drisspg/181/base 2025-08-14T21:25:13.9228476Z * [new branch] gh/drisspg/181/head -> origin/gh/drisspg/181/head 2025-08-14T21:25:13.9228635Z * [new branch] gh/drisspg/181/orig -> origin/gh/drisspg/181/orig 2025-08-14T21:25:13.9228784Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-08-14T21:25:13.9231403Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-08-14T21:25:13.9231587Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-08-14T21:25:13.9231734Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-08-14T21:25:13.9232440Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-08-14T21:25:13.9232775Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-08-14T21:25:13.9233933Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-08-14T21:25:13.9236251Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-08-14T21:25:13.9236622Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-08-14T21:25:13.9236782Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-08-14T21:25:13.9237309Z * [new branch] gh/eellison/784/base -> origin/gh/eellison/784/base 2025-08-14T21:25:13.9241081Z * [new branch] gh/eellison/784/head -> origin/gh/eellison/784/head 2025-08-14T21:25:13.9241424Z * [new branch] gh/eellison/784/orig -> origin/gh/eellison/784/orig 2025-08-14T21:25:13.9241678Z * [new branch] gh/eellison/785/base -> origin/gh/eellison/785/base 2025-08-14T21:25:13.9241853Z * [new branch] gh/eellison/785/head -> origin/gh/eellison/785/head 2025-08-14T21:25:13.9242090Z * [new branch] gh/eellison/785/orig -> origin/gh/eellison/785/orig 2025-08-14T21:25:13.9247149Z * [new branch] gh/eellison/789/base -> origin/gh/eellison/789/base 2025-08-14T21:25:13.9247514Z * [new branch] gh/eellison/789/head -> origin/gh/eellison/789/head 2025-08-14T21:25:13.9247813Z * [new branch] gh/eellison/789/orig -> origin/gh/eellison/789/orig 2025-08-14T21:25:13.9248071Z * [new branch] gh/eellison/800/base -> origin/gh/eellison/800/base 2025-08-14T21:25:13.9248312Z * [new branch] gh/eellison/800/head -> origin/gh/eellison/800/head 2025-08-14T21:25:13.9249008Z * [new branch] gh/eellison/800/orig -> origin/gh/eellison/800/orig 2025-08-14T21:25:13.9249186Z * [new branch] gh/eellison/801/base -> origin/gh/eellison/801/base 2025-08-14T21:25:13.9253850Z * [new branch] gh/eellison/801/head -> origin/gh/eellison/801/head 2025-08-14T21:25:13.9254029Z * [new branch] gh/eellison/801/orig -> origin/gh/eellison/801/orig 2025-08-14T21:25:13.9254164Z * [new branch] gh/eellison/802/base -> origin/gh/eellison/802/base 2025-08-14T21:25:13.9254341Z * [new branch] gh/eellison/802/head -> origin/gh/eellison/802/head 2025-08-14T21:25:13.9254658Z * [new branch] gh/eellison/802/orig -> origin/gh/eellison/802/orig 2025-08-14T21:25:13.9254814Z * [new branch] gh/eellison/805/base -> origin/gh/eellison/805/base 2025-08-14T21:25:13.9254968Z * [new branch] gh/eellison/805/head -> origin/gh/eellison/805/head 2025-08-14T21:25:13.9255101Z * [new branch] gh/eellison/805/orig -> origin/gh/eellison/805/orig 2025-08-14T21:25:13.9255239Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-08-14T21:25:13.9255373Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-08-14T21:25:13.9255516Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-08-14T21:25:13.9255651Z * [new branch] gh/eellison/809/base -> origin/gh/eellison/809/base 2025-08-14T21:25:13.9255785Z * [new branch] gh/eellison/809/head -> origin/gh/eellison/809/head 2025-08-14T21:25:13.9255922Z * [new branch] gh/eellison/809/orig -> origin/gh/eellison/809/orig 2025-08-14T21:25:13.9256059Z * [new branch] gh/eellison/810/base -> origin/gh/eellison/810/base 2025-08-14T21:25:13.9256204Z * [new branch] gh/eellison/810/head -> origin/gh/eellison/810/head 2025-08-14T21:25:13.9257118Z * [new branch] gh/eellison/810/orig -> origin/gh/eellison/810/orig 2025-08-14T21:25:13.9257888Z * [new branch] gh/eellison/811/base -> origin/gh/eellison/811/base 2025-08-14T21:25:13.9258474Z * [new branch] gh/eellison/811/head -> origin/gh/eellison/811/head 2025-08-14T21:25:13.9259017Z * [new branch] gh/eellison/811/orig -> origin/gh/eellison/811/orig 2025-08-14T21:25:13.9262241Z * [new branch] gh/eellison/812/base -> origin/gh/eellison/812/base 2025-08-14T21:25:13.9262400Z * [new branch] gh/eellison/812/head -> origin/gh/eellison/812/head 2025-08-14T21:25:13.9262538Z * [new branch] gh/eellison/812/orig -> origin/gh/eellison/812/orig 2025-08-14T21:25:13.9262665Z * [new branch] gh/eellison/813/base -> origin/gh/eellison/813/base 2025-08-14T21:25:13.9265369Z * [new branch] gh/eellison/813/head -> origin/gh/eellison/813/head 2025-08-14T21:25:13.9265656Z * [new branch] gh/eellison/813/orig -> origin/gh/eellison/813/orig 2025-08-14T21:25:13.9269176Z * [new branch] gh/etaf/132/base -> origin/gh/etaf/132/base 2025-08-14T21:25:13.9269470Z * [new branch] gh/etaf/132/head -> origin/gh/etaf/132/head 2025-08-14T21:25:13.9269690Z * [new branch] gh/etaf/132/orig -> origin/gh/etaf/132/orig 2025-08-14T21:25:13.9269865Z * [new branch] gh/etaf/138/base -> origin/gh/etaf/138/base 2025-08-14T21:25:13.9269995Z * [new branch] gh/etaf/138/head -> origin/gh/etaf/138/head 2025-08-14T21:25:13.9270131Z * [new branch] gh/etaf/138/orig -> origin/gh/etaf/138/orig 2025-08-14T21:25:13.9270253Z * [new branch] gh/etaf/140/base -> origin/gh/etaf/140/base 2025-08-14T21:25:13.9270507Z * [new branch] gh/etaf/140/head -> origin/gh/etaf/140/head 2025-08-14T21:25:13.9270650Z * [new branch] gh/etaf/140/orig -> origin/gh/etaf/140/orig 2025-08-14T21:25:13.9271631Z * [new branch] gh/etaf/143/base -> origin/gh/etaf/143/base 2025-08-14T21:25:13.9271965Z * [new branch] gh/etaf/143/head -> origin/gh/etaf/143/head 2025-08-14T21:25:13.9273011Z * [new branch] gh/etaf/143/orig -> origin/gh/etaf/143/orig 2025-08-14T21:25:13.9274096Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-08-14T21:25:13.9274948Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-08-14T21:25:13.9275556Z * [new branch] gh/etaf/148/base -> origin/gh/etaf/148/base 2025-08-14T21:25:13.9276233Z * [new branch] gh/etaf/148/head -> origin/gh/etaf/148/head 2025-08-14T21:25:13.9277107Z * [new branch] gh/etaf/148/orig -> origin/gh/etaf/148/orig 2025-08-14T21:25:13.9278389Z * [new branch] gh/etaf/149/base -> origin/gh/etaf/149/base 2025-08-14T21:25:13.9278522Z * [new branch] gh/etaf/149/head -> origin/gh/etaf/149/head 2025-08-14T21:25:13.9280897Z * [new branch] gh/etaf/149/orig -> origin/gh/etaf/149/orig 2025-08-14T21:25:13.9281229Z * [new branch] gh/etaf/150/base -> origin/gh/etaf/150/base 2025-08-14T21:25:13.9281486Z * [new branch] gh/etaf/150/head -> origin/gh/etaf/150/head 2025-08-14T21:25:13.9281647Z * [new branch] gh/etaf/150/orig -> origin/gh/etaf/150/orig 2025-08-14T21:25:13.9283072Z * [new branch] gh/etaf/151/base -> origin/gh/etaf/151/base 2025-08-14T21:25:13.9283355Z * [new branch] gh/etaf/151/head -> origin/gh/etaf/151/head 2025-08-14T21:25:13.9283850Z * [new branch] gh/etaf/151/orig -> origin/gh/etaf/151/orig 2025-08-14T21:25:13.9285617Z * [new branch] gh/etaf/152/base -> origin/gh/etaf/152/base 2025-08-14T21:25:13.9285940Z * [new branch] gh/etaf/152/head -> origin/gh/etaf/152/head 2025-08-14T21:25:13.9289288Z * [new branch] gh/etaf/152/orig -> origin/gh/etaf/152/orig 2025-08-14T21:25:13.9289616Z * [new branch] gh/etaf/153/base -> origin/gh/etaf/153/base 2025-08-14T21:25:13.9289987Z * [new branch] gh/etaf/153/head -> origin/gh/etaf/153/head 2025-08-14T21:25:13.9290264Z * [new branch] gh/etaf/153/orig -> origin/gh/etaf/153/orig 2025-08-14T21:25:13.9290415Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-08-14T21:25:13.9290561Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-08-14T21:25:13.9296024Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-08-14T21:25:13.9296205Z * [new branch] gh/etaf/155/base -> origin/gh/etaf/155/base 2025-08-14T21:25:13.9296335Z * [new branch] gh/etaf/155/head -> origin/gh/etaf/155/head 2025-08-14T21:25:13.9296471Z * [new branch] gh/etaf/155/orig -> origin/gh/etaf/155/orig 2025-08-14T21:25:13.9296622Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-08-14T21:25:13.9296780Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-08-14T21:25:13.9296938Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-08-14T21:25:13.9301484Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-08-14T21:25:13.9303211Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-08-14T21:25:13.9303374Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-08-14T21:25:13.9303501Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-08-14T21:25:13.9303625Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-08-14T21:25:13.9303761Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-08-14T21:25:13.9304043Z * [new branch] gh/ezyang/3068/base -> origin/gh/ezyang/3068/base 2025-08-14T21:25:13.9304205Z * [new branch] gh/ezyang/3068/head -> origin/gh/ezyang/3068/head 2025-08-14T21:25:13.9304585Z * [new branch] gh/ezyang/3068/orig -> origin/gh/ezyang/3068/orig 2025-08-14T21:25:13.9304741Z * [new branch] gh/ezyang/3071/base -> origin/gh/ezyang/3071/base 2025-08-14T21:25:13.9304974Z * [new branch] gh/ezyang/3071/head -> origin/gh/ezyang/3071/head 2025-08-14T21:25:13.9305117Z * [new branch] gh/ezyang/3071/orig -> origin/gh/ezyang/3071/orig 2025-08-14T21:25:13.9305329Z * [new branch] gh/ezyang/3074/base -> origin/gh/ezyang/3074/base 2025-08-14T21:25:13.9305471Z * [new branch] gh/ezyang/3074/head -> origin/gh/ezyang/3074/head 2025-08-14T21:25:13.9305681Z * [new branch] gh/ezyang/3074/orig -> origin/gh/ezyang/3074/orig 2025-08-14T21:25:13.9309006Z * [new branch] gh/ezyang/3088/base -> origin/gh/ezyang/3088/base 2025-08-14T21:25:13.9309284Z * [new branch] gh/ezyang/3088/head -> origin/gh/ezyang/3088/head 2025-08-14T21:25:13.9309502Z * [new branch] gh/ezyang/3088/orig -> origin/gh/ezyang/3088/orig 2025-08-14T21:25:13.9309716Z * [new branch] gh/ezyang/3092/base -> origin/gh/ezyang/3092/base 2025-08-14T21:25:13.9309958Z * [new branch] gh/ezyang/3092/head -> origin/gh/ezyang/3092/head 2025-08-14T21:25:13.9310106Z * [new branch] gh/ezyang/3092/orig -> origin/gh/ezyang/3092/orig 2025-08-14T21:25:13.9310322Z * [new branch] gh/ezyang/3097/base -> origin/gh/ezyang/3097/base 2025-08-14T21:25:13.9310458Z * [new branch] gh/ezyang/3097/head -> origin/gh/ezyang/3097/head 2025-08-14T21:25:13.9310595Z * [new branch] gh/ezyang/3097/orig -> origin/gh/ezyang/3097/orig 2025-08-14T21:25:13.9310726Z * [new branch] gh/ezyang/3098/base -> origin/gh/ezyang/3098/base 2025-08-14T21:25:13.9318001Z * [new branch] gh/ezyang/3098/head -> origin/gh/ezyang/3098/head 2025-08-14T21:25:13.9320493Z * [new branch] gh/ezyang/3098/orig -> origin/gh/ezyang/3098/orig 2025-08-14T21:25:13.9320763Z * [new branch] gh/ezyang/3099/base -> origin/gh/ezyang/3099/base 2025-08-14T21:25:13.9326001Z * [new branch] gh/ezyang/3099/head -> origin/gh/ezyang/3099/head 2025-08-14T21:25:13.9332310Z * [new branch] gh/ezyang/3099/orig -> origin/gh/ezyang/3099/orig 2025-08-14T21:25:13.9334351Z * [new branch] gh/ezyang/3100/base -> origin/gh/ezyang/3100/base 2025-08-14T21:25:13.9334703Z * [new branch] gh/ezyang/3100/head -> origin/gh/ezyang/3100/head 2025-08-14T21:25:13.9334966Z * [new branch] gh/ezyang/3100/orig -> origin/gh/ezyang/3100/orig 2025-08-14T21:25:13.9335252Z * [new branch] gh/ezyang/3101/base -> origin/gh/ezyang/3101/base 2025-08-14T21:25:13.9335474Z * [new branch] gh/ezyang/3101/head -> origin/gh/ezyang/3101/head 2025-08-14T21:25:13.9335636Z * [new branch] gh/ezyang/3101/orig -> origin/gh/ezyang/3101/orig 2025-08-14T21:25:13.9335846Z * [new branch] gh/ezyang/3102/base -> origin/gh/ezyang/3102/base 2025-08-14T21:25:13.9336065Z * [new branch] gh/ezyang/3102/head -> origin/gh/ezyang/3102/head 2025-08-14T21:25:13.9336210Z * [new branch] gh/ezyang/3102/orig -> origin/gh/ezyang/3102/orig 2025-08-14T21:25:13.9336359Z * [new branch] gh/ezyang/3103/base -> origin/gh/ezyang/3103/base 2025-08-14T21:25:13.9336485Z * [new branch] gh/ezyang/3103/head -> origin/gh/ezyang/3103/head 2025-08-14T21:25:13.9336744Z * [new branch] gh/ezyang/3103/orig -> origin/gh/ezyang/3103/orig 2025-08-14T21:25:13.9337393Z * [new branch] gh/ezyang/3104/base -> origin/gh/ezyang/3104/base 2025-08-14T21:25:13.9337788Z * [new branch] gh/ezyang/3104/head -> origin/gh/ezyang/3104/head 2025-08-14T21:25:13.9337931Z * [new branch] gh/ezyang/3104/orig -> origin/gh/ezyang/3104/orig 2025-08-14T21:25:13.9338062Z * [new branch] gh/ezyang/3105/base -> origin/gh/ezyang/3105/base 2025-08-14T21:25:13.9338192Z * [new branch] gh/ezyang/3105/head -> origin/gh/ezyang/3105/head 2025-08-14T21:25:13.9338327Z * [new branch] gh/ezyang/3105/orig -> origin/gh/ezyang/3105/orig 2025-08-14T21:25:13.9338458Z * [new branch] gh/ezyang/3106/base -> origin/gh/ezyang/3106/base 2025-08-14T21:25:13.9338595Z * [new branch] gh/ezyang/3106/head -> origin/gh/ezyang/3106/head 2025-08-14T21:25:13.9338723Z * [new branch] gh/ezyang/3106/orig -> origin/gh/ezyang/3106/orig 2025-08-14T21:25:13.9338855Z * [new branch] gh/ezyang/3107/base -> origin/gh/ezyang/3107/base 2025-08-14T21:25:13.9338993Z * [new branch] gh/ezyang/3107/head -> origin/gh/ezyang/3107/head 2025-08-14T21:25:13.9339122Z * [new branch] gh/ezyang/3107/orig -> origin/gh/ezyang/3107/orig 2025-08-14T21:25:13.9339250Z * [new branch] gh/ezyang/3108/base -> origin/gh/ezyang/3108/base 2025-08-14T21:25:13.9339385Z * [new branch] gh/ezyang/3108/head -> origin/gh/ezyang/3108/head 2025-08-14T21:25:13.9339514Z * [new branch] gh/ezyang/3108/orig -> origin/gh/ezyang/3108/orig 2025-08-14T21:25:13.9339674Z * [new branch] gh/ezyang/3109/base -> origin/gh/ezyang/3109/base 2025-08-14T21:25:13.9339805Z * [new branch] gh/ezyang/3109/head -> origin/gh/ezyang/3109/head 2025-08-14T21:25:13.9344672Z * [new branch] gh/ezyang/3109/orig -> origin/gh/ezyang/3109/orig 2025-08-14T21:25:13.9344988Z * [new branch] gh/ezyang/3110/base -> origin/gh/ezyang/3110/base 2025-08-14T21:25:13.9345159Z * [new branch] gh/ezyang/3110/head -> origin/gh/ezyang/3110/head 2025-08-14T21:25:13.9345299Z * [new branch] gh/ezyang/3110/orig -> origin/gh/ezyang/3110/orig 2025-08-14T21:25:13.9345428Z * [new branch] gh/ezyang/3111/base -> origin/gh/ezyang/3111/base 2025-08-14T21:25:13.9345560Z * [new branch] gh/ezyang/3111/head -> origin/gh/ezyang/3111/head 2025-08-14T21:25:13.9345698Z * [new branch] gh/ezyang/3111/orig -> origin/gh/ezyang/3111/orig 2025-08-14T21:25:13.9345829Z * [new branch] gh/ezyang/3112/base -> origin/gh/ezyang/3112/base 2025-08-14T21:25:13.9345967Z * [new branch] gh/ezyang/3112/head -> origin/gh/ezyang/3112/head 2025-08-14T21:25:13.9346097Z * [new branch] gh/ezyang/3112/orig -> origin/gh/ezyang/3112/orig 2025-08-14T21:25:13.9346226Z * [new branch] gh/ezyang/3113/base -> origin/gh/ezyang/3113/base 2025-08-14T21:25:13.9346365Z * [new branch] gh/ezyang/3113/head -> origin/gh/ezyang/3113/head 2025-08-14T21:25:13.9346491Z * [new branch] gh/ezyang/3113/orig -> origin/gh/ezyang/3113/orig 2025-08-14T21:25:13.9352349Z * [new branch] gh/ezyang/3114/base -> origin/gh/ezyang/3114/base 2025-08-14T21:25:13.9352681Z * [new branch] gh/ezyang/3114/head -> origin/gh/ezyang/3114/head 2025-08-14T21:25:13.9352908Z * [new branch] gh/ezyang/3114/orig -> origin/gh/ezyang/3114/orig 2025-08-14T21:25:13.9353054Z * [new branch] gh/ezyang/3115/base -> origin/gh/ezyang/3115/base 2025-08-14T21:25:13.9353184Z * [new branch] gh/ezyang/3115/head -> origin/gh/ezyang/3115/head 2025-08-14T21:25:13.9353470Z * [new branch] gh/ezyang/3115/orig -> origin/gh/ezyang/3115/orig 2025-08-14T21:25:13.9353639Z * [new branch] gh/ezyang/3116/base -> origin/gh/ezyang/3116/base 2025-08-14T21:25:13.9354050Z * [new branch] gh/ezyang/3116/head -> origin/gh/ezyang/3116/head 2025-08-14T21:25:13.9354280Z * [new branch] gh/ezyang/3116/orig -> origin/gh/ezyang/3116/orig 2025-08-14T21:25:13.9354949Z * [new branch] gh/ezyang/3117/base -> origin/gh/ezyang/3117/base 2025-08-14T21:25:13.9355125Z * [new branch] gh/ezyang/3117/head -> origin/gh/ezyang/3117/head 2025-08-14T21:25:13.9355272Z * [new branch] gh/ezyang/3117/orig -> origin/gh/ezyang/3117/orig 2025-08-14T21:25:13.9355409Z * [new branch] gh/ezyang/3118/base -> origin/gh/ezyang/3118/base 2025-08-14T21:25:13.9355544Z * [new branch] gh/ezyang/3118/head -> origin/gh/ezyang/3118/head 2025-08-14T21:25:13.9355710Z * [new branch] gh/ezyang/3118/orig -> origin/gh/ezyang/3118/orig 2025-08-14T21:25:13.9356616Z * [new branch] gh/ezyang/3119/base -> origin/gh/ezyang/3119/base 2025-08-14T21:25:13.9357126Z * [new branch] gh/ezyang/3119/head -> origin/gh/ezyang/3119/head 2025-08-14T21:25:13.9358013Z * [new branch] gh/ezyang/3119/orig -> origin/gh/ezyang/3119/orig 2025-08-14T21:25:13.9363107Z * [new branch] gh/ezyang/3120/base -> origin/gh/ezyang/3120/base 2025-08-14T21:25:13.9363281Z * [new branch] gh/ezyang/3120/head -> origin/gh/ezyang/3120/head 2025-08-14T21:25:13.9363414Z * [new branch] gh/ezyang/3120/orig -> origin/gh/ezyang/3120/orig 2025-08-14T21:25:13.9363553Z * [new branch] gh/ezyang/3121/base -> origin/gh/ezyang/3121/base 2025-08-14T21:25:13.9363682Z * [new branch] gh/ezyang/3121/head -> origin/gh/ezyang/3121/head 2025-08-14T21:25:13.9363972Z * [new branch] gh/ezyang/3121/orig -> origin/gh/ezyang/3121/orig 2025-08-14T21:25:13.9364113Z * [new branch] gh/ezyang/3122/base -> origin/gh/ezyang/3122/base 2025-08-14T21:25:13.9364418Z * [new branch] gh/ezyang/3122/head -> origin/gh/ezyang/3122/head 2025-08-14T21:25:13.9364573Z * [new branch] gh/ezyang/3122/orig -> origin/gh/ezyang/3122/orig 2025-08-14T21:25:13.9365574Z * [new branch] gh/ezyang/3123/base -> origin/gh/ezyang/3123/base 2025-08-14T21:25:13.9365903Z * [new branch] gh/ezyang/3123/head -> origin/gh/ezyang/3123/head 2025-08-14T21:25:13.9369680Z * [new branch] gh/ezyang/3123/orig -> origin/gh/ezyang/3123/orig 2025-08-14T21:25:13.9370004Z * [new branch] gh/ezyang/3124/base -> origin/gh/ezyang/3124/base 2025-08-14T21:25:13.9370219Z * [new branch] gh/ezyang/3124/head -> origin/gh/ezyang/3124/head 2025-08-14T21:25:13.9370453Z * [new branch] gh/ezyang/3124/orig -> origin/gh/ezyang/3124/orig 2025-08-14T21:25:13.9370605Z * [new branch] gh/ezyang/3125/base -> origin/gh/ezyang/3125/base 2025-08-14T21:25:13.9370824Z * [new branch] gh/ezyang/3125/head -> origin/gh/ezyang/3125/head 2025-08-14T21:25:13.9371786Z * [new branch] gh/ezyang/3125/orig -> origin/gh/ezyang/3125/orig 2025-08-14T21:25:13.9371944Z * [new branch] gh/ezyang/3126/base -> origin/gh/ezyang/3126/base 2025-08-14T21:25:13.9372180Z * [new branch] gh/ezyang/3126/head -> origin/gh/ezyang/3126/head 2025-08-14T21:25:13.9375225Z * [new branch] gh/ezyang/3126/orig -> origin/gh/ezyang/3126/orig 2025-08-14T21:25:13.9375553Z * [new branch] gh/ezyang/3127/base -> origin/gh/ezyang/3127/base 2025-08-14T21:25:13.9375781Z * [new branch] gh/ezyang/3127/head -> origin/gh/ezyang/3127/head 2025-08-14T21:25:13.9376045Z * [new branch] gh/ezyang/3127/orig -> origin/gh/ezyang/3127/orig 2025-08-14T21:25:13.9376442Z * [new branch] gh/ezyang/3128/base -> origin/gh/ezyang/3128/base 2025-08-14T21:25:13.9377098Z * [new branch] gh/ezyang/3128/head -> origin/gh/ezyang/3128/head 2025-08-14T21:25:13.9377430Z * [new branch] gh/ezyang/3128/orig -> origin/gh/ezyang/3128/orig 2025-08-14T21:25:13.9378781Z * [new branch] gh/ezyang/3129/base -> origin/gh/ezyang/3129/base 2025-08-14T21:25:13.9379112Z * [new branch] gh/ezyang/3129/head -> origin/gh/ezyang/3129/head 2025-08-14T21:25:13.9379330Z * [new branch] gh/ezyang/3129/orig -> origin/gh/ezyang/3129/orig 2025-08-14T21:25:13.9381846Z * [new branch] gh/ezyang/3130/base -> origin/gh/ezyang/3130/base 2025-08-14T21:25:13.9382172Z * [new branch] gh/ezyang/3130/head -> origin/gh/ezyang/3130/head 2025-08-14T21:25:13.9382402Z * [new branch] gh/ezyang/3130/orig -> origin/gh/ezyang/3130/orig 2025-08-14T21:25:13.9382636Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-08-14T21:25:13.9382779Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-08-14T21:25:13.9383438Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-08-14T21:25:13.9386727Z * [new branch] gh/ezyang/3132/base -> origin/gh/ezyang/3132/base 2025-08-14T21:25:13.9387050Z * [new branch] gh/ezyang/3132/head -> origin/gh/ezyang/3132/head 2025-08-14T21:25:13.9387279Z * [new branch] gh/ezyang/3132/orig -> origin/gh/ezyang/3132/orig 2025-08-14T21:25:13.9387432Z * [new branch] gh/ezyang/3133/base -> origin/gh/ezyang/3133/base 2025-08-14T21:25:13.9387655Z * [new branch] gh/ezyang/3133/head -> origin/gh/ezyang/3133/head 2025-08-14T21:25:13.9387938Z * [new branch] gh/ezyang/3133/orig -> origin/gh/ezyang/3133/orig 2025-08-14T21:25:13.9388478Z * [new branch] gh/ezyang/3134/base -> origin/gh/ezyang/3134/base 2025-08-14T21:25:13.9389445Z * [new branch] gh/ezyang/3134/head -> origin/gh/ezyang/3134/head 2025-08-14T21:25:13.9389719Z * [new branch] gh/ezyang/3134/orig -> origin/gh/ezyang/3134/orig 2025-08-14T21:25:13.9394177Z * [new branch] gh/ezyang/3135/base -> origin/gh/ezyang/3135/base 2025-08-14T21:25:13.9394569Z * [new branch] gh/ezyang/3135/head -> origin/gh/ezyang/3135/head 2025-08-14T21:25:13.9394715Z * [new branch] gh/ezyang/3135/orig -> origin/gh/ezyang/3135/orig 2025-08-14T21:25:13.9394871Z * [new branch] gh/ezyang/3136/base -> origin/gh/ezyang/3136/base 2025-08-14T21:25:13.9395041Z * [new branch] gh/ezyang/3136/head -> origin/gh/ezyang/3136/head 2025-08-14T21:25:13.9395222Z * [new branch] gh/ezyang/3136/orig -> origin/gh/ezyang/3136/orig 2025-08-14T21:25:13.9397302Z * [new branch] gh/fadara01/1/base -> origin/gh/fadara01/1/base 2025-08-14T21:25:13.9397636Z * [new branch] gh/fadara01/1/head -> origin/gh/fadara01/1/head 2025-08-14T21:25:13.9397795Z * [new branch] gh/fadara01/1/orig -> origin/gh/fadara01/1/orig 2025-08-14T21:25:13.9400493Z * [new branch] gh/fduwjj/168/base -> origin/gh/fduwjj/168/base 2025-08-14T21:25:13.9400832Z * [new branch] gh/fduwjj/168/head -> origin/gh/fduwjj/168/head 2025-08-14T21:25:13.9401072Z * [new branch] gh/fduwjj/168/orig -> origin/gh/fduwjj/168/orig 2025-08-14T21:25:13.9401451Z * [new branch] gh/fduwjj/169/base -> origin/gh/fduwjj/169/base 2025-08-14T21:25:13.9405936Z * [new branch] gh/fduwjj/169/head -> origin/gh/fduwjj/169/head 2025-08-14T21:25:13.9406416Z * [new branch] gh/fduwjj/169/orig -> origin/gh/fduwjj/169/orig 2025-08-14T21:25:13.9406668Z * [new branch] gh/fduwjj/170/base -> origin/gh/fduwjj/170/base 2025-08-14T21:25:13.9406951Z * [new branch] gh/fduwjj/170/head -> origin/gh/fduwjj/170/head 2025-08-14T21:25:13.9407110Z * [new branch] gh/fduwjj/170/orig -> origin/gh/fduwjj/170/orig 2025-08-14T21:25:13.9407326Z * [new branch] gh/fduwjj/171/base -> origin/gh/fduwjj/171/base 2025-08-14T21:25:13.9407977Z * [new branch] gh/fduwjj/171/head -> origin/gh/fduwjj/171/head 2025-08-14T21:25:13.9408158Z * [new branch] gh/fduwjj/171/orig -> origin/gh/fduwjj/171/orig 2025-08-14T21:25:13.9414150Z * [new branch] gh/fduwjj/172/base -> origin/gh/fduwjj/172/base 2025-08-14T21:25:13.9423401Z * [new branch] gh/fduwjj/172/head -> origin/gh/fduwjj/172/head 2025-08-14T21:25:13.9431331Z * [new branch] gh/fduwjj/172/orig -> origin/gh/fduwjj/172/orig 2025-08-14T21:25:13.9431638Z * [new branch] gh/fduwjj/173/base -> origin/gh/fduwjj/173/base 2025-08-14T21:25:13.9431887Z * [new branch] gh/fduwjj/173/head -> origin/gh/fduwjj/173/head 2025-08-14T21:25:13.9432110Z * [new branch] gh/fduwjj/173/orig -> origin/gh/fduwjj/173/orig 2025-08-14T21:25:13.9432264Z * [new branch] gh/fduwjj/174/base -> origin/gh/fduwjj/174/base 2025-08-14T21:25:13.9432513Z * [new branch] gh/fduwjj/174/head -> origin/gh/fduwjj/174/head 2025-08-14T21:25:13.9433144Z * [new branch] gh/fduwjj/174/orig -> origin/gh/fduwjj/174/orig 2025-08-14T21:25:13.9433328Z * [new branch] gh/fduwjj/175/base -> origin/gh/fduwjj/175/base 2025-08-14T21:25:13.9433786Z * [new branch] gh/fduwjj/175/head -> origin/gh/fduwjj/175/head 2025-08-14T21:25:13.9434018Z * [new branch] gh/fduwjj/175/orig -> origin/gh/fduwjj/175/orig 2025-08-14T21:25:13.9434184Z * [new branch] gh/fduwjj/176/base -> origin/gh/fduwjj/176/base 2025-08-14T21:25:13.9434330Z * [new branch] gh/fduwjj/176/head -> origin/gh/fduwjj/176/head 2025-08-14T21:25:13.9434718Z * [new branch] gh/fduwjj/176/orig -> origin/gh/fduwjj/176/orig 2025-08-14T21:25:13.9434861Z * [new branch] gh/fduwjj/177/base -> origin/gh/fduwjj/177/base 2025-08-14T21:25:13.9435002Z * [new branch] gh/fduwjj/177/head -> origin/gh/fduwjj/177/head 2025-08-14T21:25:13.9435148Z * [new branch] gh/fduwjj/177/orig -> origin/gh/fduwjj/177/orig 2025-08-14T21:25:13.9435288Z * [new branch] gh/fduwjj/178/base -> origin/gh/fduwjj/178/base 2025-08-14T21:25:13.9435433Z * [new branch] gh/fduwjj/178/head -> origin/gh/fduwjj/178/head 2025-08-14T21:25:13.9435577Z * [new branch] gh/fduwjj/178/orig -> origin/gh/fduwjj/178/orig 2025-08-14T21:25:13.9435722Z * [new branch] gh/fduwjj/179/base -> origin/gh/fduwjj/179/base 2025-08-14T21:25:13.9435870Z * [new branch] gh/fduwjj/179/head -> origin/gh/fduwjj/179/head 2025-08-14T21:25:13.9436015Z * [new branch] gh/fduwjj/179/orig -> origin/gh/fduwjj/179/orig 2025-08-14T21:25:13.9436402Z * [new branch] gh/fduwjj/180/base -> origin/gh/fduwjj/180/base 2025-08-14T21:25:13.9436563Z * [new branch] gh/fduwjj/180/head -> origin/gh/fduwjj/180/head 2025-08-14T21:25:13.9436709Z * [new branch] gh/fduwjj/180/orig -> origin/gh/fduwjj/180/orig 2025-08-14T21:25:13.9436870Z * [new branch] gh/fduwjj/181/base -> origin/gh/fduwjj/181/base 2025-08-14T21:25:13.9437027Z * [new branch] gh/fduwjj/181/head -> origin/gh/fduwjj/181/head 2025-08-14T21:25:13.9437233Z * [new branch] gh/fduwjj/181/orig -> origin/gh/fduwjj/181/orig 2025-08-14T21:25:13.9437385Z * [new branch] gh/fegin/306/base -> origin/gh/fegin/306/base 2025-08-14T21:25:13.9437523Z * [new branch] gh/fegin/306/head -> origin/gh/fegin/306/head 2025-08-14T21:25:13.9437678Z * [new branch] gh/fegin/306/orig -> origin/gh/fegin/306/orig 2025-08-14T21:25:13.9437825Z * [new branch] gh/fegin/307/base -> origin/gh/fegin/307/base 2025-08-14T21:25:13.9437967Z * [new branch] gh/fegin/307/head -> origin/gh/fegin/307/head 2025-08-14T21:25:13.9438106Z * [new branch] gh/fegin/307/orig -> origin/gh/fegin/307/orig 2025-08-14T21:25:13.9438414Z * [new branch] gh/fffrog/114/base -> origin/gh/fffrog/114/base 2025-08-14T21:25:13.9438583Z * [new branch] gh/fffrog/114/head -> origin/gh/fffrog/114/head 2025-08-14T21:25:13.9438806Z * [new branch] gh/fffrog/114/orig -> origin/gh/fffrog/114/orig 2025-08-14T21:25:13.9440877Z * [new branch] gh/fffrog/117/base -> origin/gh/fffrog/117/base 2025-08-14T21:25:13.9441052Z * [new branch] gh/fffrog/117/head -> origin/gh/fffrog/117/head 2025-08-14T21:25:13.9441183Z * [new branch] gh/fffrog/117/orig -> origin/gh/fffrog/117/orig 2025-08-14T21:25:13.9443330Z * [new branch] gh/fffrog/119/base -> origin/gh/fffrog/119/base 2025-08-14T21:25:13.9443619Z * [new branch] gh/fffrog/119/head -> origin/gh/fffrog/119/head 2025-08-14T21:25:13.9443846Z * [new branch] gh/fffrog/119/orig -> origin/gh/fffrog/119/orig 2025-08-14T21:25:13.9444181Z * [new branch] gh/fffrog/120/base -> origin/gh/fffrog/120/base 2025-08-14T21:25:13.9446523Z * [new branch] gh/fffrog/120/head -> origin/gh/fffrog/120/head 2025-08-14T21:25:13.9446882Z * [new branch] gh/fffrog/120/orig -> origin/gh/fffrog/120/orig 2025-08-14T21:25:13.9447114Z * [new branch] gh/fffrog/121/base -> origin/gh/fffrog/121/base 2025-08-14T21:25:13.9447261Z * [new branch] gh/fffrog/121/head -> origin/gh/fffrog/121/head 2025-08-14T21:25:13.9447736Z * [new branch] gh/fffrog/121/orig -> origin/gh/fffrog/121/orig 2025-08-14T21:25:13.9450190Z * [new branch] gh/fffrog/122/base -> origin/gh/fffrog/122/base 2025-08-14T21:25:13.9450526Z * [new branch] gh/fffrog/122/head -> origin/gh/fffrog/122/head 2025-08-14T21:25:13.9450751Z * [new branch] gh/fffrog/122/orig -> origin/gh/fffrog/122/orig 2025-08-14T21:25:13.9450903Z * [new branch] gh/fffrog/123/base -> origin/gh/fffrog/123/base 2025-08-14T21:25:13.9452077Z * [new branch] gh/fffrog/123/head -> origin/gh/fffrog/123/head 2025-08-14T21:25:13.9452421Z * [new branch] gh/fffrog/123/orig -> origin/gh/fffrog/123/orig 2025-08-14T21:25:13.9458628Z * [new branch] gh/fffrog/124/base -> origin/gh/fffrog/124/base 2025-08-14T21:25:13.9458951Z * [new branch] gh/fffrog/124/head -> origin/gh/fffrog/124/head 2025-08-14T21:25:13.9459187Z * [new branch] gh/fffrog/124/orig -> origin/gh/fffrog/124/orig 2025-08-14T21:25:13.9459339Z * [new branch] gh/fffrog/125/base -> origin/gh/fffrog/125/base 2025-08-14T21:25:13.9459467Z * [new branch] gh/fffrog/125/head -> origin/gh/fffrog/125/head 2025-08-14T21:25:13.9459604Z * [new branch] gh/fffrog/125/orig -> origin/gh/fffrog/125/orig 2025-08-14T21:25:13.9459876Z * [new branch] gh/fffrog/126/base -> origin/gh/fffrog/126/base 2025-08-14T21:25:13.9460047Z * [new branch] gh/fffrog/126/head -> origin/gh/fffrog/126/head 2025-08-14T21:25:13.9460483Z * [new branch] gh/fffrog/126/orig -> origin/gh/fffrog/126/orig 2025-08-14T21:25:13.9461109Z * [new branch] gh/fffrog/127/base -> origin/gh/fffrog/127/base 2025-08-14T21:25:13.9461289Z * [new branch] gh/fffrog/127/head -> origin/gh/fffrog/127/head 2025-08-14T21:25:13.9461423Z * [new branch] gh/fffrog/127/orig -> origin/gh/fffrog/127/orig 2025-08-14T21:25:13.9466042Z * [new branch] gh/fffrog/128/base -> origin/gh/fffrog/128/base 2025-08-14T21:25:13.9466391Z * [new branch] gh/fffrog/128/head -> origin/gh/fffrog/128/head 2025-08-14T21:25:13.9466629Z * [new branch] gh/fffrog/128/orig -> origin/gh/fffrog/128/orig 2025-08-14T21:25:13.9466798Z * [new branch] gh/fffrog/129/base -> origin/gh/fffrog/129/base 2025-08-14T21:25:13.9466950Z * [new branch] gh/fffrog/129/head -> origin/gh/fffrog/129/head 2025-08-14T21:25:13.9467087Z * [new branch] gh/fffrog/129/orig -> origin/gh/fffrog/129/orig 2025-08-14T21:25:13.9467351Z * [new branch] gh/fffrog/130/base -> origin/gh/fffrog/130/base 2025-08-14T21:25:13.9467996Z * [new branch] gh/fffrog/130/head -> origin/gh/fffrog/130/head 2025-08-14T21:25:13.9468195Z * [new branch] gh/fffrog/130/orig -> origin/gh/fffrog/130/orig 2025-08-14T21:25:13.9469540Z * [new branch] gh/fffrog/131/base -> origin/gh/fffrog/131/base 2025-08-14T21:25:13.9470004Z * [new branch] gh/fffrog/131/head -> origin/gh/fffrog/131/head 2025-08-14T21:25:13.9470590Z * [new branch] gh/fffrog/131/orig -> origin/gh/fffrog/131/orig 2025-08-14T21:25:13.9471823Z * [new branch] gh/fffrog/132/base -> origin/gh/fffrog/132/base 2025-08-14T21:25:13.9472157Z * [new branch] gh/fffrog/132/head -> origin/gh/fffrog/132/head 2025-08-14T21:25:13.9473726Z * [new branch] gh/fffrog/132/orig -> origin/gh/fffrog/132/orig 2025-08-14T21:25:13.9473975Z * [new branch] gh/fffrog/133/base -> origin/gh/fffrog/133/base 2025-08-14T21:25:13.9474568Z * [new branch] gh/fffrog/133/head -> origin/gh/fffrog/133/head 2025-08-14T21:25:13.9475401Z * [new branch] gh/fffrog/133/orig -> origin/gh/fffrog/133/orig 2025-08-14T21:25:13.9476697Z * [new branch] gh/fffrog/134/base -> origin/gh/fffrog/134/base 2025-08-14T21:25:13.9476940Z * [new branch] gh/fffrog/134/head -> origin/gh/fffrog/134/head 2025-08-14T21:25:13.9477915Z * [new branch] gh/fffrog/134/orig -> origin/gh/fffrog/134/orig 2025-08-14T21:25:13.9481247Z * [new branch] gh/fffrog/135/base -> origin/gh/fffrog/135/base 2025-08-14T21:25:13.9481442Z * [new branch] gh/fffrog/135/head -> origin/gh/fffrog/135/head 2025-08-14T21:25:13.9481578Z * [new branch] gh/fffrog/135/orig -> origin/gh/fffrog/135/orig 2025-08-14T21:25:13.9481722Z * [new branch] gh/fffrog/136/base -> origin/gh/fffrog/136/base 2025-08-14T21:25:13.9481859Z * [new branch] gh/fffrog/136/head -> origin/gh/fffrog/136/head 2025-08-14T21:25:13.9482455Z * [new branch] gh/fffrog/136/orig -> origin/gh/fffrog/136/orig 2025-08-14T21:25:13.9483926Z * [new branch] gh/fffrog/137/base -> origin/gh/fffrog/137/base 2025-08-14T21:25:13.9484071Z * [new branch] gh/fffrog/137/head -> origin/gh/fffrog/137/head 2025-08-14T21:25:13.9484421Z * [new branch] gh/fffrog/137/orig -> origin/gh/fffrog/137/orig 2025-08-14T21:25:13.9488690Z * [new branch] gh/fffrog/138/base -> origin/gh/fffrog/138/base 2025-08-14T21:25:13.9489030Z * [new branch] gh/fffrog/138/head -> origin/gh/fffrog/138/head 2025-08-14T21:25:13.9489175Z * [new branch] gh/fffrog/138/orig -> origin/gh/fffrog/138/orig 2025-08-14T21:25:13.9489331Z * [new branch] gh/gmagogsfm/1/base -> origin/gh/gmagogsfm/1/base 2025-08-14T21:25:13.9489480Z * [new branch] gh/gmagogsfm/1/head -> origin/gh/gmagogsfm/1/head 2025-08-14T21:25:13.9489797Z * [new branch] gh/gmagogsfm/1/orig -> origin/gh/gmagogsfm/1/orig 2025-08-14T21:25:13.9489962Z * [new branch] gh/gmagogsfm/2/base -> origin/gh/gmagogsfm/2/base 2025-08-14T21:25:13.9495821Z * [new branch] gh/gmagogsfm/2/head -> origin/gh/gmagogsfm/2/head 2025-08-14T21:25:13.9496170Z * [new branch] gh/gmagogsfm/2/orig -> origin/gh/gmagogsfm/2/orig 2025-08-14T21:25:13.9496452Z * [new branch] gh/gmagogsfm/3/base -> origin/gh/gmagogsfm/3/base 2025-08-14T21:25:13.9496691Z * [new branch] gh/gmagogsfm/3/head -> origin/gh/gmagogsfm/3/head 2025-08-14T21:25:13.9496956Z * [new branch] gh/gmagogsfm/3/orig -> origin/gh/gmagogsfm/3/orig 2025-08-14T21:25:13.9497236Z * [new branch] gh/gmagogsfm/4/base -> origin/gh/gmagogsfm/4/base 2025-08-14T21:25:13.9497461Z * [new branch] gh/gmagogsfm/4/head -> origin/gh/gmagogsfm/4/head 2025-08-14T21:25:13.9497685Z * [new branch] gh/gmagogsfm/4/orig -> origin/gh/gmagogsfm/4/orig 2025-08-14T21:25:13.9498295Z * [new branch] gh/guangyey/130/base -> origin/gh/guangyey/130/base 2025-08-14T21:25:13.9498480Z * [new branch] gh/guangyey/130/head -> origin/gh/guangyey/130/head 2025-08-14T21:25:13.9498640Z * [new branch] gh/guangyey/130/orig -> origin/gh/guangyey/130/orig 2025-08-14T21:25:13.9499478Z * [new branch] gh/guangyey/133/base -> origin/gh/guangyey/133/base 2025-08-14T21:25:13.9499918Z * [new branch] gh/guangyey/133/head -> origin/gh/guangyey/133/head 2025-08-14T21:25:13.9500780Z * [new branch] gh/guangyey/133/orig -> origin/gh/guangyey/133/orig 2025-08-14T21:25:13.9503775Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-08-14T21:25:13.9504109Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-08-14T21:25:13.9504355Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-08-14T21:25:13.9504512Z * [new branch] gh/guangyey/135/base -> origin/gh/guangyey/135/base 2025-08-14T21:25:13.9504758Z * [new branch] gh/guangyey/135/head -> origin/gh/guangyey/135/head 2025-08-14T21:25:13.9504912Z * [new branch] gh/guangyey/135/orig -> origin/gh/guangyey/135/orig 2025-08-14T21:25:13.9509561Z * [new branch] gh/guangyey/139/base -> origin/gh/guangyey/139/base 2025-08-14T21:25:13.9509759Z * [new branch] gh/guangyey/139/head -> origin/gh/guangyey/139/head 2025-08-14T21:25:13.9509913Z * [new branch] gh/guangyey/139/orig -> origin/gh/guangyey/139/orig 2025-08-14T21:25:13.9510068Z * [new branch] gh/guangyey/140/base -> origin/gh/guangyey/140/base 2025-08-14T21:25:13.9510213Z * [new branch] gh/guangyey/140/head -> origin/gh/guangyey/140/head 2025-08-14T21:25:13.9510360Z * [new branch] gh/guangyey/140/orig -> origin/gh/guangyey/140/orig 2025-08-14T21:25:13.9510849Z * [new branch] gh/guangyey/142/base -> origin/gh/guangyey/142/base 2025-08-14T21:25:13.9511716Z * [new branch] gh/guangyey/142/head -> origin/gh/guangyey/142/head 2025-08-14T21:25:13.9512120Z * [new branch] gh/guangyey/142/orig -> origin/gh/guangyey/142/orig 2025-08-14T21:25:13.9513416Z * [new branch] gh/guangyey/145/base -> origin/gh/guangyey/145/base 2025-08-14T21:25:13.9513736Z * [new branch] gh/guangyey/145/head -> origin/gh/guangyey/145/head 2025-08-14T21:25:13.9514766Z * [new branch] gh/guangyey/145/orig -> origin/gh/guangyey/145/orig 2025-08-14T21:25:13.9516578Z * [new branch] gh/guangyey/153/base -> origin/gh/guangyey/153/base 2025-08-14T21:25:13.9516981Z * [new branch] gh/guangyey/153/head -> origin/gh/guangyey/153/head 2025-08-14T21:25:13.9517519Z * [new branch] gh/guangyey/153/orig -> origin/gh/guangyey/153/orig 2025-08-14T21:25:13.9521280Z * [new branch] gh/guangyey/158/base -> origin/gh/guangyey/158/base 2025-08-14T21:25:13.9521470Z * [new branch] gh/guangyey/158/head -> origin/gh/guangyey/158/head 2025-08-14T21:25:13.9521649Z * [new branch] gh/guangyey/158/orig -> origin/gh/guangyey/158/orig 2025-08-14T21:25:13.9521814Z * [new branch] gh/guangyey/159/base -> origin/gh/guangyey/159/base 2025-08-14T21:25:13.9521960Z * [new branch] gh/guangyey/159/head -> origin/gh/guangyey/159/head 2025-08-14T21:25:13.9522277Z * [new branch] gh/guangyey/159/orig -> origin/gh/guangyey/159/orig 2025-08-14T21:25:13.9522823Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-08-14T21:25:13.9524806Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-08-14T21:25:13.9525145Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-08-14T21:25:13.9525389Z * [new branch] gh/guangyey/165/base -> origin/gh/guangyey/165/base 2025-08-14T21:25:13.9525714Z * [new branch] gh/guangyey/165/head -> origin/gh/guangyey/165/head 2025-08-14T21:25:13.9528105Z * [new branch] gh/guangyey/165/orig -> origin/gh/guangyey/165/orig 2025-08-14T21:25:13.9528491Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-08-14T21:25:13.9528743Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-08-14T21:25:13.9528905Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-08-14T21:25:13.9529601Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-08-14T21:25:13.9530276Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-08-14T21:25:13.9531018Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-08-14T21:25:13.9532424Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-08-14T21:25:13.9532576Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-08-14T21:25:13.9533091Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-08-14T21:25:13.9537727Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-08-14T21:25:13.9537917Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-08-14T21:25:13.9538073Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-08-14T21:25:13.9538213Z * [new branch] gh/guangyey/172/base -> origin/gh/guangyey/172/base 2025-08-14T21:25:13.9538364Z * [new branch] gh/guangyey/172/head -> origin/gh/guangyey/172/head 2025-08-14T21:25:13.9538500Z * [new branch] gh/guangyey/172/orig -> origin/gh/guangyey/172/orig 2025-08-14T21:25:13.9538676Z * [new branch] gh/guangyey/173/base -> origin/gh/guangyey/173/base 2025-08-14T21:25:13.9538853Z * [new branch] gh/guangyey/173/head -> origin/gh/guangyey/173/head 2025-08-14T21:25:13.9543264Z * [new branch] gh/guangyey/173/orig -> origin/gh/guangyey/173/orig 2025-08-14T21:25:13.9543462Z * [new branch] gh/guangyey/174/base -> origin/gh/guangyey/174/base 2025-08-14T21:25:13.9543605Z * [new branch] gh/guangyey/174/head -> origin/gh/guangyey/174/head 2025-08-14T21:25:13.9543744Z * [new branch] gh/guangyey/174/orig -> origin/gh/guangyey/174/orig 2025-08-14T21:25:13.9543887Z * [new branch] gh/guangyey/175/base -> origin/gh/guangyey/175/base 2025-08-14T21:25:13.9544032Z * [new branch] gh/guangyey/175/head -> origin/gh/guangyey/175/head 2025-08-14T21:25:13.9545254Z * [new branch] gh/guangyey/175/orig -> origin/gh/guangyey/175/orig 2025-08-14T21:25:13.9546042Z * [new branch] gh/guangyey/176/base -> origin/gh/guangyey/176/base 2025-08-14T21:25:13.9546314Z * [new branch] gh/guangyey/176/head -> origin/gh/guangyey/176/head 2025-08-14T21:25:13.9546473Z * [new branch] gh/guangyey/176/orig -> origin/gh/guangyey/176/orig 2025-08-14T21:25:13.9547574Z * [new branch] gh/guangyey/177/base -> origin/gh/guangyey/177/base 2025-08-14T21:25:13.9547911Z * [new branch] gh/guangyey/177/head -> origin/gh/guangyey/177/head 2025-08-14T21:25:13.9550062Z * [new branch] gh/guangyey/177/orig -> origin/gh/guangyey/177/orig 2025-08-14T21:25:13.9550253Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-08-14T21:25:13.9550403Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-08-14T21:25:13.9550986Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-08-14T21:25:13.9552125Z * [new branch] gh/guangyey/179/base -> origin/gh/guangyey/179/base 2025-08-14T21:25:13.9552787Z * [new branch] gh/guangyey/179/head -> origin/gh/guangyey/179/head 2025-08-14T21:25:13.9553226Z * [new branch] gh/guangyey/179/orig -> origin/gh/guangyey/179/orig 2025-08-14T21:25:13.9554468Z * [new branch] gh/guangyey/180/base -> origin/gh/guangyey/180/base 2025-08-14T21:25:13.9555008Z * [new branch] gh/guangyey/180/head -> origin/gh/guangyey/180/head 2025-08-14T21:25:13.9555711Z * [new branch] gh/guangyey/180/orig -> origin/gh/guangyey/180/orig 2025-08-14T21:25:13.9557438Z * [new branch] gh/guangyey/181/base -> origin/gh/guangyey/181/base 2025-08-14T21:25:13.9557619Z * [new branch] gh/guangyey/181/head -> origin/gh/guangyey/181/head 2025-08-14T21:25:13.9557875Z * [new branch] gh/guangyey/181/orig -> origin/gh/guangyey/181/orig 2025-08-14T21:25:13.9565419Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-08-14T21:25:13.9565628Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-08-14T21:25:13.9565794Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-08-14T21:25:13.9565937Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-08-14T21:25:13.9566088Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-08-14T21:25:13.9566230Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-08-14T21:25:13.9566368Z * [new branch] gh/guangyey/184/base -> origin/gh/guangyey/184/base 2025-08-14T21:25:13.9566515Z * [new branch] gh/guangyey/184/head -> origin/gh/guangyey/184/head 2025-08-14T21:25:13.9566655Z * [new branch] gh/guangyey/184/orig -> origin/gh/guangyey/184/orig 2025-08-14T21:25:13.9566799Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-08-14T21:25:13.9567101Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-08-14T21:25:13.9567246Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-08-14T21:25:13.9567450Z * [new branch] gh/guangyey/79/base -> origin/gh/guangyey/79/base 2025-08-14T21:25:13.9569850Z * [new branch] gh/guangyey/79/head -> origin/gh/guangyey/79/head 2025-08-14T21:25:13.9570203Z * [new branch] gh/guangyey/79/orig -> origin/gh/guangyey/79/orig 2025-08-14T21:25:13.9570447Z * [new branch] gh/guangyey/89/base -> origin/gh/guangyey/89/base 2025-08-14T21:25:13.9570609Z * [new branch] gh/guangyey/89/head -> origin/gh/guangyey/89/head 2025-08-14T21:25:13.9571568Z * [new branch] gh/guangyey/89/orig -> origin/gh/guangyey/89/orig 2025-08-14T21:25:13.9574704Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-08-14T21:25:13.9575094Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-08-14T21:25:13.9575384Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-08-14T21:25:13.9575571Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-08-14T21:25:13.9575756Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-08-14T21:25:13.9577219Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-08-14T21:25:13.9577512Z * [new branch] gh/guilhermeleobas/124/base -> origin/gh/guilhermeleobas/124/base 2025-08-14T21:25:13.9579749Z * [new branch] gh/guilhermeleobas/124/head -> origin/gh/guilhermeleobas/124/head 2025-08-14T21:25:13.9580302Z * [new branch] gh/guilhermeleobas/124/orig -> origin/gh/guilhermeleobas/124/orig 2025-08-14T21:25:13.9580638Z * [new branch] gh/guilhermeleobas/147/base -> origin/gh/guilhermeleobas/147/base 2025-08-14T21:25:13.9580918Z * [new branch] gh/guilhermeleobas/147/head -> origin/gh/guilhermeleobas/147/head 2025-08-14T21:25:13.9581112Z * [new branch] gh/guilhermeleobas/147/orig -> origin/gh/guilhermeleobas/147/orig 2025-08-14T21:25:13.9587498Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-08-14T21:25:13.9587853Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-08-14T21:25:13.9588127Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-08-14T21:25:13.9588399Z * [new branch] gh/guilhermeleobas/163/base -> origin/gh/guilhermeleobas/163/base 2025-08-14T21:25:13.9588647Z * [new branch] gh/guilhermeleobas/163/head -> origin/gh/guilhermeleobas/163/head 2025-08-14T21:25:13.9588856Z * [new branch] gh/guilhermeleobas/163/orig -> origin/gh/guilhermeleobas/163/orig 2025-08-14T21:25:13.9589136Z * [new branch] gh/guilhermeleobas/164/base -> origin/gh/guilhermeleobas/164/base 2025-08-14T21:25:13.9589822Z * [new branch] gh/guilhermeleobas/164/head -> origin/gh/guilhermeleobas/164/head 2025-08-14T21:25:13.9590042Z * [new branch] gh/guilhermeleobas/164/orig -> origin/gh/guilhermeleobas/164/orig 2025-08-14T21:25:13.9590222Z * [new branch] gh/guilhermeleobas/165/base -> origin/gh/guilhermeleobas/165/base 2025-08-14T21:25:13.9590384Z * [new branch] gh/guilhermeleobas/165/head -> origin/gh/guilhermeleobas/165/head 2025-08-14T21:25:13.9590562Z * [new branch] gh/guilhermeleobas/165/orig -> origin/gh/guilhermeleobas/165/orig 2025-08-14T21:25:13.9591547Z * [new branch] gh/guilhermeleobas/166/base -> origin/gh/guilhermeleobas/166/base 2025-08-14T21:25:13.9591919Z * [new branch] gh/guilhermeleobas/166/head -> origin/gh/guilhermeleobas/166/head 2025-08-14T21:25:13.9593116Z * [new branch] gh/guilhermeleobas/166/orig -> origin/gh/guilhermeleobas/166/orig 2025-08-14T21:25:13.9593493Z * [new branch] gh/guilhermeleobas/167/base -> origin/gh/guilhermeleobas/167/base 2025-08-14T21:25:13.9594556Z * [new branch] gh/guilhermeleobas/167/head -> origin/gh/guilhermeleobas/167/head 2025-08-14T21:25:13.9595452Z * [new branch] gh/guilhermeleobas/167/orig -> origin/gh/guilhermeleobas/167/orig 2025-08-14T21:25:13.9595932Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-08-14T21:25:13.9596729Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-08-14T21:25:13.9597699Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-08-14T21:25:13.9598653Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-08-14T21:25:13.9598981Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-08-14T21:25:13.9600207Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-08-14T21:25:13.9602904Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-08-14T21:25:13.9603484Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-08-14T21:25:13.9603678Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-08-14T21:25:13.9603845Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-08-14T21:25:13.9604027Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-08-14T21:25:13.9605000Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-08-14T21:25:13.9605537Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-08-14T21:25:13.9606343Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-08-14T21:25:13.9606914Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-08-14T21:25:13.9611170Z * [new branch] gh/guilhermeleobas/181/base -> origin/gh/guilhermeleobas/181/base 2025-08-14T21:25:13.9611389Z * [new branch] gh/guilhermeleobas/181/head -> origin/gh/guilhermeleobas/181/head 2025-08-14T21:25:13.9615879Z * [new branch] gh/guilhermeleobas/181/orig -> origin/gh/guilhermeleobas/181/orig 2025-08-14T21:25:13.9618023Z * [new branch] gh/guilhermeleobas/182/base -> origin/gh/guilhermeleobas/182/base 2025-08-14T21:25:13.9618241Z * [new branch] gh/guilhermeleobas/182/head -> origin/gh/guilhermeleobas/182/head 2025-08-14T21:25:13.9618447Z * [new branch] gh/guilhermeleobas/182/orig -> origin/gh/guilhermeleobas/182/orig 2025-08-14T21:25:13.9618645Z * [new branch] gh/guilhermeleobas/183/base -> origin/gh/guilhermeleobas/183/base 2025-08-14T21:25:13.9618818Z * [new branch] gh/guilhermeleobas/183/head -> origin/gh/guilhermeleobas/183/head 2025-08-14T21:25:13.9624244Z * [new branch] gh/guilhermeleobas/183/orig -> origin/gh/guilhermeleobas/183/orig 2025-08-14T21:25:13.9628797Z * [new branch] gh/guilhermeleobas/184/base -> origin/gh/guilhermeleobas/184/base 2025-08-14T21:25:13.9629024Z * [new branch] gh/guilhermeleobas/184/head -> origin/gh/guilhermeleobas/184/head 2025-08-14T21:25:13.9629235Z * [new branch] gh/guilhermeleobas/184/orig -> origin/gh/guilhermeleobas/184/orig 2025-08-14T21:25:13.9629423Z * [new branch] gh/guilhermeleobas/185/base -> origin/gh/guilhermeleobas/185/base 2025-08-14T21:25:13.9629606Z * [new branch] gh/guilhermeleobas/185/head -> origin/gh/guilhermeleobas/185/head 2025-08-14T21:25:13.9630000Z * [new branch] gh/guilhermeleobas/185/orig -> origin/gh/guilhermeleobas/185/orig 2025-08-14T21:25:13.9630175Z * [new branch] gh/guilhermeleobas/188/base -> origin/gh/guilhermeleobas/188/base 2025-08-14T21:25:13.9630348Z * [new branch] gh/guilhermeleobas/188/head -> origin/gh/guilhermeleobas/188/head 2025-08-14T21:25:13.9630518Z * [new branch] gh/guilhermeleobas/188/orig -> origin/gh/guilhermeleobas/188/orig 2025-08-14T21:25:13.9630687Z * [new branch] gh/guilhermeleobas/189/base -> origin/gh/guilhermeleobas/189/base 2025-08-14T21:25:13.9630863Z * [new branch] gh/guilhermeleobas/189/head -> origin/gh/guilhermeleobas/189/head 2025-08-14T21:25:13.9631064Z * [new branch] gh/guilhermeleobas/189/orig -> origin/gh/guilhermeleobas/189/orig 2025-08-14T21:25:13.9631246Z * [new branch] gh/guilhermeleobas/190/base -> origin/gh/guilhermeleobas/190/base 2025-08-14T21:25:13.9631427Z * [new branch] gh/guilhermeleobas/190/head -> origin/gh/guilhermeleobas/190/head 2025-08-14T21:25:13.9631595Z * [new branch] gh/guilhermeleobas/190/orig -> origin/gh/guilhermeleobas/190/orig 2025-08-14T21:25:13.9631771Z * [new branch] gh/guilhermeleobas/192/base -> origin/gh/guilhermeleobas/192/base 2025-08-14T21:25:13.9631946Z * [new branch] gh/guilhermeleobas/192/head -> origin/gh/guilhermeleobas/192/head 2025-08-14T21:25:13.9632308Z * [new branch] gh/guilhermeleobas/192/orig -> origin/gh/guilhermeleobas/192/orig 2025-08-14T21:25:13.9633824Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-08-14T21:25:13.9634011Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-08-14T21:25:13.9635060Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-08-14T21:25:13.9635721Z * [new branch] gh/guilhermeleobas/194/base -> origin/gh/guilhermeleobas/194/base 2025-08-14T21:25:13.9636393Z * [new branch] gh/guilhermeleobas/194/head -> origin/gh/guilhermeleobas/194/head 2025-08-14T21:25:13.9637338Z * [new branch] gh/guilhermeleobas/194/orig -> origin/gh/guilhermeleobas/194/orig 2025-08-14T21:25:13.9638517Z * [new branch] gh/guilhermeleobas/203/base -> origin/gh/guilhermeleobas/203/base 2025-08-14T21:25:13.9639041Z * [new branch] gh/guilhermeleobas/203/head -> origin/gh/guilhermeleobas/203/head 2025-08-14T21:25:13.9639753Z * [new branch] gh/guilhermeleobas/203/orig -> origin/gh/guilhermeleobas/203/orig 2025-08-14T21:25:13.9640886Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-08-14T21:25:13.9641940Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-08-14T21:25:13.9642202Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-08-14T21:25:13.9643539Z * [new branch] gh/guilhermeleobas/205/base -> origin/gh/guilhermeleobas/205/base 2025-08-14T21:25:13.9644273Z * [new branch] gh/guilhermeleobas/205/head -> origin/gh/guilhermeleobas/205/head 2025-08-14T21:25:13.9644530Z * [new branch] gh/guilhermeleobas/205/orig -> origin/gh/guilhermeleobas/205/orig 2025-08-14T21:25:13.9650164Z * [new branch] gh/guilhermeleobas/206/base -> origin/gh/guilhermeleobas/206/base 2025-08-14T21:25:13.9650393Z * [new branch] gh/guilhermeleobas/206/head -> origin/gh/guilhermeleobas/206/head 2025-08-14T21:25:13.9650577Z * [new branch] gh/guilhermeleobas/206/orig -> origin/gh/guilhermeleobas/206/orig 2025-08-14T21:25:13.9650758Z * [new branch] gh/guilhermeleobas/207/base -> origin/gh/guilhermeleobas/207/base 2025-08-14T21:25:13.9650970Z * [new branch] gh/guilhermeleobas/207/head -> origin/gh/guilhermeleobas/207/head 2025-08-14T21:25:13.9651317Z * [new branch] gh/guilhermeleobas/207/orig -> origin/gh/guilhermeleobas/207/orig 2025-08-14T21:25:13.9651502Z * [new branch] gh/guilhermeleobas/208/base -> origin/gh/guilhermeleobas/208/base 2025-08-14T21:25:13.9651679Z * [new branch] gh/guilhermeleobas/208/head -> origin/gh/guilhermeleobas/208/head 2025-08-14T21:25:13.9651863Z * [new branch] gh/guilhermeleobas/208/orig -> origin/gh/guilhermeleobas/208/orig 2025-08-14T21:25:13.9652459Z * [new branch] gh/guilhermeleobas/209/base -> origin/gh/guilhermeleobas/209/base 2025-08-14T21:25:13.9652992Z * [new branch] gh/guilhermeleobas/209/head -> origin/gh/guilhermeleobas/209/head 2025-08-14T21:25:13.9654072Z * [new branch] gh/guilhermeleobas/209/orig -> origin/gh/guilhermeleobas/209/orig 2025-08-14T21:25:13.9655067Z * [new branch] gh/guilhermeleobas/210/base -> origin/gh/guilhermeleobas/210/base 2025-08-14T21:25:13.9655515Z * [new branch] gh/guilhermeleobas/210/head -> origin/gh/guilhermeleobas/210/head 2025-08-14T21:25:13.9656381Z * [new branch] gh/guilhermeleobas/210/orig -> origin/gh/guilhermeleobas/210/orig 2025-08-14T21:25:13.9657408Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-08-14T21:25:13.9657808Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-08-14T21:25:13.9658376Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-08-14T21:25:13.9660310Z * [new branch] gh/guilhermeleobas/212/base -> origin/gh/guilhermeleobas/212/base 2025-08-14T21:25:13.9660536Z * [new branch] gh/guilhermeleobas/212/head -> origin/gh/guilhermeleobas/212/head 2025-08-14T21:25:13.9660716Z * [new branch] gh/guilhermeleobas/212/orig -> origin/gh/guilhermeleobas/212/orig 2025-08-14T21:25:13.9662550Z * [new branch] gh/guilhermeleobas/213/base -> origin/gh/guilhermeleobas/213/base 2025-08-14T21:25:13.9662784Z * [new branch] gh/guilhermeleobas/213/head -> origin/gh/guilhermeleobas/213/head 2025-08-14T21:25:13.9662956Z * [new branch] gh/guilhermeleobas/213/orig -> origin/gh/guilhermeleobas/213/orig 2025-08-14T21:25:13.9668215Z * [new branch] gh/guilhermeleobas/214/base -> origin/gh/guilhermeleobas/214/base 2025-08-14T21:25:13.9668429Z * [new branch] gh/guilhermeleobas/214/head -> origin/gh/guilhermeleobas/214/head 2025-08-14T21:25:13.9668595Z * [new branch] gh/guilhermeleobas/214/orig -> origin/gh/guilhermeleobas/214/orig 2025-08-14T21:25:13.9668770Z * [new branch] gh/guilhermeleobas/215/base -> origin/gh/guilhermeleobas/215/base 2025-08-14T21:25:13.9668935Z * [new branch] gh/guilhermeleobas/215/head -> origin/gh/guilhermeleobas/215/head 2025-08-14T21:25:13.9669113Z * [new branch] gh/guilhermeleobas/215/orig -> origin/gh/guilhermeleobas/215/orig 2025-08-14T21:25:13.9670767Z * [new branch] gh/guilhermeleobas/216/base -> origin/gh/guilhermeleobas/216/base 2025-08-14T21:25:13.9670944Z * [new branch] gh/guilhermeleobas/216/head -> origin/gh/guilhermeleobas/216/head 2025-08-14T21:25:13.9671107Z * [new branch] gh/guilhermeleobas/216/orig -> origin/gh/guilhermeleobas/216/orig 2025-08-14T21:25:13.9671303Z * [new branch] gh/guilhermeleobas/217/base -> origin/gh/guilhermeleobas/217/base 2025-08-14T21:25:13.9671477Z * [new branch] gh/guilhermeleobas/217/head -> origin/gh/guilhermeleobas/217/head 2025-08-14T21:25:13.9671835Z * [new branch] gh/guilhermeleobas/217/orig -> origin/gh/guilhermeleobas/217/orig 2025-08-14T21:25:13.9672866Z * [new branch] gh/guilhermeleobas/218/base -> origin/gh/guilhermeleobas/218/base 2025-08-14T21:25:13.9673170Z * [new branch] gh/guilhermeleobas/218/head -> origin/gh/guilhermeleobas/218/head 2025-08-14T21:25:13.9673998Z * [new branch] gh/guilhermeleobas/218/orig -> origin/gh/guilhermeleobas/218/orig 2025-08-14T21:25:13.9675212Z * [new branch] gh/guilhermeleobas/219/base -> origin/gh/guilhermeleobas/219/base 2025-08-14T21:25:13.9675532Z * [new branch] gh/guilhermeleobas/219/head -> origin/gh/guilhermeleobas/219/head 2025-08-14T21:25:13.9680178Z * [new branch] gh/guilhermeleobas/219/orig -> origin/gh/guilhermeleobas/219/orig 2025-08-14T21:25:13.9680564Z * [new branch] gh/guilhermeleobas/220/base -> origin/gh/guilhermeleobas/220/base 2025-08-14T21:25:13.9680781Z * [new branch] gh/guilhermeleobas/220/head -> origin/gh/guilhermeleobas/220/head 2025-08-14T21:25:13.9680990Z * [new branch] gh/guilhermeleobas/220/orig -> origin/gh/guilhermeleobas/220/orig 2025-08-14T21:25:13.9681272Z * [new branch] gh/guilhermeleobas/221/base -> origin/gh/guilhermeleobas/221/base 2025-08-14T21:25:13.9681479Z * [new branch] gh/guilhermeleobas/221/head -> origin/gh/guilhermeleobas/221/head 2025-08-14T21:25:13.9682138Z * [new branch] gh/guilhermeleobas/221/orig -> origin/gh/guilhermeleobas/221/orig 2025-08-14T21:25:13.9682647Z * [new branch] gh/guilhermeleobas/222/base -> origin/gh/guilhermeleobas/222/base 2025-08-14T21:25:13.9686369Z * [new branch] gh/guilhermeleobas/222/head -> origin/gh/guilhermeleobas/222/head 2025-08-14T21:25:13.9687009Z * [new branch] gh/guilhermeleobas/222/orig -> origin/gh/guilhermeleobas/222/orig 2025-08-14T21:25:13.9687227Z * [new branch] gh/guilhermeleobas/223/base -> origin/gh/guilhermeleobas/223/base 2025-08-14T21:25:13.9687409Z * [new branch] gh/guilhermeleobas/223/head -> origin/gh/guilhermeleobas/223/head 2025-08-14T21:25:13.9687579Z * [new branch] gh/guilhermeleobas/223/orig -> origin/gh/guilhermeleobas/223/orig 2025-08-14T21:25:13.9687903Z * [new branch] gh/guilhermeleobas/224/base -> origin/gh/guilhermeleobas/224/base 2025-08-14T21:25:13.9688192Z * [new branch] gh/guilhermeleobas/224/head -> origin/gh/guilhermeleobas/224/head 2025-08-14T21:25:13.9688562Z * [new branch] gh/guilhermeleobas/224/orig -> origin/gh/guilhermeleobas/224/orig 2025-08-14T21:25:13.9691491Z * [new branch] gh/guilhermeleobas/225/base -> origin/gh/guilhermeleobas/225/base 2025-08-14T21:25:13.9691699Z * [new branch] gh/guilhermeleobas/225/head -> origin/gh/guilhermeleobas/225/head 2025-08-14T21:25:13.9691866Z * [new branch] gh/guilhermeleobas/225/orig -> origin/gh/guilhermeleobas/225/orig 2025-08-14T21:25:13.9692044Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-08-14T21:25:13.9692590Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-08-14T21:25:13.9693152Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-08-14T21:25:13.9696482Z * [new branch] gh/guilhermeleobas/227/base -> origin/gh/guilhermeleobas/227/base 2025-08-14T21:25:13.9696871Z * [new branch] gh/guilhermeleobas/227/head -> origin/gh/guilhermeleobas/227/head 2025-08-14T21:25:13.9697148Z * [new branch] gh/guilhermeleobas/227/orig -> origin/gh/guilhermeleobas/227/orig 2025-08-14T21:25:13.9697348Z * [new branch] gh/guilhermeleobas/228/base -> origin/gh/guilhermeleobas/228/base 2025-08-14T21:25:13.9697609Z * [new branch] gh/guilhermeleobas/228/head -> origin/gh/guilhermeleobas/228/head 2025-08-14T21:25:13.9698392Z * [new branch] gh/guilhermeleobas/228/orig -> origin/gh/guilhermeleobas/228/orig 2025-08-14T21:25:13.9699523Z * [new branch] gh/guilhermeleobas/229/base -> origin/gh/guilhermeleobas/229/base 2025-08-14T21:25:13.9699833Z * [new branch] gh/guilhermeleobas/229/head -> origin/gh/guilhermeleobas/229/head 2025-08-14T21:25:13.9700394Z * [new branch] gh/guilhermeleobas/229/orig -> origin/gh/guilhermeleobas/229/orig 2025-08-14T21:25:13.9702807Z * [new branch] gh/guilhermeleobas/230/base -> origin/gh/guilhermeleobas/230/base 2025-08-14T21:25:13.9703150Z * [new branch] gh/guilhermeleobas/230/head -> origin/gh/guilhermeleobas/230/head 2025-08-14T21:25:13.9703361Z * [new branch] gh/guilhermeleobas/230/orig -> origin/gh/guilhermeleobas/230/orig 2025-08-14T21:25:13.9704761Z * [new branch] gh/guilhermeleobas/231/base -> origin/gh/guilhermeleobas/231/base 2025-08-14T21:25:13.9705126Z * [new branch] gh/guilhermeleobas/231/head -> origin/gh/guilhermeleobas/231/head 2025-08-14T21:25:13.9705481Z * [new branch] gh/guilhermeleobas/231/orig -> origin/gh/guilhermeleobas/231/orig 2025-08-14T21:25:13.9707836Z * [new branch] gh/guilhermeleobas/232/base -> origin/gh/guilhermeleobas/232/base 2025-08-14T21:25:13.9708201Z * [new branch] gh/guilhermeleobas/232/head -> origin/gh/guilhermeleobas/232/head 2025-08-14T21:25:13.9708415Z * [new branch] gh/guilhermeleobas/232/orig -> origin/gh/guilhermeleobas/232/orig 2025-08-14T21:25:13.9708850Z * [new branch] gh/guilhermeleobas/233/base -> origin/gh/guilhermeleobas/233/base 2025-08-14T21:25:13.9710758Z * [new branch] gh/guilhermeleobas/233/head -> origin/gh/guilhermeleobas/233/head 2025-08-14T21:25:13.9710979Z * [new branch] gh/guilhermeleobas/233/orig -> origin/gh/guilhermeleobas/233/orig 2025-08-14T21:25:13.9711517Z * [new branch] gh/guilhermeleobas/73/base -> origin/gh/guilhermeleobas/73/base 2025-08-14T21:25:13.9712189Z * [new branch] gh/guilhermeleobas/73/head -> origin/gh/guilhermeleobas/73/head 2025-08-14T21:25:13.9713250Z * [new branch] gh/guilhermeleobas/73/orig -> origin/gh/guilhermeleobas/73/orig 2025-08-14T21:25:13.9714580Z * [new branch] gh/henrylhtsang/103/base -> origin/gh/henrylhtsang/103/base 2025-08-14T21:25:13.9714755Z * [new branch] gh/henrylhtsang/103/head -> origin/gh/henrylhtsang/103/head 2025-08-14T21:25:13.9715878Z * [new branch] gh/henrylhtsang/103/orig -> origin/gh/henrylhtsang/103/orig 2025-08-14T21:25:13.9716603Z * [new branch] gh/henrylhtsang/108/base -> origin/gh/henrylhtsang/108/base 2025-08-14T21:25:13.9720949Z * [new branch] gh/henrylhtsang/108/head -> origin/gh/henrylhtsang/108/head 2025-08-14T21:25:13.9721309Z * [new branch] gh/henrylhtsang/108/orig -> origin/gh/henrylhtsang/108/orig 2025-08-14T21:25:13.9721550Z * [new branch] gh/henrylhtsang/118/base -> origin/gh/henrylhtsang/118/base 2025-08-14T21:25:13.9721752Z * [new branch] gh/henrylhtsang/118/head -> origin/gh/henrylhtsang/118/head 2025-08-14T21:25:13.9721937Z * [new branch] gh/henrylhtsang/118/orig -> origin/gh/henrylhtsang/118/orig 2025-08-14T21:25:13.9722226Z * [new branch] gh/henrylhtsang/123/base -> origin/gh/henrylhtsang/123/base 2025-08-14T21:25:13.9724078Z * [new branch] gh/henrylhtsang/123/head -> origin/gh/henrylhtsang/123/head 2025-08-14T21:25:13.9724459Z * [new branch] gh/henrylhtsang/123/orig -> origin/gh/henrylhtsang/123/orig 2025-08-14T21:25:13.9724935Z * [new branch] gh/henrylhtsang/124/base -> origin/gh/henrylhtsang/124/base 2025-08-14T21:25:13.9725168Z * [new branch] gh/henrylhtsang/124/head -> origin/gh/henrylhtsang/124/head 2025-08-14T21:25:13.9727187Z * [new branch] gh/henrylhtsang/124/orig -> origin/gh/henrylhtsang/124/orig 2025-08-14T21:25:13.9727559Z * [new branch] gh/henrylhtsang/125/base -> origin/gh/henrylhtsang/125/base 2025-08-14T21:25:13.9727765Z * [new branch] gh/henrylhtsang/125/head -> origin/gh/henrylhtsang/125/head 2025-08-14T21:25:13.9728299Z * [new branch] gh/henrylhtsang/125/orig -> origin/gh/henrylhtsang/125/orig 2025-08-14T21:25:13.9728888Z * [new branch] gh/henrylhtsang/126/base -> origin/gh/henrylhtsang/126/base 2025-08-14T21:25:13.9730427Z * [new branch] gh/henrylhtsang/126/head -> origin/gh/henrylhtsang/126/head 2025-08-14T21:25:13.9730780Z * [new branch] gh/henrylhtsang/126/orig -> origin/gh/henrylhtsang/126/orig 2025-08-14T21:25:13.9731170Z * [new branch] gh/henrylhtsang/127/base -> origin/gh/henrylhtsang/127/base 2025-08-14T21:25:13.9734286Z * [new branch] gh/henrylhtsang/127/head -> origin/gh/henrylhtsang/127/head 2025-08-14T21:25:13.9734483Z * [new branch] gh/henrylhtsang/127/orig -> origin/gh/henrylhtsang/127/orig 2025-08-14T21:25:13.9734640Z * [new branch] gh/henrylhtsang/128/base -> origin/gh/henrylhtsang/128/base 2025-08-14T21:25:13.9734818Z * [new branch] gh/henrylhtsang/128/head -> origin/gh/henrylhtsang/128/head 2025-08-14T21:25:13.9735143Z * [new branch] gh/henrylhtsang/128/orig -> origin/gh/henrylhtsang/128/orig 2025-08-14T21:25:13.9736841Z * [new branch] gh/henrylhtsang/129/base -> origin/gh/henrylhtsang/129/base 2025-08-14T21:25:13.9737212Z * [new branch] gh/henrylhtsang/129/head -> origin/gh/henrylhtsang/129/head 2025-08-14T21:25:13.9737476Z * [new branch] gh/henrylhtsang/129/orig -> origin/gh/henrylhtsang/129/orig 2025-08-14T21:25:13.9737816Z * [new branch] gh/henrylhtsang/130/base -> origin/gh/henrylhtsang/130/base 2025-08-14T21:25:13.9741836Z * [new branch] gh/henrylhtsang/130/head -> origin/gh/henrylhtsang/130/head 2025-08-14T21:25:13.9742195Z * [new branch] gh/henrylhtsang/131/base -> origin/gh/henrylhtsang/131/base 2025-08-14T21:25:13.9742539Z * [new branch] gh/henrylhtsang/131/head -> origin/gh/henrylhtsang/131/head 2025-08-14T21:25:13.9746802Z * [new branch] gh/henrylhtsang/131/orig -> origin/gh/henrylhtsang/131/orig 2025-08-14T21:25:13.9748953Z * [new branch] gh/henrylhtsang/132/base -> origin/gh/henrylhtsang/132/base 2025-08-14T21:25:13.9749283Z * [new branch] gh/henrylhtsang/132/head -> origin/gh/henrylhtsang/132/head 2025-08-14T21:25:13.9749464Z * [new branch] gh/henrylhtsang/132/orig -> origin/gh/henrylhtsang/132/orig 2025-08-14T21:25:13.9749707Z * [new branch] gh/henrylhtsang/133/base -> origin/gh/henrylhtsang/133/base 2025-08-14T21:25:13.9750053Z * [new branch] gh/henrylhtsang/133/head -> origin/gh/henrylhtsang/133/head 2025-08-14T21:25:13.9750371Z * [new branch] gh/henrylhtsang/133/orig -> origin/gh/henrylhtsang/133/orig 2025-08-14T21:25:13.9750540Z * [new branch] gh/henrylhtsang/134/base -> origin/gh/henrylhtsang/134/base 2025-08-14T21:25:13.9750717Z * [new branch] gh/henrylhtsang/134/head -> origin/gh/henrylhtsang/134/head 2025-08-14T21:25:13.9750874Z * [new branch] gh/henrylhtsang/134/orig -> origin/gh/henrylhtsang/134/orig 2025-08-14T21:25:13.9751024Z * [new branch] gh/henrylhtsang/135/base -> origin/gh/henrylhtsang/135/base 2025-08-14T21:25:13.9751181Z * [new branch] gh/henrylhtsang/135/head -> origin/gh/henrylhtsang/135/head 2025-08-14T21:25:13.9751327Z * [new branch] gh/henrylhtsang/135/orig -> origin/gh/henrylhtsang/135/orig 2025-08-14T21:25:13.9751940Z * [new branch] gh/henrylhtsang/136/base -> origin/gh/henrylhtsang/136/base 2025-08-14T21:25:13.9752143Z * [new branch] gh/henrylhtsang/136/head -> origin/gh/henrylhtsang/136/head 2025-08-14T21:25:13.9752302Z * [new branch] gh/henrylhtsang/136/orig -> origin/gh/henrylhtsang/136/orig 2025-08-14T21:25:13.9752503Z * [new branch] gh/henrylhtsang/137/base -> origin/gh/henrylhtsang/137/base 2025-08-14T21:25:13.9753215Z * [new branch] gh/henrylhtsang/137/head -> origin/gh/henrylhtsang/137/head 2025-08-14T21:25:13.9754441Z * [new branch] gh/henrylhtsang/137/orig -> origin/gh/henrylhtsang/137/orig 2025-08-14T21:25:13.9754691Z * [new branch] gh/henrylhtsang/138/base -> origin/gh/henrylhtsang/138/base 2025-08-14T21:25:13.9757340Z * [new branch] gh/henrylhtsang/138/head -> origin/gh/henrylhtsang/138/head 2025-08-14T21:25:13.9757716Z * [new branch] gh/henrylhtsang/138/orig -> origin/gh/henrylhtsang/138/orig 2025-08-14T21:25:13.9757992Z * [new branch] gh/henrylhtsang/139/base -> origin/gh/henrylhtsang/139/base 2025-08-14T21:25:13.9758535Z * [new branch] gh/henrylhtsang/139/head -> origin/gh/henrylhtsang/139/head 2025-08-14T21:25:13.9758879Z * [new branch] gh/henrylhtsang/139/orig -> origin/gh/henrylhtsang/139/orig 2025-08-14T21:25:13.9760309Z * [new branch] gh/henrylhtsang/140/base -> origin/gh/henrylhtsang/140/base 2025-08-14T21:25:13.9760652Z * [new branch] gh/henrylhtsang/140/head -> origin/gh/henrylhtsang/140/head 2025-08-14T21:25:13.9761144Z * [new branch] gh/henrylhtsang/140/orig -> origin/gh/henrylhtsang/140/orig 2025-08-14T21:25:13.9763394Z * [new branch] gh/henrylhtsang/141/base -> origin/gh/henrylhtsang/141/base 2025-08-14T21:25:13.9763760Z * [new branch] gh/henrylhtsang/141/head -> origin/gh/henrylhtsang/141/head 2025-08-14T21:25:13.9763925Z * [new branch] gh/henrylhtsang/141/orig -> origin/gh/henrylhtsang/141/orig 2025-08-14T21:25:13.9768889Z * [new branch] gh/henrylhtsang/142/base -> origin/gh/henrylhtsang/142/base 2025-08-14T21:25:13.9770882Z * [new branch] gh/henrylhtsang/142/head -> origin/gh/henrylhtsang/142/head 2025-08-14T21:25:13.9771367Z * [new branch] gh/henrylhtsang/142/orig -> origin/gh/henrylhtsang/142/orig 2025-08-14T21:25:13.9771673Z * [new branch] gh/henrylhtsang/143/base -> origin/gh/henrylhtsang/143/base 2025-08-14T21:25:13.9771846Z * [new branch] gh/henrylhtsang/143/head -> origin/gh/henrylhtsang/143/head 2025-08-14T21:25:13.9772097Z * [new branch] gh/henrylhtsang/143/orig -> origin/gh/henrylhtsang/143/orig 2025-08-14T21:25:13.9772266Z * [new branch] gh/henrylhtsang/144/base -> origin/gh/henrylhtsang/144/base 2025-08-14T21:25:13.9772519Z * [new branch] gh/henrylhtsang/144/head -> origin/gh/henrylhtsang/144/head 2025-08-14T21:25:13.9772819Z * [new branch] gh/henrylhtsang/144/orig -> origin/gh/henrylhtsang/144/orig 2025-08-14T21:25:13.9772977Z * [new branch] gh/henrylhtsang/145/base -> origin/gh/henrylhtsang/145/base 2025-08-14T21:25:13.9773133Z * [new branch] gh/henrylhtsang/145/head -> origin/gh/henrylhtsang/145/head 2025-08-14T21:25:13.9773457Z * [new branch] gh/henrylhtsang/145/orig -> origin/gh/henrylhtsang/145/orig 2025-08-14T21:25:13.9778793Z * [new branch] gh/henrylhtsang/146/base -> origin/gh/henrylhtsang/146/base 2025-08-14T21:25:13.9784262Z * [new branch] gh/henrylhtsang/146/head -> origin/gh/henrylhtsang/146/head 2025-08-14T21:25:13.9789148Z * [new branch] gh/henrylhtsang/146/orig -> origin/gh/henrylhtsang/146/orig 2025-08-14T21:25:13.9789525Z * [new branch] gh/huydhn/1/head -> origin/gh/huydhn/1/head 2025-08-14T21:25:13.9789666Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-08-14T21:25:13.9789809Z * [new branch] gh/huydhn/2/head -> origin/gh/huydhn/2/head 2025-08-14T21:25:13.9789958Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-08-14T21:25:13.9790105Z * [new branch] gh/huydhn/2/orig -> origin/gh/huydhn/2/orig 2025-08-14T21:25:13.9790247Z * [new branch] gh/huydhn/3/head -> origin/gh/huydhn/3/head 2025-08-14T21:25:13.9790579Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-08-14T21:25:13.9790732Z * [new branch] gh/huydhn/3/orig -> origin/gh/huydhn/3/orig 2025-08-14T21:25:13.9790864Z * [new branch] gh/huydhn/4/head -> origin/gh/huydhn/4/head 2025-08-14T21:25:13.9791012Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-08-14T21:25:13.9791154Z * [new branch] gh/huydhn/4/orig -> origin/gh/huydhn/4/orig 2025-08-14T21:25:13.9791302Z * [new branch] gh/huydhn/5/head -> origin/gh/huydhn/5/head 2025-08-14T21:25:13.9791447Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-08-14T21:25:13.9791596Z * [new branch] gh/huydhn/5/orig -> origin/gh/huydhn/5/orig 2025-08-14T21:25:13.9791745Z * [new branch] gh/huydhn/6/head -> origin/gh/huydhn/6/head 2025-08-14T21:25:13.9791881Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-08-14T21:25:13.9792024Z * [new branch] gh/huydhn/6/orig -> origin/gh/huydhn/6/orig 2025-08-14T21:25:13.9792176Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-08-14T21:25:13.9792310Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-08-14T21:25:13.9792461Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-08-14T21:25:13.9792613Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-08-14T21:25:13.9792761Z * [new branch] gh/isuruf/116/base -> origin/gh/isuruf/116/base 2025-08-14T21:25:13.9797413Z * [new branch] gh/isuruf/116/head -> origin/gh/isuruf/116/head 2025-08-14T21:25:13.9797644Z * [new branch] gh/isuruf/116/orig -> origin/gh/isuruf/116/orig 2025-08-14T21:25:13.9797872Z * [new branch] gh/isuruf/141/base -> origin/gh/isuruf/141/base 2025-08-14T21:25:13.9798112Z * [new branch] gh/isuruf/141/head -> origin/gh/isuruf/141/head 2025-08-14T21:25:13.9798293Z * [new branch] gh/isuruf/141/orig -> origin/gh/isuruf/141/orig 2025-08-14T21:25:13.9798449Z * [new branch] gh/isuruf/142/base -> origin/gh/isuruf/142/base 2025-08-14T21:25:13.9798577Z * [new branch] gh/isuruf/142/head -> origin/gh/isuruf/142/head 2025-08-14T21:25:13.9798732Z * [new branch] gh/isuruf/142/orig -> origin/gh/isuruf/142/orig 2025-08-14T21:25:13.9802495Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-08-14T21:25:13.9802789Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-08-14T21:25:13.9806735Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-08-14T21:25:13.9807037Z * [new branch] gh/jamesjwu/140/base -> origin/gh/jamesjwu/140/base 2025-08-14T21:25:13.9810546Z * [new branch] gh/jamesjwu/140/head -> origin/gh/jamesjwu/140/head 2025-08-14T21:25:13.9810832Z * [new branch] gh/jamesjwu/140/orig -> origin/gh/jamesjwu/140/orig 2025-08-14T21:25:13.9815879Z * [new branch] gh/jamesjwu/150/base -> origin/gh/jamesjwu/150/base 2025-08-14T21:25:13.9817685Z * [new branch] gh/jamesjwu/150/head -> origin/gh/jamesjwu/150/head 2025-08-14T21:25:13.9817889Z * [new branch] gh/jamesjwu/150/orig -> origin/gh/jamesjwu/150/orig 2025-08-14T21:25:13.9818067Z * [new branch] gh/jamesjwu/154/base -> origin/gh/jamesjwu/154/base 2025-08-14T21:25:13.9818282Z * [new branch] gh/jamesjwu/154/head -> origin/gh/jamesjwu/154/head 2025-08-14T21:25:13.9818660Z * [new branch] gh/jamesjwu/154/orig -> origin/gh/jamesjwu/154/orig 2025-08-14T21:25:13.9818813Z * [new branch] gh/jamesjwu/155/base -> origin/gh/jamesjwu/155/base 2025-08-14T21:25:13.9818963Z * [new branch] gh/jamesjwu/155/head -> origin/gh/jamesjwu/155/head 2025-08-14T21:25:13.9819297Z * [new branch] gh/jamesjwu/155/orig -> origin/gh/jamesjwu/155/orig 2025-08-14T21:25:13.9823961Z * [new branch] gh/jamesjwu/159/base -> origin/gh/jamesjwu/159/base 2025-08-14T21:25:13.9827529Z * [new branch] gh/jamesjwu/159/head -> origin/gh/jamesjwu/159/head 2025-08-14T21:25:13.9829767Z * [new branch] gh/jamesjwu/159/orig -> origin/gh/jamesjwu/159/orig 2025-08-14T21:25:13.9830135Z * [new branch] gh/jamesjwu/163/base -> origin/gh/jamesjwu/163/base 2025-08-14T21:25:13.9830338Z * [new branch] gh/jamesjwu/163/head -> origin/gh/jamesjwu/163/head 2025-08-14T21:25:13.9830533Z * [new branch] gh/jamesjwu/163/orig -> origin/gh/jamesjwu/163/orig 2025-08-14T21:25:13.9830713Z * [new branch] gh/jamesjwu/171/base -> origin/gh/jamesjwu/171/base 2025-08-14T21:25:13.9830881Z * [new branch] gh/jamesjwu/171/head -> origin/gh/jamesjwu/171/head 2025-08-14T21:25:13.9831042Z * [new branch] gh/jamesjwu/171/orig -> origin/gh/jamesjwu/171/orig 2025-08-14T21:25:13.9831197Z * [new branch] gh/jamesjwu/174/base -> origin/gh/jamesjwu/174/base 2025-08-14T21:25:13.9831472Z * [new branch] gh/jamesjwu/174/head -> origin/gh/jamesjwu/174/head 2025-08-14T21:25:13.9832144Z * [new branch] gh/jamesjwu/174/orig -> origin/gh/jamesjwu/174/orig 2025-08-14T21:25:13.9832353Z * [new branch] gh/jamesjwu/175/base -> origin/gh/jamesjwu/175/base 2025-08-14T21:25:13.9832708Z * [new branch] gh/jamesjwu/175/head -> origin/gh/jamesjwu/175/head 2025-08-14T21:25:13.9832884Z * [new branch] gh/jamesjwu/175/orig -> origin/gh/jamesjwu/175/orig 2025-08-14T21:25:13.9833032Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-08-14T21:25:13.9833181Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-08-14T21:25:13.9833331Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-08-14T21:25:13.9833474Z * [new branch] gh/jamesjwu/177/base -> origin/gh/jamesjwu/177/base 2025-08-14T21:25:13.9833626Z * [new branch] gh/jamesjwu/177/head -> origin/gh/jamesjwu/177/head 2025-08-14T21:25:13.9833769Z * [new branch] gh/jamesjwu/177/orig -> origin/gh/jamesjwu/177/orig 2025-08-14T21:25:13.9833924Z * [new branch] gh/jamesjwu/178/base -> origin/gh/jamesjwu/178/base 2025-08-14T21:25:13.9834071Z * [new branch] gh/jamesjwu/178/head -> origin/gh/jamesjwu/178/head 2025-08-14T21:25:13.9834217Z * [new branch] gh/jamesjwu/178/orig -> origin/gh/jamesjwu/178/orig 2025-08-14T21:25:13.9834363Z * [new branch] gh/jamesjwu/179/base -> origin/gh/jamesjwu/179/base 2025-08-14T21:25:13.9834518Z * [new branch] gh/jamesjwu/179/head -> origin/gh/jamesjwu/179/head 2025-08-14T21:25:13.9834675Z * [new branch] gh/jamesjwu/179/orig -> origin/gh/jamesjwu/179/orig 2025-08-14T21:25:13.9834830Z * [new branch] gh/jamesjwu/180/base -> origin/gh/jamesjwu/180/base 2025-08-14T21:25:13.9834971Z * [new branch] gh/jamesjwu/180/head -> origin/gh/jamesjwu/180/head 2025-08-14T21:25:13.9835114Z * [new branch] gh/jamesjwu/180/orig -> origin/gh/jamesjwu/180/orig 2025-08-14T21:25:13.9835623Z * [new branch] gh/jamesjwu/181/base -> origin/gh/jamesjwu/181/base 2025-08-14T21:25:13.9836506Z * [new branch] gh/jamesjwu/181/head -> origin/gh/jamesjwu/181/head 2025-08-14T21:25:13.9836905Z * [new branch] gh/jamesjwu/181/orig -> origin/gh/jamesjwu/181/orig 2025-08-14T21:25:13.9840838Z * [new branch] gh/jamesjwu/182/base -> origin/gh/jamesjwu/182/base 2025-08-14T21:25:13.9841026Z * [new branch] gh/jamesjwu/182/head -> origin/gh/jamesjwu/182/head 2025-08-14T21:25:13.9841174Z * [new branch] gh/jamesjwu/182/orig -> origin/gh/jamesjwu/182/orig 2025-08-14T21:25:13.9841330Z * [new branch] gh/jamesjwu/183/base -> origin/gh/jamesjwu/183/base 2025-08-14T21:25:13.9841480Z * [new branch] gh/jamesjwu/183/head -> origin/gh/jamesjwu/183/head 2025-08-14T21:25:13.9847216Z * [new branch] gh/jamesjwu/183/orig -> origin/gh/jamesjwu/183/orig 2025-08-14T21:25:13.9847425Z * [new branch] gh/jamesjwu/184/base -> origin/gh/jamesjwu/184/base 2025-08-14T21:25:13.9847590Z * [new branch] gh/jamesjwu/184/head -> origin/gh/jamesjwu/184/head 2025-08-14T21:25:13.9847737Z * [new branch] gh/jamesjwu/184/orig -> origin/gh/jamesjwu/184/orig 2025-08-14T21:25:13.9847894Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-08-14T21:25:13.9848038Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-08-14T21:25:13.9848180Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-08-14T21:25:13.9848331Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-08-14T21:25:13.9848480Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-08-14T21:25:13.9848626Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-08-14T21:25:13.9853530Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-08-14T21:25:13.9853889Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-08-14T21:25:13.9854075Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-08-14T21:25:13.9854225Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-08-14T21:25:13.9854373Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-08-14T21:25:13.9854516Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-08-14T21:25:13.9854665Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-08-14T21:25:13.9854931Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-08-14T21:25:13.9855398Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-08-14T21:25:13.9855619Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-08-14T21:25:13.9857041Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-08-14T21:25:13.9857189Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-08-14T21:25:13.9857777Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-08-14T21:25:13.9858299Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-08-14T21:25:13.9862081Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-08-14T21:25:13.9862251Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-08-14T21:25:13.9862398Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-08-14T21:25:13.9862530Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-08-14T21:25:13.9862694Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-08-14T21:25:13.9862991Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-08-14T21:25:13.9863743Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-08-14T21:25:13.9864212Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-08-14T21:25:13.9865818Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-08-14T21:25:13.9866195Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-08-14T21:25:13.9866744Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-08-14T21:25:13.9867776Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-08-14T21:25:13.9868085Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-08-14T21:25:13.9868916Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-08-14T21:25:13.9870364Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-08-14T21:25:13.9870521Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-08-14T21:25:13.9871226Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-08-14T21:25:13.9872520Z * [new branch] gh/janeyx99/256/base -> origin/gh/janeyx99/256/base 2025-08-14T21:25:13.9872780Z * [new branch] gh/janeyx99/256/head -> origin/gh/janeyx99/256/head 2025-08-14T21:25:13.9873835Z * [new branch] gh/janeyx99/256/orig -> origin/gh/janeyx99/256/orig 2025-08-14T21:25:13.9874879Z * [new branch] gh/janeyx99/268/base -> origin/gh/janeyx99/268/base 2025-08-14T21:25:13.9875253Z * [new branch] gh/janeyx99/268/head -> origin/gh/janeyx99/268/head 2025-08-14T21:25:13.9876444Z * [new branch] gh/janeyx99/268/orig -> origin/gh/janeyx99/268/orig 2025-08-14T21:25:13.9877385Z * [new branch] gh/janeyx99/269/base -> origin/gh/janeyx99/269/base 2025-08-14T21:25:13.9877641Z * [new branch] gh/janeyx99/269/head -> origin/gh/janeyx99/269/head 2025-08-14T21:25:13.9878680Z * [new branch] gh/janeyx99/269/orig -> origin/gh/janeyx99/269/orig 2025-08-14T21:25:13.9879904Z * [new branch] gh/janeyx99/274/base -> origin/gh/janeyx99/274/base 2025-08-14T21:25:13.9880054Z * [new branch] gh/janeyx99/274/head -> origin/gh/janeyx99/274/head 2025-08-14T21:25:13.9880600Z * [new branch] gh/janeyx99/274/orig -> origin/gh/janeyx99/274/orig 2025-08-14T21:25:13.9881847Z * [new branch] gh/janeyx99/276/base -> origin/gh/janeyx99/276/base 2025-08-14T21:25:13.9882119Z * [new branch] gh/janeyx99/276/head -> origin/gh/janeyx99/276/head 2025-08-14T21:25:13.9883179Z * [new branch] gh/janeyx99/276/orig -> origin/gh/janeyx99/276/orig 2025-08-14T21:25:13.9883698Z * [new branch] gh/janeyx99/277/base -> origin/gh/janeyx99/277/base 2025-08-14T21:25:13.9884581Z * [new branch] gh/janeyx99/277/head -> origin/gh/janeyx99/277/head 2025-08-14T21:25:13.9884916Z * [new branch] gh/janeyx99/277/orig -> origin/gh/janeyx99/277/orig 2025-08-14T21:25:13.9886316Z * [new branch] gh/janeyx99/278/base -> origin/gh/janeyx99/278/base 2025-08-14T21:25:13.9886560Z * [new branch] gh/janeyx99/278/head -> origin/gh/janeyx99/278/head 2025-08-14T21:25:13.9887728Z * [new branch] gh/janeyx99/278/orig -> origin/gh/janeyx99/278/orig 2025-08-14T21:25:13.9888530Z * [new branch] gh/janeyx99/279/base -> origin/gh/janeyx99/279/base 2025-08-14T21:25:13.9888817Z * [new branch] gh/janeyx99/279/head -> origin/gh/janeyx99/279/head 2025-08-14T21:25:13.9889845Z * [new branch] gh/janeyx99/279/orig -> origin/gh/janeyx99/279/orig 2025-08-14T21:25:13.9890859Z * [new branch] gh/janeyx99/280/base -> origin/gh/janeyx99/280/base 2025-08-14T21:25:13.9891108Z * [new branch] gh/janeyx99/280/head -> origin/gh/janeyx99/280/head 2025-08-14T21:25:13.9892065Z * [new branch] gh/janeyx99/280/orig -> origin/gh/janeyx99/280/orig 2025-08-14T21:25:13.9892600Z * [new branch] gh/janeyx99/281/base -> origin/gh/janeyx99/281/base 2025-08-14T21:25:13.9893315Z * [new branch] gh/janeyx99/281/head -> origin/gh/janeyx99/281/head 2025-08-14T21:25:13.9893820Z * [new branch] gh/janeyx99/281/orig -> origin/gh/janeyx99/281/orig 2025-08-14T21:25:13.9895061Z * [new branch] gh/janeyx99/282/base -> origin/gh/janeyx99/282/base 2025-08-14T21:25:13.9895305Z * [new branch] gh/janeyx99/282/head -> origin/gh/janeyx99/282/head 2025-08-14T21:25:13.9896307Z * [new branch] gh/janeyx99/282/orig -> origin/gh/janeyx99/282/orig 2025-08-14T21:25:13.9897247Z * [new branch] gh/janeyx99/283/base -> origin/gh/janeyx99/283/base 2025-08-14T21:25:13.9897518Z * [new branch] gh/janeyx99/283/head -> origin/gh/janeyx99/283/head 2025-08-14T21:25:13.9898610Z * [new branch] gh/janeyx99/283/orig -> origin/gh/janeyx99/283/orig 2025-08-14T21:25:13.9899815Z * [new branch] gh/janeyx99/284/base -> origin/gh/janeyx99/284/base 2025-08-14T21:25:13.9900083Z * [new branch] gh/janeyx99/284/head -> origin/gh/janeyx99/284/head 2025-08-14T21:25:13.9900996Z * [new branch] gh/janeyx99/284/orig -> origin/gh/janeyx99/284/orig 2025-08-14T21:25:13.9902230Z * [new branch] gh/janeyx99/285/base -> origin/gh/janeyx99/285/base 2025-08-14T21:25:13.9902375Z * [new branch] gh/janeyx99/285/head -> origin/gh/janeyx99/285/head 2025-08-14T21:25:13.9903441Z * [new branch] gh/janeyx99/285/orig -> origin/gh/janeyx99/285/orig 2025-08-14T21:25:13.9904440Z * [new branch] gh/janeyx99/286/base -> origin/gh/janeyx99/286/base 2025-08-14T21:25:13.9904790Z * [new branch] gh/janeyx99/286/head -> origin/gh/janeyx99/286/head 2025-08-14T21:25:13.9905843Z * [new branch] gh/janeyx99/286/orig -> origin/gh/janeyx99/286/orig 2025-08-14T21:25:13.9906653Z * [new branch] gh/janeyx99/287/base -> origin/gh/janeyx99/287/base 2025-08-14T21:25:13.9906983Z * [new branch] gh/janeyx99/287/head -> origin/gh/janeyx99/287/head 2025-08-14T21:25:13.9910526Z * [new branch] gh/janeyx99/287/orig -> origin/gh/janeyx99/287/orig 2025-08-14T21:25:13.9910761Z * [new branch] gh/janeyx99/288/base -> origin/gh/janeyx99/288/base 2025-08-14T21:25:13.9910919Z * [new branch] gh/janeyx99/288/head -> origin/gh/janeyx99/288/head 2025-08-14T21:25:13.9911081Z * [new branch] gh/janeyx99/288/orig -> origin/gh/janeyx99/288/orig 2025-08-14T21:25:13.9912269Z * [new branch] gh/janeyx99/289/base -> origin/gh/janeyx99/289/base 2025-08-14T21:25:13.9912543Z * [new branch] gh/janeyx99/289/head -> origin/gh/janeyx99/289/head 2025-08-14T21:25:13.9913960Z * [new branch] gh/janeyx99/289/orig -> origin/gh/janeyx99/289/orig 2025-08-14T21:25:13.9914411Z * [new branch] gh/janeyx99/290/base -> origin/gh/janeyx99/290/base 2025-08-14T21:25:13.9915167Z * [new branch] gh/janeyx99/290/head -> origin/gh/janeyx99/290/head 2025-08-14T21:25:13.9915729Z * [new branch] gh/janeyx99/290/orig -> origin/gh/janeyx99/290/orig 2025-08-14T21:25:13.9919784Z * [new branch] gh/janeyx99/291/base -> origin/gh/janeyx99/291/base 2025-08-14T21:25:13.9920320Z * [new branch] gh/janeyx99/291/head -> origin/gh/janeyx99/291/head 2025-08-14T21:25:13.9920557Z * [new branch] gh/janeyx99/291/orig -> origin/gh/janeyx99/291/orig 2025-08-14T21:25:13.9925991Z * [new branch] gh/janeyx99/292/base -> origin/gh/janeyx99/292/base 2025-08-14T21:25:13.9926337Z * [new branch] gh/janeyx99/292/head -> origin/gh/janeyx99/292/head 2025-08-14T21:25:13.9926502Z * [new branch] gh/janeyx99/292/orig -> origin/gh/janeyx99/292/orig 2025-08-14T21:25:13.9926656Z * [new branch] gh/janeyx99/293/base -> origin/gh/janeyx99/293/base 2025-08-14T21:25:13.9926937Z * [new branch] gh/janeyx99/293/head -> origin/gh/janeyx99/293/head 2025-08-14T21:25:13.9927109Z * [new branch] gh/janeyx99/293/orig -> origin/gh/janeyx99/293/orig 2025-08-14T21:25:13.9927370Z * [new branch] gh/janeyx99/294/base -> origin/gh/janeyx99/294/base 2025-08-14T21:25:13.9927537Z * [new branch] gh/janeyx99/294/head -> origin/gh/janeyx99/294/head 2025-08-14T21:25:13.9927766Z * [new branch] gh/janeyx99/294/orig -> origin/gh/janeyx99/294/orig 2025-08-14T21:25:13.9927934Z * [new branch] gh/janeyx99/295/base -> origin/gh/janeyx99/295/base 2025-08-14T21:25:13.9928549Z * [new branch] gh/janeyx99/295/head -> origin/gh/janeyx99/295/head 2025-08-14T21:25:13.9928934Z * [new branch] gh/janeyx99/295/orig -> origin/gh/janeyx99/295/orig 2025-08-14T21:25:13.9933336Z * [new branch] gh/janeyx99/296/base -> origin/gh/janeyx99/296/base 2025-08-14T21:25:13.9933651Z * [new branch] gh/janeyx99/296/head -> origin/gh/janeyx99/296/head 2025-08-14T21:25:13.9934093Z * [new branch] gh/janeyx99/296/orig -> origin/gh/janeyx99/296/orig 2025-08-14T21:25:13.9934338Z * [new branch] gh/janeyx99/297/base -> origin/gh/janeyx99/297/base 2025-08-14T21:25:13.9934650Z * [new branch] gh/janeyx99/297/head -> origin/gh/janeyx99/297/head 2025-08-14T21:25:13.9935333Z * [new branch] gh/janeyx99/297/orig -> origin/gh/janeyx99/297/orig 2025-08-14T21:25:13.9935530Z * [new branch] gh/janeyx99/298/base -> origin/gh/janeyx99/298/base 2025-08-14T21:25:13.9936014Z * [new branch] gh/janeyx99/298/head -> origin/gh/janeyx99/298/head 2025-08-14T21:25:13.9936204Z * [new branch] gh/janeyx99/298/orig -> origin/gh/janeyx99/298/orig 2025-08-14T21:25:13.9936350Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-08-14T21:25:13.9936506Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-08-14T21:25:13.9936670Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-08-14T21:25:13.9941495Z * [new branch] gh/janeyx99/300/base -> origin/gh/janeyx99/300/base 2025-08-14T21:25:13.9941877Z * [new branch] gh/janeyx99/300/head -> origin/gh/janeyx99/300/head 2025-08-14T21:25:13.9942063Z * [new branch] gh/janeyx99/300/orig -> origin/gh/janeyx99/300/orig 2025-08-14T21:25:13.9942231Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-08-14T21:25:13.9942379Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-08-14T21:25:13.9942636Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-08-14T21:25:13.9948305Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-08-14T21:25:13.9948521Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-08-14T21:25:13.9948697Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-08-14T21:25:13.9948979Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-08-14T21:25:13.9949120Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-08-14T21:25:13.9949251Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-08-14T21:25:13.9949387Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-08-14T21:25:13.9949525Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-08-14T21:25:13.9949656Z * [new branch] gh/jansel/531/base -> origin/gh/jansel/531/base 2025-08-14T21:25:13.9949793Z * [new branch] gh/jansel/531/head -> origin/gh/jansel/531/head 2025-08-14T21:25:13.9949918Z * [new branch] gh/jansel/531/orig -> origin/gh/jansel/531/orig 2025-08-14T21:25:13.9950049Z * [new branch] gh/jansel/534/base -> origin/gh/jansel/534/base 2025-08-14T21:25:13.9950186Z * [new branch] gh/jansel/534/head -> origin/gh/jansel/534/head 2025-08-14T21:25:13.9950308Z * [new branch] gh/jansel/534/orig -> origin/gh/jansel/534/orig 2025-08-14T21:25:13.9950637Z * [new branch] gh/jbschlosser/226/base -> origin/gh/jbschlosser/226/base 2025-08-14T21:25:13.9950814Z * [new branch] gh/jbschlosser/226/head -> origin/gh/jbschlosser/226/head 2025-08-14T21:25:13.9951152Z * [new branch] gh/jbschlosser/226/orig -> origin/gh/jbschlosser/226/orig 2025-08-14T21:25:13.9952501Z * [new branch] gh/jbschlosser/239/base -> origin/gh/jbschlosser/239/base 2025-08-14T21:25:13.9953184Z * [new branch] gh/jbschlosser/239/head -> origin/gh/jbschlosser/239/head 2025-08-14T21:25:13.9953973Z * [new branch] gh/jbschlosser/239/orig -> origin/gh/jbschlosser/239/orig 2025-08-14T21:25:13.9954632Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-08-14T21:25:13.9955844Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-08-14T21:25:13.9956125Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-08-14T21:25:13.9959743Z * [new branch] gh/jbschlosser/248/base -> origin/gh/jbschlosser/248/base 2025-08-14T21:25:13.9960111Z * [new branch] gh/jbschlosser/248/head -> origin/gh/jbschlosser/248/head 2025-08-14T21:25:13.9960357Z * [new branch] gh/jbschlosser/248/orig -> origin/gh/jbschlosser/248/orig 2025-08-14T21:25:13.9960537Z * [new branch] gh/jbschlosser/249/base -> origin/gh/jbschlosser/249/base 2025-08-14T21:25:13.9960769Z * [new branch] gh/jbschlosser/249/head -> origin/gh/jbschlosser/249/head 2025-08-14T21:25:13.9961491Z * [new branch] gh/jbschlosser/249/orig -> origin/gh/jbschlosser/249/orig 2025-08-14T21:25:13.9962089Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-08-14T21:25:13.9968839Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-08-14T21:25:13.9970899Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-08-14T21:25:13.9971203Z * [new branch] gh/jiayisunx/57/base -> origin/gh/jiayisunx/57/base 2025-08-14T21:25:13.9974087Z * [new branch] gh/jiayisunx/57/head -> origin/gh/jiayisunx/57/head 2025-08-14T21:25:13.9974384Z * [new branch] gh/jiayisunx/57/orig -> origin/gh/jiayisunx/57/orig 2025-08-14T21:25:13.9974606Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-08-14T21:25:13.9974776Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-08-14T21:25:13.9975067Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-08-14T21:25:13.9975239Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-08-14T21:25:13.9975394Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-08-14T21:25:13.9975555Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-08-14T21:25:13.9975706Z * [new branch] gh/jiayisunx/63/base -> origin/gh/jiayisunx/63/base 2025-08-14T21:25:13.9975857Z * [new branch] gh/jiayisunx/63/head -> origin/gh/jiayisunx/63/head 2025-08-14T21:25:13.9976043Z * [new branch] gh/jiayisunx/63/orig -> origin/gh/jiayisunx/63/orig 2025-08-14T21:25:13.9976182Z * [new branch] gh/jiayisunx/64/base -> origin/gh/jiayisunx/64/base 2025-08-14T21:25:13.9976335Z * [new branch] gh/jiayisunx/64/head -> origin/gh/jiayisunx/64/head 2025-08-14T21:25:13.9976499Z * [new branch] gh/jiayisunx/64/orig -> origin/gh/jiayisunx/64/orig 2025-08-14T21:25:13.9976645Z * [new branch] gh/jiayisunx/65/base -> origin/gh/jiayisunx/65/base 2025-08-14T21:25:13.9976910Z * [new branch] gh/jiayisunx/65/head -> origin/gh/jiayisunx/65/head 2025-08-14T21:25:13.9977074Z * [new branch] gh/jiayisunx/65/orig -> origin/gh/jiayisunx/65/orig 2025-08-14T21:25:13.9980649Z * [new branch] gh/jiayisunx/66/base -> origin/gh/jiayisunx/66/base 2025-08-14T21:25:13.9980837Z * [new branch] gh/jiayisunx/66/head -> origin/gh/jiayisunx/66/head 2025-08-14T21:25:13.9980995Z * [new branch] gh/jiayisunx/66/orig -> origin/gh/jiayisunx/66/orig 2025-08-14T21:25:13.9981140Z * [new branch] gh/jiayisunx/67/base -> origin/gh/jiayisunx/67/base 2025-08-14T21:25:13.9981440Z * [new branch] gh/jiayisunx/67/head -> origin/gh/jiayisunx/67/head 2025-08-14T21:25:13.9981604Z * [new branch] gh/jiayisunx/67/orig -> origin/gh/jiayisunx/67/orig 2025-08-14T21:25:13.9982135Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-08-14T21:25:13.9987462Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-08-14T21:25:13.9987646Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-08-14T21:25:13.9987827Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-08-14T21:25:13.9987982Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-08-14T21:25:13.9988165Z * [new branch] gh/justinchuby/111/base -> origin/gh/justinchuby/111/base 2025-08-14T21:25:13.9988503Z * [new branch] gh/justinchuby/111/head -> origin/gh/justinchuby/111/head 2025-08-14T21:25:13.9988708Z * [new branch] gh/justinchuby/111/orig -> origin/gh/justinchuby/111/orig 2025-08-14T21:25:13.9989034Z * [new branch] gh/kurtamohler/32/base -> origin/gh/kurtamohler/32/base 2025-08-14T21:25:13.9992110Z * [new branch] gh/kurtamohler/32/head -> origin/gh/kurtamohler/32/head 2025-08-14T21:25:13.9992309Z * [new branch] gh/kurtamohler/32/orig -> origin/gh/kurtamohler/32/orig 2025-08-14T21:25:13.9992469Z * [new branch] gh/kurtamohler/33/base -> origin/gh/kurtamohler/33/base 2025-08-14T21:25:13.9992628Z * [new branch] gh/kurtamohler/33/head -> origin/gh/kurtamohler/33/head 2025-08-14T21:25:13.9992783Z * [new branch] gh/kurtamohler/33/orig -> origin/gh/kurtamohler/33/orig 2025-08-14T21:25:13.9993423Z * [new branch] gh/kurtamohler/34/base -> origin/gh/kurtamohler/34/base 2025-08-14T21:25:13.9995051Z * [new branch] gh/kurtamohler/34/head -> origin/gh/kurtamohler/34/head 2025-08-14T21:25:13.9995399Z * [new branch] gh/kurtamohler/34/orig -> origin/gh/kurtamohler/34/orig 2025-08-14T21:25:13.9996273Z * [new branch] gh/kurtamohler/40/base -> origin/gh/kurtamohler/40/base 2025-08-14T21:25:13.9996851Z * [new branch] gh/kurtamohler/40/head -> origin/gh/kurtamohler/40/head 2025-08-14T21:25:14.0002175Z * [new branch] gh/kurtamohler/40/orig -> origin/gh/kurtamohler/40/orig 2025-08-14T21:25:14.0005815Z * [new branch] gh/kurtamohler/41/base -> origin/gh/kurtamohler/41/base 2025-08-14T21:25:14.0006143Z * [new branch] gh/kurtamohler/41/head -> origin/gh/kurtamohler/41/head 2025-08-14T21:25:14.0006301Z * [new branch] gh/kurtamohler/41/orig -> origin/gh/kurtamohler/41/orig 2025-08-14T21:25:14.0006534Z * [new branch] gh/kurtamohler/42/base -> origin/gh/kurtamohler/42/base 2025-08-14T21:25:14.0006708Z * [new branch] gh/kurtamohler/42/head -> origin/gh/kurtamohler/42/head 2025-08-14T21:25:14.0006860Z * [new branch] gh/kurtamohler/42/orig -> origin/gh/kurtamohler/42/orig 2025-08-14T21:25:14.0007126Z * [new branch] gh/kurtamohler/43/base -> origin/gh/kurtamohler/43/base 2025-08-14T21:25:14.0007831Z * [new branch] gh/kurtamohler/43/head -> origin/gh/kurtamohler/43/head 2025-08-14T21:25:14.0008016Z * [new branch] gh/kurtamohler/43/orig -> origin/gh/kurtamohler/43/orig 2025-08-14T21:25:14.0008166Z * [new branch] gh/kurtamohler/44/base -> origin/gh/kurtamohler/44/base 2025-08-14T21:25:14.0008316Z * [new branch] gh/kurtamohler/44/head -> origin/gh/kurtamohler/44/head 2025-08-14T21:25:14.0008459Z * [new branch] gh/kurtamohler/44/orig -> origin/gh/kurtamohler/44/orig 2025-08-14T21:25:14.0008939Z * [new branch] gh/kurtamohler/45/base -> origin/gh/kurtamohler/45/base 2025-08-14T21:25:14.0009116Z * [new branch] gh/kurtamohler/45/head -> origin/gh/kurtamohler/45/head 2025-08-14T21:25:14.0013294Z * [new branch] gh/kurtamohler/45/orig -> origin/gh/kurtamohler/45/orig 2025-08-14T21:25:14.0013622Z * [new branch] gh/kurtamohler/46/base -> origin/gh/kurtamohler/46/base 2025-08-14T21:25:14.0013801Z * [new branch] gh/kurtamohler/46/head -> origin/gh/kurtamohler/46/head 2025-08-14T21:25:14.0014071Z * [new branch] gh/kurtamohler/46/orig -> origin/gh/kurtamohler/46/orig 2025-08-14T21:25:14.0022430Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-08-14T21:25:14.0027222Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-08-14T21:25:14.0029007Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-08-14T21:25:14.0029353Z * [new branch] gh/kwen2501/142/base -> origin/gh/kwen2501/142/base 2025-08-14T21:25:14.0029527Z * [new branch] gh/kwen2501/142/head -> origin/gh/kwen2501/142/head 2025-08-14T21:25:14.0029660Z * [new branch] gh/kwen2501/142/orig -> origin/gh/kwen2501/142/orig 2025-08-14T21:25:14.0029945Z * [new branch] gh/kwen2501/15/base -> origin/gh/kwen2501/15/base 2025-08-14T21:25:14.0030275Z * [new branch] gh/kwen2501/15/head -> origin/gh/kwen2501/15/head 2025-08-14T21:25:14.0030447Z * [new branch] gh/kwen2501/156/base -> origin/gh/kwen2501/156/base 2025-08-14T21:25:14.0030604Z * [new branch] gh/kwen2501/156/head -> origin/gh/kwen2501/156/head 2025-08-14T21:25:14.0031153Z * [new branch] gh/kwen2501/156/orig -> origin/gh/kwen2501/156/orig 2025-08-14T21:25:14.0031345Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-08-14T21:25:14.0031519Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-08-14T21:25:14.0031882Z * [new branch] gh/kwen2501/179/base -> origin/gh/kwen2501/179/base 2025-08-14T21:25:14.0032027Z * [new branch] gh/kwen2501/179/head -> origin/gh/kwen2501/179/head 2025-08-14T21:25:14.0032164Z * [new branch] gh/kwen2501/179/orig -> origin/gh/kwen2501/179/orig 2025-08-14T21:25:14.0032313Z * [new branch] gh/kwen2501/181/base -> origin/gh/kwen2501/181/base 2025-08-14T21:25:14.0032453Z * [new branch] gh/kwen2501/181/head -> origin/gh/kwen2501/181/head 2025-08-14T21:25:14.0032591Z * [new branch] gh/kwen2501/181/orig -> origin/gh/kwen2501/181/orig 2025-08-14T21:25:14.0032738Z * [new branch] gh/kwen2501/183/base -> origin/gh/kwen2501/183/base 2025-08-14T21:25:14.0032873Z * [new branch] gh/kwen2501/183/head -> origin/gh/kwen2501/183/head 2025-08-14T21:25:14.0033026Z * [new branch] gh/kwen2501/183/orig -> origin/gh/kwen2501/183/orig 2025-08-14T21:25:14.0033181Z * [new branch] gh/kwen2501/184/base -> origin/gh/kwen2501/184/base 2025-08-14T21:25:14.0033324Z * [new branch] gh/kwen2501/184/head -> origin/gh/kwen2501/184/head 2025-08-14T21:25:14.0033473Z * [new branch] gh/kwen2501/184/orig -> origin/gh/kwen2501/184/orig 2025-08-14T21:25:14.0033615Z * [new branch] gh/kwen2501/186/base -> origin/gh/kwen2501/186/base 2025-08-14T21:25:14.0033762Z * [new branch] gh/kwen2501/186/head -> origin/gh/kwen2501/186/head 2025-08-14T21:25:14.0033904Z * [new branch] gh/kwen2501/186/orig -> origin/gh/kwen2501/186/orig 2025-08-14T21:25:14.0034046Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-08-14T21:25:14.0034242Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-08-14T21:25:14.0034395Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-08-14T21:25:14.0034547Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-08-14T21:25:14.0034690Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-08-14T21:25:14.0036292Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-08-14T21:25:14.0036647Z * [new branch] gh/kwen2501/194/base -> origin/gh/kwen2501/194/base 2025-08-14T21:25:14.0044230Z * [new branch] gh/kwen2501/194/head -> origin/gh/kwen2501/194/head 2025-08-14T21:25:14.0044564Z * [new branch] gh/kwen2501/194/orig -> origin/gh/kwen2501/194/orig 2025-08-14T21:25:14.0044725Z * [new branch] gh/kwen2501/195/base -> origin/gh/kwen2501/195/base 2025-08-14T21:25:14.0045006Z * [new branch] gh/kwen2501/195/head -> origin/gh/kwen2501/195/head 2025-08-14T21:25:14.0045234Z * [new branch] gh/kwen2501/195/orig -> origin/gh/kwen2501/195/orig 2025-08-14T21:25:14.0045415Z * [new branch] gh/kwen2501/196/base -> origin/gh/kwen2501/196/base 2025-08-14T21:25:14.0045561Z * [new branch] gh/kwen2501/196/head -> origin/gh/kwen2501/196/head 2025-08-14T21:25:14.0045709Z * [new branch] gh/kwen2501/196/orig -> origin/gh/kwen2501/196/orig 2025-08-14T21:25:14.0045862Z * [new branch] gh/kwen2501/197/base -> origin/gh/kwen2501/197/base 2025-08-14T21:25:14.0045988Z * [new branch] gh/kwen2501/197/head -> origin/gh/kwen2501/197/head 2025-08-14T21:25:14.0046147Z * [new branch] gh/kwen2501/197/orig -> origin/gh/kwen2501/197/orig 2025-08-14T21:25:14.0046293Z * [new branch] gh/kwen2501/198/base -> origin/gh/kwen2501/198/base 2025-08-14T21:25:14.0046445Z * [new branch] gh/kwen2501/198/head -> origin/gh/kwen2501/198/head 2025-08-14T21:25:14.0046760Z * [new branch] gh/kwen2501/198/orig -> origin/gh/kwen2501/198/orig 2025-08-14T21:25:14.0051013Z * [new branch] gh/kwen2501/199/base -> origin/gh/kwen2501/199/base 2025-08-14T21:25:14.0051198Z * [new branch] gh/kwen2501/199/head -> origin/gh/kwen2501/199/head 2025-08-14T21:25:14.0051481Z * [new branch] gh/kwen2501/199/orig -> origin/gh/kwen2501/199/orig 2025-08-14T21:25:14.0051623Z * [new branch] gh/kwen2501/200/base -> origin/gh/kwen2501/200/base 2025-08-14T21:25:14.0051748Z * [new branch] gh/kwen2501/200/head -> origin/gh/kwen2501/200/head 2025-08-14T21:25:14.0051874Z * [new branch] gh/kwen2501/200/orig -> origin/gh/kwen2501/200/orig 2025-08-14T21:25:14.0052166Z * [new branch] gh/kwen2501/201/base -> origin/gh/kwen2501/201/base 2025-08-14T21:25:14.0053491Z * [new branch] gh/kwen2501/201/head -> origin/gh/kwen2501/201/head 2025-08-14T21:25:14.0053847Z * [new branch] gh/kwen2501/201/orig -> origin/gh/kwen2501/201/orig 2025-08-14T21:25:14.0054317Z * [new branch] gh/kwen2501/202/base -> origin/gh/kwen2501/202/base 2025-08-14T21:25:14.0056000Z * [new branch] gh/kwen2501/202/head -> origin/gh/kwen2501/202/head 2025-08-14T21:25:14.0056337Z * [new branch] gh/kwen2501/202/orig -> origin/gh/kwen2501/202/orig 2025-08-14T21:25:14.0059510Z * [new branch] gh/kwen2501/203/base -> origin/gh/kwen2501/203/base 2025-08-14T21:25:14.0059685Z * [new branch] gh/kwen2501/203/head -> origin/gh/kwen2501/203/head 2025-08-14T21:25:14.0059814Z * [new branch] gh/kwen2501/203/orig -> origin/gh/kwen2501/203/orig 2025-08-14T21:25:14.0060163Z * [new branch] gh/laithsakka/152/base -> origin/gh/laithsakka/152/base 2025-08-14T21:25:14.0060327Z * [new branch] gh/laithsakka/152/head -> origin/gh/laithsakka/152/head 2025-08-14T21:25:14.0064517Z * [new branch] gh/laithsakka/152/orig -> origin/gh/laithsakka/152/orig 2025-08-14T21:25:14.0064687Z * [new branch] gh/laithsakka/156/base -> origin/gh/laithsakka/156/base 2025-08-14T21:25:14.0064826Z * [new branch] gh/laithsakka/156/head -> origin/gh/laithsakka/156/head 2025-08-14T21:25:14.0064979Z * [new branch] gh/laithsakka/156/orig -> origin/gh/laithsakka/156/orig 2025-08-14T21:25:14.0065117Z * [new branch] gh/laithsakka/159/base -> origin/gh/laithsakka/159/base 2025-08-14T21:25:14.0065263Z * [new branch] gh/laithsakka/159/head -> origin/gh/laithsakka/159/head 2025-08-14T21:25:14.0065400Z * [new branch] gh/laithsakka/159/orig -> origin/gh/laithsakka/159/orig 2025-08-14T21:25:14.0066042Z * [new branch] gh/laithsakka/160/base -> origin/gh/laithsakka/160/base 2025-08-14T21:25:14.0066727Z * [new branch] gh/laithsakka/160/head -> origin/gh/laithsakka/160/head 2025-08-14T21:25:14.0067321Z * [new branch] gh/laithsakka/160/orig -> origin/gh/laithsakka/160/orig 2025-08-14T21:25:14.0072048Z * [new branch] gh/laithsakka/178/base -> origin/gh/laithsakka/178/base 2025-08-14T21:25:14.0072244Z * [new branch] gh/laithsakka/178/head -> origin/gh/laithsakka/178/head 2025-08-14T21:25:14.0072405Z * [new branch] gh/laithsakka/178/orig -> origin/gh/laithsakka/178/orig 2025-08-14T21:25:14.0072559Z * [new branch] gh/laithsakka/191/base -> origin/gh/laithsakka/191/base 2025-08-14T21:25:14.0072715Z * [new branch] gh/laithsakka/191/head -> origin/gh/laithsakka/191/head 2025-08-14T21:25:14.0072867Z * [new branch] gh/laithsakka/191/orig -> origin/gh/laithsakka/191/orig 2025-08-14T21:25:14.0073177Z * [new branch] gh/laithsakka/234/base -> origin/gh/laithsakka/234/base 2025-08-14T21:25:14.0073830Z * [new branch] gh/laithsakka/234/head -> origin/gh/laithsakka/234/head 2025-08-14T21:25:14.0073992Z * [new branch] gh/laithsakka/234/orig -> origin/gh/laithsakka/234/orig 2025-08-14T21:25:14.0078105Z * [new branch] gh/laithsakka/237/base -> origin/gh/laithsakka/237/base 2025-08-14T21:25:14.0078250Z * [new branch] gh/laithsakka/237/head -> origin/gh/laithsakka/237/head 2025-08-14T21:25:14.0078389Z * [new branch] gh/laithsakka/237/orig -> origin/gh/laithsakka/237/orig 2025-08-14T21:25:14.0078533Z * [new branch] gh/laithsakka/238/base -> origin/gh/laithsakka/238/base 2025-08-14T21:25:14.0078666Z * [new branch] gh/laithsakka/238/head -> origin/gh/laithsakka/238/head 2025-08-14T21:25:14.0078821Z * [new branch] gh/laithsakka/238/orig -> origin/gh/laithsakka/238/orig 2025-08-14T21:25:14.0081865Z * [new branch] gh/laithsakka/239/base -> origin/gh/laithsakka/239/base 2025-08-14T21:25:14.0082181Z * [new branch] gh/laithsakka/239/head -> origin/gh/laithsakka/239/head 2025-08-14T21:25:14.0082335Z * [new branch] gh/laithsakka/239/orig -> origin/gh/laithsakka/239/orig 2025-08-14T21:25:14.0082471Z * [new branch] gh/laithsakka/240/base -> origin/gh/laithsakka/240/base 2025-08-14T21:25:14.0082610Z * [new branch] gh/laithsakka/240/head -> origin/gh/laithsakka/240/head 2025-08-14T21:25:14.0082760Z * [new branch] gh/laithsakka/240/orig -> origin/gh/laithsakka/240/orig 2025-08-14T21:25:14.0087459Z * [new branch] gh/laithsakka/242/base -> origin/gh/laithsakka/242/base 2025-08-14T21:25:14.0087735Z * [new branch] gh/laithsakka/242/head -> origin/gh/laithsakka/242/head 2025-08-14T21:25:14.0087958Z * [new branch] gh/laithsakka/242/orig -> origin/gh/laithsakka/242/orig 2025-08-14T21:25:14.0088112Z * [new branch] gh/laithsakka/243/base -> origin/gh/laithsakka/243/base 2025-08-14T21:25:14.0088264Z * [new branch] gh/laithsakka/243/head -> origin/gh/laithsakka/243/head 2025-08-14T21:25:14.0088406Z * [new branch] gh/laithsakka/243/orig -> origin/gh/laithsakka/243/orig 2025-08-14T21:25:14.0088550Z * [new branch] gh/laithsakka/244/base -> origin/gh/laithsakka/244/base 2025-08-14T21:25:14.0088701Z * [new branch] gh/laithsakka/244/head -> origin/gh/laithsakka/244/head 2025-08-14T21:25:14.0097711Z * [new branch] gh/laithsakka/244/orig -> origin/gh/laithsakka/244/orig 2025-08-14T21:25:14.0097917Z * [new branch] gh/laithsakka/245/base -> origin/gh/laithsakka/245/base 2025-08-14T21:25:14.0098080Z * [new branch] gh/laithsakka/245/head -> origin/gh/laithsakka/245/head 2025-08-14T21:25:14.0098231Z * [new branch] gh/laithsakka/245/orig -> origin/gh/laithsakka/245/orig 2025-08-14T21:25:14.0098381Z * [new branch] gh/laithsakka/246/base -> origin/gh/laithsakka/246/base 2025-08-14T21:25:14.0098522Z * [new branch] gh/laithsakka/246/head -> origin/gh/laithsakka/246/head 2025-08-14T21:25:14.0098671Z * [new branch] gh/laithsakka/246/orig -> origin/gh/laithsakka/246/orig 2025-08-14T21:25:14.0098814Z * [new branch] gh/laithsakka/247/base -> origin/gh/laithsakka/247/base 2025-08-14T21:25:14.0098956Z * [new branch] gh/laithsakka/247/head -> origin/gh/laithsakka/247/head 2025-08-14T21:25:14.0099105Z * [new branch] gh/laithsakka/247/orig -> origin/gh/laithsakka/247/orig 2025-08-14T21:25:14.0099245Z * [new branch] gh/laithsakka/248/base -> origin/gh/laithsakka/248/base 2025-08-14T21:25:14.0099401Z * [new branch] gh/laithsakka/248/head -> origin/gh/laithsakka/248/head 2025-08-14T21:25:14.0099675Z * [new branch] gh/laithsakka/248/orig -> origin/gh/laithsakka/248/orig 2025-08-14T21:25:14.0099812Z * [new branch] gh/laithsakka/249/base -> origin/gh/laithsakka/249/base 2025-08-14T21:25:14.0099954Z * [new branch] gh/laithsakka/249/head -> origin/gh/laithsakka/249/head 2025-08-14T21:25:14.0107773Z * [new branch] gh/laithsakka/249/orig -> origin/gh/laithsakka/249/orig 2025-08-14T21:25:14.0109625Z * [new branch] gh/laithsakka/250/base -> origin/gh/laithsakka/250/base 2025-08-14T21:25:14.0109790Z * [new branch] gh/laithsakka/250/head -> origin/gh/laithsakka/250/head 2025-08-14T21:25:14.0109945Z * [new branch] gh/laithsakka/250/orig -> origin/gh/laithsakka/250/orig 2025-08-14T21:25:14.0110099Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-08-14T21:25:14.0110260Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-08-14T21:25:14.0110429Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-08-14T21:25:14.0110571Z * [new branch] gh/laithsakka/252/base -> origin/gh/laithsakka/252/base 2025-08-14T21:25:14.0110716Z * [new branch] gh/laithsakka/252/head -> origin/gh/laithsakka/252/head 2025-08-14T21:25:14.0110864Z * [new branch] gh/laithsakka/252/orig -> origin/gh/laithsakka/252/orig 2025-08-14T21:25:14.0111007Z * [new branch] gh/laithsakka/253/base -> origin/gh/laithsakka/253/base 2025-08-14T21:25:14.0111152Z * [new branch] gh/laithsakka/253/head -> origin/gh/laithsakka/253/head 2025-08-14T21:25:14.0111295Z * [new branch] gh/laithsakka/253/orig -> origin/gh/laithsakka/253/orig 2025-08-14T21:25:14.0111618Z * [new branch] gh/laithsakka/254/base -> origin/gh/laithsakka/254/base 2025-08-14T21:25:14.0111849Z * [new branch] gh/laithsakka/254/head -> origin/gh/laithsakka/254/head 2025-08-14T21:25:14.0112547Z * [new branch] gh/laithsakka/254/orig -> origin/gh/laithsakka/254/orig 2025-08-14T21:25:14.0112750Z * [new branch] gh/laithsakka/255/base -> origin/gh/laithsakka/255/base 2025-08-14T21:25:14.0112904Z * [new branch] gh/laithsakka/255/head -> origin/gh/laithsakka/255/head 2025-08-14T21:25:14.0113071Z * [new branch] gh/laithsakka/255/orig -> origin/gh/laithsakka/255/orig 2025-08-14T21:25:14.0113225Z * [new branch] gh/laithsakka/256/base -> origin/gh/laithsakka/256/base 2025-08-14T21:25:14.0113623Z * [new branch] gh/laithsakka/256/head -> origin/gh/laithsakka/256/head 2025-08-14T21:25:14.0114328Z * [new branch] gh/laithsakka/256/orig -> origin/gh/laithsakka/256/orig 2025-08-14T21:25:14.0115502Z * [new branch] gh/laithsakka/257/base -> origin/gh/laithsakka/257/base 2025-08-14T21:25:14.0115764Z * [new branch] gh/laithsakka/257/head -> origin/gh/laithsakka/257/head 2025-08-14T21:25:14.0116969Z * [new branch] gh/laithsakka/257/orig -> origin/gh/laithsakka/257/orig 2025-08-14T21:25:14.0117455Z * [new branch] gh/laithsakka/258/base -> origin/gh/laithsakka/258/base 2025-08-14T21:25:14.0119897Z * [new branch] gh/laithsakka/258/head -> origin/gh/laithsakka/258/head 2025-08-14T21:25:14.0120247Z * [new branch] gh/laithsakka/258/orig -> origin/gh/laithsakka/258/orig 2025-08-14T21:25:14.0120450Z * [new branch] gh/laithsakka/259/base -> origin/gh/laithsakka/259/base 2025-08-14T21:25:14.0120618Z * [new branch] gh/laithsakka/259/head -> origin/gh/laithsakka/259/head 2025-08-14T21:25:14.0121174Z * [new branch] gh/laithsakka/259/orig -> origin/gh/laithsakka/259/orig 2025-08-14T21:25:14.0122403Z * [new branch] gh/laithsakka/260/base -> origin/gh/laithsakka/260/base 2025-08-14T21:25:14.0122638Z * [new branch] gh/laithsakka/260/head -> origin/gh/laithsakka/260/head 2025-08-14T21:25:14.0125328Z * [new branch] gh/laithsakka/260/orig -> origin/gh/laithsakka/260/orig 2025-08-14T21:25:14.0125652Z * [new branch] gh/laithsakka/261/base -> origin/gh/laithsakka/261/base 2025-08-14T21:25:14.0125812Z * [new branch] gh/laithsakka/261/head -> origin/gh/laithsakka/261/head 2025-08-14T21:25:14.0126041Z * [new branch] gh/laithsakka/261/orig -> origin/gh/laithsakka/261/orig 2025-08-14T21:25:14.0127599Z * [new branch] gh/laithsakka/262/base -> origin/gh/laithsakka/262/base 2025-08-14T21:25:14.0127779Z * [new branch] gh/laithsakka/262/head -> origin/gh/laithsakka/262/head 2025-08-14T21:25:14.0132777Z * [new branch] gh/laithsakka/262/orig -> origin/gh/laithsakka/262/orig 2025-08-14T21:25:14.0132972Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-08-14T21:25:14.0133123Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-08-14T21:25:14.0133260Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-08-14T21:25:14.0133399Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-08-14T21:25:14.0133541Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-08-14T21:25:14.0133675Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-08-14T21:25:14.0138858Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-08-14T21:25:14.0139228Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-08-14T21:25:14.0139721Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-08-14T21:25:14.0140047Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-08-14T21:25:14.0140714Z * [new branch] gh/lucaskabela/10/base -> origin/gh/lucaskabela/10/base 2025-08-14T21:25:14.0140928Z * [new branch] gh/lucaskabela/10/head -> origin/gh/lucaskabela/10/head 2025-08-14T21:25:14.0141099Z * [new branch] gh/lucaskabela/10/orig -> origin/gh/lucaskabela/10/orig 2025-08-14T21:25:14.0141280Z * [new branch] gh/lucaskabela/11/base -> origin/gh/lucaskabela/11/base 2025-08-14T21:25:14.0141611Z * [new branch] gh/lucaskabela/11/head -> origin/gh/lucaskabela/11/head 2025-08-14T21:25:14.0141808Z * [new branch] gh/lucaskabela/11/orig -> origin/gh/lucaskabela/11/orig 2025-08-14T21:25:14.0146087Z * [new branch] gh/lucaskabela/12/base -> origin/gh/lucaskabela/12/base 2025-08-14T21:25:14.0146472Z * [new branch] gh/lucaskabela/12/head -> origin/gh/lucaskabela/12/head 2025-08-14T21:25:14.0146749Z * [new branch] gh/lucaskabela/12/orig -> origin/gh/lucaskabela/12/orig 2025-08-14T21:25:14.0146940Z * [new branch] gh/lucaskabela/13/base -> origin/gh/lucaskabela/13/base 2025-08-14T21:25:14.0147244Z * [new branch] gh/lucaskabela/13/head -> origin/gh/lucaskabela/13/head 2025-08-14T21:25:14.0147941Z * [new branch] gh/lucaskabela/13/orig -> origin/gh/lucaskabela/13/orig 2025-08-14T21:25:14.0148278Z * [new branch] gh/lucaskabela/14/base -> origin/gh/lucaskabela/14/base 2025-08-14T21:25:14.0148455Z * [new branch] gh/lucaskabela/14/head -> origin/gh/lucaskabela/14/head 2025-08-14T21:25:14.0148637Z * [new branch] gh/lucaskabela/14/orig -> origin/gh/lucaskabela/14/orig 2025-08-14T21:25:14.0148800Z * [new branch] gh/lucaskabela/15/base -> origin/gh/lucaskabela/15/base 2025-08-14T21:25:14.0149350Z * [new branch] gh/lucaskabela/15/head -> origin/gh/lucaskabela/15/head 2025-08-14T21:25:14.0150427Z * [new branch] gh/lucaskabela/15/orig -> origin/gh/lucaskabela/15/orig 2025-08-14T21:25:14.0151072Z * [new branch] gh/lucaskabela/16/base -> origin/gh/lucaskabela/16/base 2025-08-14T21:25:14.0151347Z * [new branch] gh/lucaskabela/16/head -> origin/gh/lucaskabela/16/head 2025-08-14T21:25:14.0153269Z * [new branch] gh/lucaskabela/16/orig -> origin/gh/lucaskabela/16/orig 2025-08-14T21:25:14.0153473Z * [new branch] gh/lucaskabela/17/base -> origin/gh/lucaskabela/17/base 2025-08-14T21:25:14.0154037Z * [new branch] gh/lucaskabela/17/head -> origin/gh/lucaskabela/17/head 2025-08-14T21:25:14.0155107Z * [new branch] gh/lucaskabela/17/orig -> origin/gh/lucaskabela/17/orig 2025-08-14T21:25:14.0155754Z * [new branch] gh/lucaskabela/2/base -> origin/gh/lucaskabela/2/base 2025-08-14T21:25:14.0156828Z * [new branch] gh/lucaskabela/2/head -> origin/gh/lucaskabela/2/head 2025-08-14T21:25:14.0156994Z * [new branch] gh/lucaskabela/2/orig -> origin/gh/lucaskabela/2/orig 2025-08-14T21:25:14.0160453Z * [new branch] gh/lucaskabela/3/base -> origin/gh/lucaskabela/3/base 2025-08-14T21:25:14.0165913Z * [new branch] gh/lucaskabela/3/head -> origin/gh/lucaskabela/3/head 2025-08-14T21:25:14.0171491Z * [new branch] gh/lucaskabela/3/orig -> origin/gh/lucaskabela/3/orig 2025-08-14T21:25:14.0175695Z * [new branch] gh/lucaskabela/4/base -> origin/gh/lucaskabela/4/base 2025-08-14T21:25:14.0180475Z * [new branch] gh/lucaskabela/4/head -> origin/gh/lucaskabela/4/head 2025-08-14T21:25:14.0186273Z * [new branch] gh/lucaskabela/4/orig -> origin/gh/lucaskabela/4/orig 2025-08-14T21:25:14.0186498Z * [new branch] gh/lucaskabela/5/base -> origin/gh/lucaskabela/5/base 2025-08-14T21:25:14.0186660Z * [new branch] gh/lucaskabela/5/head -> origin/gh/lucaskabela/5/head 2025-08-14T21:25:14.0186809Z * [new branch] gh/lucaskabela/5/orig -> origin/gh/lucaskabela/5/orig 2025-08-14T21:25:14.0186946Z * [new branch] gh/lucaskabela/6/base -> origin/gh/lucaskabela/6/base 2025-08-14T21:25:14.0187114Z * [new branch] gh/lucaskabela/6/head -> origin/gh/lucaskabela/6/head 2025-08-14T21:25:14.0187255Z * [new branch] gh/lucaskabela/6/orig -> origin/gh/lucaskabela/6/orig 2025-08-14T21:25:14.0187398Z * [new branch] gh/lucaskabela/7/base -> origin/gh/lucaskabela/7/base 2025-08-14T21:25:14.0187537Z * [new branch] gh/lucaskabela/7/head -> origin/gh/lucaskabela/7/head 2025-08-14T21:25:14.0187675Z * [new branch] gh/lucaskabela/7/orig -> origin/gh/lucaskabela/7/orig 2025-08-14T21:25:14.0187822Z * [new branch] gh/lucaskabela/8/base -> origin/gh/lucaskabela/8/base 2025-08-14T21:25:14.0187958Z * [new branch] gh/lucaskabela/8/head -> origin/gh/lucaskabela/8/head 2025-08-14T21:25:14.0188099Z * [new branch] gh/lucaskabela/8/orig -> origin/gh/lucaskabela/8/orig 2025-08-14T21:25:14.0188234Z * [new branch] gh/lucaskabela/9/base -> origin/gh/lucaskabela/9/base 2025-08-14T21:25:14.0188371Z * [new branch] gh/lucaskabela/9/head -> origin/gh/lucaskabela/9/head 2025-08-14T21:25:14.0188526Z * [new branch] gh/lucaskabela/9/orig -> origin/gh/lucaskabela/9/orig 2025-08-14T21:25:14.0188648Z * [new branch] gh/lw/1/base -> origin/gh/lw/1/base 2025-08-14T21:25:14.0188764Z * [new branch] gh/lw/1/head -> origin/gh/lw/1/head 2025-08-14T21:25:14.0188877Z * [new branch] gh/lw/1/orig -> origin/gh/lw/1/orig 2025-08-14T21:25:14.0189038Z * [new branch] gh/lw/2/base -> origin/gh/lw/2/base 2025-08-14T21:25:14.0189155Z * [new branch] gh/lw/2/head -> origin/gh/lw/2/head 2025-08-14T21:25:14.0189267Z * [new branch] gh/lw/2/orig -> origin/gh/lw/2/orig 2025-08-14T21:25:14.0189375Z * [new branch] gh/lw/3/base -> origin/gh/lw/3/base 2025-08-14T21:25:14.0189494Z * [new branch] gh/lw/3/head -> origin/gh/lw/3/head 2025-08-14T21:25:14.0189603Z * [new branch] gh/lw/3/orig -> origin/gh/lw/3/orig 2025-08-14T21:25:14.0189742Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-08-14T21:25:14.0189879Z * [new branch] gh/malfet/330/base -> origin/gh/malfet/330/base 2025-08-14T21:25:14.0190012Z * [new branch] gh/malfet/330/head -> origin/gh/malfet/330/head 2025-08-14T21:25:14.0190149Z * [new branch] gh/malfet/330/orig -> origin/gh/malfet/330/orig 2025-08-14T21:25:14.0190275Z * [new branch] gh/malfet/396/base -> origin/gh/malfet/396/base 2025-08-14T21:25:14.0190409Z * [new branch] gh/malfet/396/head -> origin/gh/malfet/396/head 2025-08-14T21:25:14.0190533Z * [new branch] gh/malfet/396/orig -> origin/gh/malfet/396/orig 2025-08-14T21:25:14.0190668Z * [new branch] gh/malfet/397/base -> origin/gh/malfet/397/base 2025-08-14T21:25:14.0190807Z * [new branch] gh/malfet/397/head -> origin/gh/malfet/397/head 2025-08-14T21:25:14.0190938Z * [new branch] gh/malfet/397/orig -> origin/gh/malfet/397/orig 2025-08-14T21:25:14.0191085Z * [new branch] gh/malfet/398/base -> origin/gh/malfet/398/base 2025-08-14T21:25:14.0191264Z * [new branch] gh/malfet/398/head -> origin/gh/malfet/398/head 2025-08-14T21:25:14.0191400Z * [new branch] gh/malfet/398/orig -> origin/gh/malfet/398/orig 2025-08-14T21:25:14.0191539Z * [new branch] gh/malfet/399/base -> origin/gh/malfet/399/base 2025-08-14T21:25:14.0191682Z * [new branch] gh/malfet/399/head -> origin/gh/malfet/399/head 2025-08-14T21:25:14.0191828Z * [new branch] gh/malfet/399/orig -> origin/gh/malfet/399/orig 2025-08-14T21:25:14.0191973Z * [new branch] gh/malfet/414/base -> origin/gh/malfet/414/base 2025-08-14T21:25:14.0192108Z * [new branch] gh/malfet/414/head -> origin/gh/malfet/414/head 2025-08-14T21:25:14.0193318Z * [new branch] gh/malfet/414/orig -> origin/gh/malfet/414/orig 2025-08-14T21:25:14.0193863Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-08-14T21:25:14.0194651Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-08-14T21:25:14.0195183Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-08-14T21:25:14.0199447Z * [new branch] gh/malfet/418/base -> origin/gh/malfet/418/base 2025-08-14T21:25:14.0199630Z * [new branch] gh/malfet/418/head -> origin/gh/malfet/418/head 2025-08-14T21:25:14.0199766Z * [new branch] gh/malfet/418/orig -> origin/gh/malfet/418/orig 2025-08-14T21:25:14.0199907Z * [new branch] gh/malfet/422/base -> origin/gh/malfet/422/base 2025-08-14T21:25:14.0200039Z * [new branch] gh/malfet/422/head -> origin/gh/malfet/422/head 2025-08-14T21:25:14.0200184Z * [new branch] gh/malfet/422/orig -> origin/gh/malfet/422/orig 2025-08-14T21:25:14.0200957Z * [new branch] gh/malfet/438/base -> origin/gh/malfet/438/base 2025-08-14T21:25:14.0201529Z * [new branch] gh/malfet/438/head -> origin/gh/malfet/438/head 2025-08-14T21:25:14.0202199Z * [new branch] gh/malfet/438/orig -> origin/gh/malfet/438/orig 2025-08-14T21:25:14.0206865Z * [new branch] gh/malfet/439/base -> origin/gh/malfet/439/base 2025-08-14T21:25:14.0207240Z * [new branch] gh/malfet/439/head -> origin/gh/malfet/439/head 2025-08-14T21:25:14.0207382Z * [new branch] gh/malfet/439/orig -> origin/gh/malfet/439/orig 2025-08-14T21:25:14.0207521Z * [new branch] gh/malfet/440/base -> origin/gh/malfet/440/base 2025-08-14T21:25:14.0207666Z * [new branch] gh/malfet/440/head -> origin/gh/malfet/440/head 2025-08-14T21:25:14.0207803Z * [new branch] gh/malfet/440/orig -> origin/gh/malfet/440/orig 2025-08-14T21:25:14.0207952Z * [new branch] gh/malfet/441/base -> origin/gh/malfet/441/base 2025-08-14T21:25:14.0208479Z * [new branch] gh/malfet/441/head -> origin/gh/malfet/441/head 2025-08-14T21:25:14.0209425Z * [new branch] gh/malfet/441/orig -> origin/gh/malfet/441/orig 2025-08-14T21:25:14.0214375Z * [new branch] gh/malfet/442/base -> origin/gh/malfet/442/base 2025-08-14T21:25:14.0219051Z * [new branch] gh/malfet/442/head -> origin/gh/malfet/442/head 2025-08-14T21:25:14.0223778Z * [new branch] gh/malfet/442/orig -> origin/gh/malfet/442/orig 2025-08-14T21:25:14.0227931Z * [new branch] gh/malfet/443/base -> origin/gh/malfet/443/base 2025-08-14T21:25:14.0230190Z * [new branch] gh/malfet/443/head -> origin/gh/malfet/443/head 2025-08-14T21:25:14.0230551Z * [new branch] gh/malfet/443/orig -> origin/gh/malfet/443/orig 2025-08-14T21:25:14.0230931Z * [new branch] gh/malfet/444/base -> origin/gh/malfet/444/base 2025-08-14T21:25:14.0231216Z * [new branch] gh/malfet/444/head -> origin/gh/malfet/444/head 2025-08-14T21:25:14.0231360Z * [new branch] gh/malfet/444/orig -> origin/gh/malfet/444/orig 2025-08-14T21:25:14.0231500Z * [new branch] gh/malfet/445/base -> origin/gh/malfet/445/base 2025-08-14T21:25:14.0231721Z * [new branch] gh/malfet/445/head -> origin/gh/malfet/445/head 2025-08-14T21:25:14.0231862Z * [new branch] gh/malfet/445/orig -> origin/gh/malfet/445/orig 2025-08-14T21:25:14.0232073Z * [new branch] gh/malfet/446/base -> origin/gh/malfet/446/base 2025-08-14T21:25:14.0232218Z * [new branch] gh/malfet/446/head -> origin/gh/malfet/446/head 2025-08-14T21:25:14.0232431Z * [new branch] gh/malfet/446/orig -> origin/gh/malfet/446/orig 2025-08-14T21:25:14.0233172Z * [new branch] gh/malfet/447/base -> origin/gh/malfet/447/base 2025-08-14T21:25:14.0233525Z * [new branch] gh/malfet/447/head -> origin/gh/malfet/447/head 2025-08-14T21:25:14.0233684Z * [new branch] gh/malfet/448/base -> origin/gh/malfet/448/base 2025-08-14T21:25:14.0233918Z * [new branch] gh/malfet/448/head -> origin/gh/malfet/448/head 2025-08-14T21:25:14.0234191Z * [new branch] gh/malfet/449/base -> origin/gh/malfet/449/base 2025-08-14T21:25:14.0234337Z * [new branch] gh/malfet/449/head -> origin/gh/malfet/449/head 2025-08-14T21:25:14.0234793Z * [new branch] gh/malfet/450/base -> origin/gh/malfet/450/base 2025-08-14T21:25:14.0234957Z * [new branch] gh/malfet/450/head -> origin/gh/malfet/450/head 2025-08-14T21:25:14.0235104Z * [new branch] gh/malfet/451/base -> origin/gh/malfet/451/base 2025-08-14T21:25:14.0235256Z * [new branch] gh/malfet/451/head -> origin/gh/malfet/451/head 2025-08-14T21:25:14.0235485Z * [new branch] gh/malfet/452/base -> origin/gh/malfet/452/base 2025-08-14T21:25:14.0235626Z * [new branch] gh/malfet/452/head -> origin/gh/malfet/452/head 2025-08-14T21:25:14.0235763Z * [new branch] gh/malfet/452/orig -> origin/gh/malfet/452/orig 2025-08-14T21:25:14.0235907Z * [new branch] gh/malfet/453/base -> origin/gh/malfet/453/base 2025-08-14T21:25:14.0236129Z * [new branch] gh/malfet/453/head -> origin/gh/malfet/453/head 2025-08-14T21:25:14.0236281Z * [new branch] gh/malfet/453/orig -> origin/gh/malfet/453/orig 2025-08-14T21:25:14.0236417Z * [new branch] gh/malfet/454/base -> origin/gh/malfet/454/base 2025-08-14T21:25:14.0236581Z * [new branch] gh/malfet/454/head -> origin/gh/malfet/454/head 2025-08-14T21:25:14.0236747Z * [new branch] gh/malfet/454/orig -> origin/gh/malfet/454/orig 2025-08-14T21:25:14.0236893Z * [new branch] gh/malfet/455/base -> origin/gh/malfet/455/base 2025-08-14T21:25:14.0266230Z * [new branch] gh/malfet/455/head -> origin/gh/malfet/455/head 2025-08-14T21:25:14.0266595Z * [new branch] gh/malfet/455/orig -> origin/gh/malfet/455/orig 2025-08-14T21:25:14.0266886Z * [new branch] gh/malfet/456/base -> origin/gh/malfet/456/base 2025-08-14T21:25:14.0267176Z * [new branch] gh/malfet/456/head -> origin/gh/malfet/456/head 2025-08-14T21:25:14.0267305Z * [new branch] gh/malfet/456/orig -> origin/gh/malfet/456/orig 2025-08-14T21:25:14.0267444Z * [new branch] gh/malfet/457/base -> origin/gh/malfet/457/base 2025-08-14T21:25:14.0267637Z * [new branch] gh/malfet/457/head -> origin/gh/malfet/457/head 2025-08-14T21:25:14.0267903Z * [new branch] gh/malfet/457/orig -> origin/gh/malfet/457/orig 2025-08-14T21:25:14.0268039Z * [new branch] gh/malfet/458/base -> origin/gh/malfet/458/base 2025-08-14T21:25:14.0268165Z * [new branch] gh/malfet/458/head -> origin/gh/malfet/458/head 2025-08-14T21:25:14.0268299Z * [new branch] gh/malfet/458/orig -> origin/gh/malfet/458/orig 2025-08-14T21:25:14.0268423Z * [new branch] gh/malfet/459/base -> origin/gh/malfet/459/base 2025-08-14T21:25:14.0268557Z * [new branch] gh/malfet/459/head -> origin/gh/malfet/459/head 2025-08-14T21:25:14.0268682Z * [new branch] gh/malfet/459/orig -> origin/gh/malfet/459/orig 2025-08-14T21:25:14.0268807Z * [new branch] gh/malfet/460/base -> origin/gh/malfet/460/base 2025-08-14T21:25:14.0268942Z * [new branch] gh/malfet/460/head -> origin/gh/malfet/460/head 2025-08-14T21:25:14.0269072Z * [new branch] gh/malfet/460/orig -> origin/gh/malfet/460/orig 2025-08-14T21:25:14.0269208Z * [new branch] gh/malfet/461/base -> origin/gh/malfet/461/base 2025-08-14T21:25:14.0269332Z * [new branch] gh/malfet/461/head -> origin/gh/malfet/461/head 2025-08-14T21:25:14.0269456Z * [new branch] gh/malfet/461/orig -> origin/gh/malfet/461/orig 2025-08-14T21:25:14.0269589Z * [new branch] gh/malfet/462/base -> origin/gh/malfet/462/base 2025-08-14T21:25:14.0269714Z * [new branch] gh/malfet/462/head -> origin/gh/malfet/462/head 2025-08-14T21:25:14.0269845Z * [new branch] gh/malfet/462/orig -> origin/gh/malfet/462/orig 2025-08-14T21:25:14.0269969Z * [new branch] gh/malfet/463/base -> origin/gh/malfet/463/base 2025-08-14T21:25:14.0270098Z * [new branch] gh/malfet/463/head -> origin/gh/malfet/463/head 2025-08-14T21:25:14.0270240Z * [new branch] gh/malfet/463/orig -> origin/gh/malfet/463/orig 2025-08-14T21:25:14.0270438Z * [new branch] gh/malfet/464/base -> origin/gh/malfet/464/base 2025-08-14T21:25:14.0270566Z * [new branch] gh/malfet/464/head -> origin/gh/malfet/464/head 2025-08-14T21:25:14.0270703Z * [new branch] gh/malfet/464/orig -> origin/gh/malfet/464/orig 2025-08-14T21:25:14.0270834Z * [new branch] gh/malfet/465/base -> origin/gh/malfet/465/base 2025-08-14T21:25:14.0270971Z * [new branch] gh/malfet/465/head -> origin/gh/malfet/465/head 2025-08-14T21:25:14.0271101Z * [new branch] gh/malfet/465/orig -> origin/gh/malfet/465/orig 2025-08-14T21:25:14.0271230Z * [new branch] gh/malfet/466/base -> origin/gh/malfet/466/base 2025-08-14T21:25:14.0271368Z * [new branch] gh/malfet/466/head -> origin/gh/malfet/466/head 2025-08-14T21:25:14.0271500Z * [new branch] gh/malfet/466/orig -> origin/gh/malfet/466/orig 2025-08-14T21:25:14.0271641Z * [new branch] gh/malfet/467/base -> origin/gh/malfet/467/base 2025-08-14T21:25:14.0271771Z * [new branch] gh/malfet/467/head -> origin/gh/malfet/467/head 2025-08-14T21:25:14.0271900Z * [new branch] gh/malfet/467/orig -> origin/gh/malfet/467/orig 2025-08-14T21:25:14.0272039Z * [new branch] gh/malfet/468/base -> origin/gh/malfet/468/base 2025-08-14T21:25:14.0272168Z * [new branch] gh/malfet/468/head -> origin/gh/malfet/468/head 2025-08-14T21:25:14.0272313Z * [new branch] gh/malfet/468/orig -> origin/gh/malfet/468/orig 2025-08-14T21:25:14.0272442Z * [new branch] gh/malfet/469/base -> origin/gh/malfet/469/base 2025-08-14T21:25:14.0272574Z * [new branch] gh/malfet/469/head -> origin/gh/malfet/469/head 2025-08-14T21:25:14.0272743Z * [new branch] gh/malfet/469/orig -> origin/gh/malfet/469/orig 2025-08-14T21:25:14.0272890Z * [new branch] gh/malfet/470/base -> origin/gh/malfet/470/base 2025-08-14T21:25:14.0273468Z * [new branch] gh/malfet/470/head -> origin/gh/malfet/470/head 2025-08-14T21:25:14.0273656Z * [new branch] gh/malfet/470/orig -> origin/gh/malfet/470/orig 2025-08-14T21:25:14.0273814Z * [new branch] gh/malfet/471/base -> origin/gh/malfet/471/base 2025-08-14T21:25:14.0273965Z * [new branch] gh/malfet/471/head -> origin/gh/malfet/471/head 2025-08-14T21:25:14.0274112Z * [new branch] gh/malfet/471/orig -> origin/gh/malfet/471/orig 2025-08-14T21:25:14.0274274Z * [new branch] gh/malfet/472/base -> origin/gh/malfet/472/base 2025-08-14T21:25:14.0274444Z * [new branch] gh/malfet/472/head -> origin/gh/malfet/472/head 2025-08-14T21:25:14.0274632Z * [new branch] gh/malfet/472/orig -> origin/gh/malfet/472/orig 2025-08-14T21:25:14.0274802Z * [new branch] gh/malfet/473/base -> origin/gh/malfet/473/base 2025-08-14T21:25:14.0275236Z * [new branch] gh/malfet/473/head -> origin/gh/malfet/473/head 2025-08-14T21:25:14.0276590Z * [new branch] gh/malfet/473/orig -> origin/gh/malfet/473/orig 2025-08-14T21:25:14.0280850Z * [new branch] gh/malfet/474/base -> origin/gh/malfet/474/base 2025-08-14T21:25:14.0281549Z * [new branch] gh/malfet/474/head -> origin/gh/malfet/474/head 2025-08-14T21:25:14.0281722Z * [new branch] gh/malfet/474/orig -> origin/gh/malfet/474/orig 2025-08-14T21:25:14.0281890Z * [new branch] gh/malfet/475/base -> origin/gh/malfet/475/base 2025-08-14T21:25:14.0282052Z * [new branch] gh/malfet/475/head -> origin/gh/malfet/475/head 2025-08-14T21:25:14.0282468Z * [new branch] gh/malfet/475/orig -> origin/gh/malfet/475/orig 2025-08-14T21:25:14.0282625Z * [new branch] gh/malfet/476/base -> origin/gh/malfet/476/base 2025-08-14T21:25:14.0285455Z * [new branch] gh/malfet/476/head -> origin/gh/malfet/476/head 2025-08-14T21:25:14.0285615Z * [new branch] gh/malfet/476/orig -> origin/gh/malfet/476/orig 2025-08-14T21:25:14.0285914Z * [new branch] gh/malfet/477/base -> origin/gh/malfet/477/base 2025-08-14T21:25:14.0286045Z * [new branch] gh/malfet/477/head -> origin/gh/malfet/477/head 2025-08-14T21:25:14.0286171Z * [new branch] gh/malfet/477/orig -> origin/gh/malfet/477/orig 2025-08-14T21:25:14.0289394Z * [new branch] gh/malfet/478/base -> origin/gh/malfet/478/base 2025-08-14T21:25:14.0289713Z * [new branch] gh/malfet/478/head -> origin/gh/malfet/478/head 2025-08-14T21:25:14.0289863Z * [new branch] gh/malfet/478/orig -> origin/gh/malfet/478/orig 2025-08-14T21:25:14.0289994Z * [new branch] gh/malfet/479/base -> origin/gh/malfet/479/base 2025-08-14T21:25:14.0290121Z * [new branch] gh/malfet/479/head -> origin/gh/malfet/479/head 2025-08-14T21:25:14.0290257Z * [new branch] gh/malfet/479/orig -> origin/gh/malfet/479/orig 2025-08-14T21:25:14.0290382Z * [new branch] gh/malfet/480/base -> origin/gh/malfet/480/base 2025-08-14T21:25:14.0300552Z * [new branch] gh/malfet/480/head -> origin/gh/malfet/480/head 2025-08-14T21:25:14.0300729Z * [new branch] gh/malfet/480/orig -> origin/gh/malfet/480/orig 2025-08-14T21:25:14.0300880Z * [new branch] gh/malfet/481/base -> origin/gh/malfet/481/base 2025-08-14T21:25:14.0301211Z * [new branch] gh/malfet/481/head -> origin/gh/malfet/481/head 2025-08-14T21:25:14.0301386Z * [new branch] gh/malfet/481/orig -> origin/gh/malfet/481/orig 2025-08-14T21:25:14.0301556Z * [new branch] gh/malfet/482/base -> origin/gh/malfet/482/base 2025-08-14T21:25:14.0301705Z * [new branch] gh/malfet/482/head -> origin/gh/malfet/482/head 2025-08-14T21:25:14.0301855Z * [new branch] gh/malfet/482/orig -> origin/gh/malfet/482/orig 2025-08-14T21:25:14.0302007Z * [new branch] gh/malfet/483/base -> origin/gh/malfet/483/base 2025-08-14T21:25:14.0302149Z * [new branch] gh/malfet/483/head -> origin/gh/malfet/483/head 2025-08-14T21:25:14.0302289Z * [new branch] gh/malfet/483/orig -> origin/gh/malfet/483/orig 2025-08-14T21:25:14.0302452Z * [new branch] gh/malfet/484/base -> origin/gh/malfet/484/base 2025-08-14T21:25:14.0302597Z * [new branch] gh/malfet/484/head -> origin/gh/malfet/484/head 2025-08-14T21:25:14.0302737Z * [new branch] gh/malfet/484/orig -> origin/gh/malfet/484/orig 2025-08-14T21:25:14.0302865Z * [new branch] gh/malfet/485/base -> origin/gh/malfet/485/base 2025-08-14T21:25:14.0303009Z * [new branch] gh/malfet/485/head -> origin/gh/malfet/485/head 2025-08-14T21:25:14.0306669Z * [new branch] gh/malfet/485/orig -> origin/gh/malfet/485/orig 2025-08-14T21:25:14.0306840Z * [new branch] gh/malfet/486/base -> origin/gh/malfet/486/base 2025-08-14T21:25:14.0306994Z * [new branch] gh/malfet/486/head -> origin/gh/malfet/486/head 2025-08-14T21:25:14.0307136Z * [new branch] gh/malfet/486/orig -> origin/gh/malfet/486/orig 2025-08-14T21:25:14.0307278Z * [new branch] gh/malfet/487/base -> origin/gh/malfet/487/base 2025-08-14T21:25:14.0307435Z * [new branch] gh/malfet/487/head -> origin/gh/malfet/487/head 2025-08-14T21:25:14.0307756Z * [new branch] gh/malfet/487/orig -> origin/gh/malfet/487/orig 2025-08-14T21:25:14.0307902Z * [new branch] gh/malfet/488/base -> origin/gh/malfet/488/base 2025-08-14T21:25:14.0308595Z * [new branch] gh/malfet/488/head -> origin/gh/malfet/488/head 2025-08-14T21:25:14.0309381Z * [new branch] gh/malfet/488/orig -> origin/gh/malfet/488/orig 2025-08-14T21:25:14.0309654Z * [new branch] gh/malfet/489/base -> origin/gh/malfet/489/base 2025-08-14T21:25:14.0309948Z * [new branch] gh/malfet/489/head -> origin/gh/malfet/489/head 2025-08-14T21:25:14.0310113Z * [new branch] gh/malfet/489/orig -> origin/gh/malfet/489/orig 2025-08-14T21:25:14.0310353Z * [new branch] gh/malfet/490/base -> origin/gh/malfet/490/base 2025-08-14T21:25:14.0310529Z * [new branch] gh/malfet/490/head -> origin/gh/malfet/490/head 2025-08-14T21:25:14.0310693Z * [new branch] gh/malfet/490/orig -> origin/gh/malfet/490/orig 2025-08-14T21:25:14.0311723Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-08-14T21:25:14.0315165Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-08-14T21:25:14.0315380Z * [new branch] gh/manuelcandales/10/base -> origin/gh/manuelcandales/10/base 2025-08-14T21:25:14.0315541Z * [new branch] gh/manuelcandales/10/head -> origin/gh/manuelcandales/10/head 2025-08-14T21:25:14.0315706Z * [new branch] gh/manuelcandales/10/orig -> origin/gh/manuelcandales/10/orig 2025-08-14T21:25:14.0315883Z * [new branch] gh/manuelcandales/9/base -> origin/gh/manuelcandales/9/base 2025-08-14T21:25:14.0316385Z * [new branch] gh/manuelcandales/9/head -> origin/gh/manuelcandales/9/head 2025-08-14T21:25:14.0317347Z * [new branch] gh/manuelcandales/9/orig -> origin/gh/manuelcandales/9/orig 2025-08-14T21:25:14.0324737Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-08-14T21:25:14.0326854Z * [new branch] gh/masnesral/204/base -> origin/gh/masnesral/204/base 2025-08-14T21:25:14.0327175Z * [new branch] gh/masnesral/204/head -> origin/gh/masnesral/204/head 2025-08-14T21:25:14.0332712Z * [new branch] gh/masnesral/204/orig -> origin/gh/masnesral/204/orig 2025-08-14T21:25:14.0334969Z * [new branch] gh/masnesral/223/base -> origin/gh/masnesral/223/base 2025-08-14T21:25:14.0335232Z * [new branch] gh/masnesral/223/head -> origin/gh/masnesral/223/head 2025-08-14T21:25:14.0339243Z * [new branch] gh/masnesral/223/orig -> origin/gh/masnesral/223/orig 2025-08-14T21:25:14.0339587Z * [new branch] gh/masnesral/224/base -> origin/gh/masnesral/224/base 2025-08-14T21:25:14.0339814Z * [new branch] gh/masnesral/224/head -> origin/gh/masnesral/224/head 2025-08-14T21:25:14.0339972Z * [new branch] gh/masnesral/224/orig -> origin/gh/masnesral/224/orig 2025-08-14T21:25:14.0340120Z * [new branch] gh/masnesral/225/base -> origin/gh/masnesral/225/base 2025-08-14T21:25:14.0340392Z * [new branch] gh/masnesral/225/head -> origin/gh/masnesral/225/head 2025-08-14T21:25:14.0340532Z * [new branch] gh/masnesral/225/orig -> origin/gh/masnesral/225/orig 2025-08-14T21:25:14.0340667Z * [new branch] gh/masnesral/226/base -> origin/gh/masnesral/226/base 2025-08-14T21:25:14.0340972Z * [new branch] gh/masnesral/226/head -> origin/gh/masnesral/226/head 2025-08-14T21:25:14.0341113Z * [new branch] gh/masnesral/226/orig -> origin/gh/masnesral/226/orig 2025-08-14T21:25:14.0341598Z * [new branch] gh/masnesral/227/base -> origin/gh/masnesral/227/base 2025-08-14T21:25:14.0341974Z * [new branch] gh/masnesral/227/head -> origin/gh/masnesral/227/head 2025-08-14T21:25:14.0342120Z * [new branch] gh/masnesral/227/orig -> origin/gh/masnesral/227/orig 2025-08-14T21:25:14.0342253Z * [new branch] gh/masnesral/228/base -> origin/gh/masnesral/228/base 2025-08-14T21:25:14.0342386Z * [new branch] gh/masnesral/228/head -> origin/gh/masnesral/228/head 2025-08-14T21:25:14.0342528Z * [new branch] gh/masnesral/228/orig -> origin/gh/masnesral/228/orig 2025-08-14T21:25:14.0342659Z * [new branch] gh/masnesral/229/base -> origin/gh/masnesral/229/base 2025-08-14T21:25:14.0342797Z * [new branch] gh/masnesral/229/head -> origin/gh/masnesral/229/head 2025-08-14T21:25:14.0342931Z * [new branch] gh/masnesral/229/orig -> origin/gh/masnesral/229/orig 2025-08-14T21:25:14.0343070Z * [new branch] gh/masnesral/230/base -> origin/gh/masnesral/230/base 2025-08-14T21:25:14.0343210Z * [new branch] gh/masnesral/230/head -> origin/gh/masnesral/230/head 2025-08-14T21:25:14.0343340Z * [new branch] gh/masnesral/230/orig -> origin/gh/masnesral/230/orig 2025-08-14T21:25:14.0343490Z * [new branch] gh/masnesral/231/base -> origin/gh/masnesral/231/base 2025-08-14T21:25:14.0343621Z * [new branch] gh/masnesral/231/head -> origin/gh/masnesral/231/head 2025-08-14T21:25:14.0343749Z * [new branch] gh/masnesral/231/orig -> origin/gh/masnesral/231/orig 2025-08-14T21:25:14.0343884Z * [new branch] gh/masnesral/232/base -> origin/gh/masnesral/232/base 2025-08-14T21:25:14.0344014Z * [new branch] gh/masnesral/232/head -> origin/gh/masnesral/232/head 2025-08-14T21:25:14.0344201Z * [new branch] gh/masnesral/232/orig -> origin/gh/masnesral/232/orig 2025-08-14T21:25:14.0344339Z * [new branch] gh/masnesral/233/base -> origin/gh/masnesral/233/base 2025-08-14T21:25:14.0344480Z * [new branch] gh/masnesral/233/head -> origin/gh/masnesral/233/head 2025-08-14T21:25:14.0344620Z * [new branch] gh/masnesral/233/orig -> origin/gh/masnesral/233/orig 2025-08-14T21:25:14.0346110Z * [new branch] gh/masnesral/234/base -> origin/gh/masnesral/234/base 2025-08-14T21:25:14.0346265Z * [new branch] gh/masnesral/234/head -> origin/gh/masnesral/234/head 2025-08-14T21:25:14.0347259Z * [new branch] gh/masnesral/234/orig -> origin/gh/masnesral/234/orig 2025-08-14T21:25:14.0347987Z * [new branch] gh/masnesral/235/base -> origin/gh/masnesral/235/base 2025-08-14T21:25:14.0348686Z * [new branch] gh/masnesral/235/head -> origin/gh/masnesral/235/head 2025-08-14T21:25:14.0348978Z * [new branch] gh/masnesral/235/orig -> origin/gh/masnesral/235/orig 2025-08-14T21:25:14.0350265Z * [new branch] gh/masnesral/236/base -> origin/gh/masnesral/236/base 2025-08-14T21:25:14.0350595Z * [new branch] gh/masnesral/236/head -> origin/gh/masnesral/236/head 2025-08-14T21:25:14.0352998Z * [new branch] gh/masnesral/236/orig -> origin/gh/masnesral/236/orig 2025-08-14T21:25:14.0353189Z * [new branch] gh/masnesral/34/base -> origin/gh/masnesral/34/base 2025-08-14T21:25:14.0353369Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-08-14T21:25:14.0354187Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-08-14T21:25:14.0354979Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-08-14T21:25:14.0355698Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-08-14T21:25:14.0357139Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-08-14T21:25:14.0357430Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-08-14T21:25:14.0361214Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-08-14T21:25:14.0361539Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-08-14T21:25:14.0361720Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-08-14T21:25:14.0361869Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-08-14T21:25:14.0361995Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-08-14T21:25:14.0362257Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-08-14T21:25:14.0362532Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-08-14T21:25:14.0363749Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-08-14T21:25:14.0368157Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-08-14T21:25:14.0368366Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-08-14T21:25:14.0368544Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-08-14T21:25:14.0368727Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-08-14T21:25:14.0368907Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-08-14T21:25:14.0369085Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-08-14T21:25:14.0376856Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-08-14T21:25:14.0378962Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-08-14T21:25:14.0379336Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-08-14T21:25:14.0388190Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-08-14T21:25:14.0390106Z * [new branch] gh/mikaylagawarecki/313/base -> origin/gh/mikaylagawarecki/313/base 2025-08-14T21:25:14.0390280Z * [new branch] gh/mikaylagawarecki/313/head -> origin/gh/mikaylagawarecki/313/head 2025-08-14T21:25:14.0390454Z * [new branch] gh/mikaylagawarecki/313/orig -> origin/gh/mikaylagawarecki/313/orig 2025-08-14T21:25:14.0390616Z * [new branch] gh/mikaylagawarecki/317/base -> origin/gh/mikaylagawarecki/317/base 2025-08-14T21:25:14.0390807Z * [new branch] gh/mikaylagawarecki/317/head -> origin/gh/mikaylagawarecki/317/head 2025-08-14T21:25:14.0390986Z * [new branch] gh/mikaylagawarecki/317/orig -> origin/gh/mikaylagawarecki/317/orig 2025-08-14T21:25:14.0391159Z * [new branch] gh/mikaylagawarecki/318/base -> origin/gh/mikaylagawarecki/318/base 2025-08-14T21:25:14.0391317Z * [new branch] gh/mikaylagawarecki/318/head -> origin/gh/mikaylagawarecki/318/head 2025-08-14T21:25:14.0391473Z * [new branch] gh/mikaylagawarecki/318/orig -> origin/gh/mikaylagawarecki/318/orig 2025-08-14T21:25:14.0391638Z * [new branch] gh/mikaylagawarecki/319/base -> origin/gh/mikaylagawarecki/319/base 2025-08-14T21:25:14.0391793Z * [new branch] gh/mikaylagawarecki/319/head -> origin/gh/mikaylagawarecki/319/head 2025-08-14T21:25:14.0391949Z * [new branch] gh/mikaylagawarecki/319/orig -> origin/gh/mikaylagawarecki/319/orig 2025-08-14T21:25:14.0392111Z * [new branch] gh/mikaylagawarecki/320/base -> origin/gh/mikaylagawarecki/320/base 2025-08-14T21:25:14.0392277Z * [new branch] gh/mikaylagawarecki/320/head -> origin/gh/mikaylagawarecki/320/head 2025-08-14T21:25:14.0392577Z * [new branch] gh/mikaylagawarecki/320/orig -> origin/gh/mikaylagawarecki/320/orig 2025-08-14T21:25:14.0392738Z * [new branch] gh/mikaylagawarecki/321/base -> origin/gh/mikaylagawarecki/321/base 2025-08-14T21:25:14.0392929Z * [new branch] gh/mikaylagawarecki/321/head -> origin/gh/mikaylagawarecki/321/head 2025-08-14T21:25:14.0393116Z * [new branch] gh/mikaylagawarecki/321/orig -> origin/gh/mikaylagawarecki/321/orig 2025-08-14T21:25:14.0393293Z * [new branch] gh/mikaylagawarecki/322/base -> origin/gh/mikaylagawarecki/322/base 2025-08-14T21:25:14.0393472Z * [new branch] gh/mikaylagawarecki/322/head -> origin/gh/mikaylagawarecki/322/head 2025-08-14T21:25:14.0393648Z * [new branch] gh/mikaylagawarecki/322/orig -> origin/gh/mikaylagawarecki/322/orig 2025-08-14T21:25:14.0393823Z * [new branch] gh/mikaylagawarecki/323/base -> origin/gh/mikaylagawarecki/323/base 2025-08-14T21:25:14.0394009Z * [new branch] gh/mikaylagawarecki/323/head -> origin/gh/mikaylagawarecki/323/head 2025-08-14T21:25:14.0394183Z * [new branch] gh/mikaylagawarecki/323/orig -> origin/gh/mikaylagawarecki/323/orig 2025-08-14T21:25:14.0394362Z * [new branch] gh/mikaylagawarecki/324/base -> origin/gh/mikaylagawarecki/324/base 2025-08-14T21:25:14.0394542Z * [new branch] gh/mikaylagawarecki/324/head -> origin/gh/mikaylagawarecki/324/head 2025-08-14T21:25:14.0394715Z * [new branch] gh/mikaylagawarecki/324/orig -> origin/gh/mikaylagawarecki/324/orig 2025-08-14T21:25:14.0394895Z * [new branch] gh/mikaylagawarecki/325/base -> origin/gh/mikaylagawarecki/325/base 2025-08-14T21:25:14.0395071Z * [new branch] gh/mikaylagawarecki/325/head -> origin/gh/mikaylagawarecki/325/head 2025-08-14T21:25:14.0395322Z * [new branch] gh/mikaylagawarecki/325/orig -> origin/gh/mikaylagawarecki/325/orig 2025-08-14T21:25:14.0395503Z * [new branch] gh/mikaylagawarecki/326/base -> origin/gh/mikaylagawarecki/326/base 2025-08-14T21:25:14.0395709Z * [new branch] gh/mikaylagawarecki/326/head -> origin/gh/mikaylagawarecki/326/head 2025-08-14T21:25:14.0396713Z * [new branch] gh/mikaylagawarecki/326/orig -> origin/gh/mikaylagawarecki/326/orig 2025-08-14T21:25:14.0398180Z * [new branch] gh/mikaylagawarecki/327/base -> origin/gh/mikaylagawarecki/327/base 2025-08-14T21:25:14.0398365Z * [new branch] gh/mikaylagawarecki/327/head -> origin/gh/mikaylagawarecki/327/head 2025-08-14T21:25:14.0398969Z * [new branch] gh/mikaylagawarecki/327/orig -> origin/gh/mikaylagawarecki/327/orig 2025-08-14T21:25:14.0404476Z * [new branch] gh/mikaylagawarecki/328/base -> origin/gh/mikaylagawarecki/328/base 2025-08-14T21:25:14.0404707Z * [new branch] gh/mikaylagawarecki/328/head -> origin/gh/mikaylagawarecki/328/head 2025-08-14T21:25:14.0404895Z * [new branch] gh/mikaylagawarecki/328/orig -> origin/gh/mikaylagawarecki/328/orig 2025-08-14T21:25:14.0405091Z * [new branch] gh/mikaylagawarecki/329/base -> origin/gh/mikaylagawarecki/329/base 2025-08-14T21:25:14.0405271Z * [new branch] gh/mikaylagawarecki/329/head -> origin/gh/mikaylagawarecki/329/head 2025-08-14T21:25:14.0405451Z * [new branch] gh/mikaylagawarecki/329/orig -> origin/gh/mikaylagawarecki/329/orig 2025-08-14T21:25:14.0405628Z * [new branch] gh/mikaylagawarecki/330/base -> origin/gh/mikaylagawarecki/330/base 2025-08-14T21:25:14.0405810Z * [new branch] gh/mikaylagawarecki/330/head -> origin/gh/mikaylagawarecki/330/head 2025-08-14T21:25:14.0405999Z * [new branch] gh/mikaylagawarecki/330/orig -> origin/gh/mikaylagawarecki/330/orig 2025-08-14T21:25:14.0411909Z * [new branch] gh/mikaylagawarecki/331/base -> origin/gh/mikaylagawarecki/331/base 2025-08-14T21:25:14.0412138Z * [new branch] gh/mikaylagawarecki/331/head -> origin/gh/mikaylagawarecki/331/head 2025-08-14T21:25:14.0412505Z * [new branch] gh/mikaylagawarecki/331/orig -> origin/gh/mikaylagawarecki/331/orig 2025-08-14T21:25:14.0412687Z * [new branch] gh/mikaylagawarecki/332/base -> origin/gh/mikaylagawarecki/332/base 2025-08-14T21:25:14.0414254Z * [new branch] gh/mikaylagawarecki/332/head -> origin/gh/mikaylagawarecki/332/head 2025-08-14T21:25:14.0414574Z * [new branch] gh/mikaylagawarecki/332/orig -> origin/gh/mikaylagawarecki/332/orig 2025-08-14T21:25:14.0415232Z * [new branch] gh/mikaylagawarecki/333/base -> origin/gh/mikaylagawarecki/333/base 2025-08-14T21:25:14.0416419Z * [new branch] gh/mikaylagawarecki/333/head -> origin/gh/mikaylagawarecki/333/head 2025-08-14T21:25:14.0416772Z * [new branch] gh/mikaylagawarecki/333/orig -> origin/gh/mikaylagawarecki/333/orig 2025-08-14T21:25:14.0422034Z * [new branch] gh/mikaylagawarecki/334/base -> origin/gh/mikaylagawarecki/334/base 2025-08-14T21:25:14.0427641Z * [new branch] gh/mikaylagawarecki/334/head -> origin/gh/mikaylagawarecki/334/head 2025-08-14T21:25:14.0429555Z * [new branch] gh/mikaylagawarecki/334/orig -> origin/gh/mikaylagawarecki/334/orig 2025-08-14T21:25:14.0429722Z * [new branch] gh/mlazos/1/base -> origin/gh/mlazos/1/base 2025-08-14T21:25:14.0429855Z * [new branch] gh/mlazos/1/head -> origin/gh/mlazos/1/head 2025-08-14T21:25:14.0429990Z * [new branch] gh/mlazos/1/orig -> origin/gh/mlazos/1/orig 2025-08-14T21:25:14.0430134Z * [new branch] gh/mlazos/10/base -> origin/gh/mlazos/10/base 2025-08-14T21:25:14.0430278Z * [new branch] gh/mlazos/10/head -> origin/gh/mlazos/10/head 2025-08-14T21:25:14.0430622Z * [new branch] gh/mlazos/10/orig -> origin/gh/mlazos/10/orig 2025-08-14T21:25:14.0430768Z * [new branch] gh/mlazos/11/base -> origin/gh/mlazos/11/base 2025-08-14T21:25:14.0430909Z * [new branch] gh/mlazos/11/head -> origin/gh/mlazos/11/head 2025-08-14T21:25:14.0431040Z * [new branch] gh/mlazos/11/orig -> origin/gh/mlazos/11/orig 2025-08-14T21:25:14.0431169Z * [new branch] gh/mlazos/12/base -> origin/gh/mlazos/12/base 2025-08-14T21:25:14.0431316Z * [new branch] gh/mlazos/12/head -> origin/gh/mlazos/12/head 2025-08-14T21:25:14.0431444Z * [new branch] gh/mlazos/12/orig -> origin/gh/mlazos/12/orig 2025-08-14T21:25:14.0431577Z * [new branch] gh/mlazos/13/base -> origin/gh/mlazos/13/base 2025-08-14T21:25:14.0431707Z * [new branch] gh/mlazos/13/head -> origin/gh/mlazos/13/head 2025-08-14T21:25:14.0431837Z * [new branch] gh/mlazos/13/orig -> origin/gh/mlazos/13/orig 2025-08-14T21:25:14.0431985Z * [new branch] gh/mlazos/2/base -> origin/gh/mlazos/2/base 2025-08-14T21:25:14.0432119Z * [new branch] gh/mlazos/2/head -> origin/gh/mlazos/2/head 2025-08-14T21:25:14.0433247Z * [new branch] gh/mlazos/2/orig -> origin/gh/mlazos/2/orig 2025-08-14T21:25:14.0433853Z * [new branch] gh/mlazos/3/base -> origin/gh/mlazos/3/base 2025-08-14T21:25:14.0434817Z * [new branch] gh/mlazos/3/head -> origin/gh/mlazos/3/head 2025-08-14T21:25:14.0435224Z * [new branch] gh/mlazos/3/orig -> origin/gh/mlazos/3/orig 2025-08-14T21:25:14.0436600Z * [new branch] gh/mlazos/4/base -> origin/gh/mlazos/4/base 2025-08-14T21:25:14.0436772Z * [new branch] gh/mlazos/4/head -> origin/gh/mlazos/4/head 2025-08-14T21:25:14.0441152Z * [new branch] gh/mlazos/4/orig -> origin/gh/mlazos/4/orig 2025-08-14T21:25:14.0441498Z * [new branch] gh/mlazos/5/base -> origin/gh/mlazos/5/base 2025-08-14T21:25:14.0441636Z * [new branch] gh/mlazos/5/head -> origin/gh/mlazos/5/head 2025-08-14T21:25:14.0441768Z * [new branch] gh/mlazos/5/orig -> origin/gh/mlazos/5/orig 2025-08-14T21:25:14.0441909Z * [new branch] gh/mlazos/6/base -> origin/gh/mlazos/6/base 2025-08-14T21:25:14.0442225Z * [new branch] gh/mlazos/6/head -> origin/gh/mlazos/6/head 2025-08-14T21:25:14.0442384Z * [new branch] gh/mlazos/6/orig -> origin/gh/mlazos/6/orig 2025-08-14T21:25:14.0444233Z * [new branch] gh/mlazos/7/base -> origin/gh/mlazos/7/base 2025-08-14T21:25:14.0444586Z * [new branch] gh/mlazos/7/head -> origin/gh/mlazos/7/head 2025-08-14T21:25:14.0445127Z * [new branch] gh/mlazos/7/orig -> origin/gh/mlazos/7/orig 2025-08-14T21:25:14.0447237Z * [new branch] gh/mlazos/8/base -> origin/gh/mlazos/8/base 2025-08-14T21:25:14.0447601Z * [new branch] gh/mlazos/8/head -> origin/gh/mlazos/8/head 2025-08-14T21:25:14.0447831Z * [new branch] gh/mlazos/8/orig -> origin/gh/mlazos/8/orig 2025-08-14T21:25:14.0447979Z * [new branch] gh/mlazos/9/base -> origin/gh/mlazos/9/base 2025-08-14T21:25:14.0449445Z * [new branch] gh/mlazos/9/head -> origin/gh/mlazos/9/head 2025-08-14T21:25:14.0449779Z * [new branch] gh/mlazos/9/orig -> origin/gh/mlazos/9/orig 2025-08-14T21:25:14.0452070Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-08-14T21:25:14.0452237Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-08-14T21:25:14.0452724Z * [new branch] gh/muchulee8/62/base -> origin/gh/muchulee8/62/base 2025-08-14T21:25:14.0453907Z * [new branch] gh/muchulee8/62/head -> origin/gh/muchulee8/62/head 2025-08-14T21:25:14.0454073Z * [new branch] gh/muchulee8/62/orig -> origin/gh/muchulee8/62/orig 2025-08-14T21:25:14.0458698Z * [new branch] gh/muchulee8/63/base -> origin/gh/muchulee8/63/base 2025-08-14T21:25:14.0458875Z * [new branch] gh/muchulee8/63/head -> origin/gh/muchulee8/63/head 2025-08-14T21:25:14.0459019Z * [new branch] gh/muchulee8/63/orig -> origin/gh/muchulee8/63/orig 2025-08-14T21:25:14.0459150Z * [new branch] gh/muchulee8/64/base -> origin/gh/muchulee8/64/base 2025-08-14T21:25:14.0459294Z * [new branch] gh/muchulee8/64/head -> origin/gh/muchulee8/64/head 2025-08-14T21:25:14.0459431Z * [new branch] gh/muchulee8/64/orig -> origin/gh/muchulee8/64/orig 2025-08-14T21:25:14.0459881Z * [new branch] gh/muchulee8/65/base -> origin/gh/muchulee8/65/base 2025-08-14T21:25:14.0460587Z * [new branch] gh/muchulee8/65/head -> origin/gh/muchulee8/65/head 2025-08-14T21:25:14.0461458Z * [new branch] gh/muchulee8/65/orig -> origin/gh/muchulee8/65/orig 2025-08-14T21:25:14.0465554Z * [new branch] gh/oulgen/35/base -> origin/gh/oulgen/35/base 2025-08-14T21:25:14.0465716Z * [new branch] gh/oulgen/35/head -> origin/gh/oulgen/35/head 2025-08-14T21:25:14.0465838Z * [new branch] gh/oulgen/35/orig -> origin/gh/oulgen/35/orig 2025-08-14T21:25:14.0465965Z * [new branch] gh/oulgen/44/base -> origin/gh/oulgen/44/base 2025-08-14T21:25:14.0466087Z * [new branch] gh/oulgen/44/head -> origin/gh/oulgen/44/head 2025-08-14T21:25:14.0466248Z * [new branch] gh/oulgen/44/orig -> origin/gh/oulgen/44/orig 2025-08-14T21:25:14.0470502Z * [new branch] gh/oulgen/45/base -> origin/gh/oulgen/45/base 2025-08-14T21:25:14.0470833Z * [new branch] gh/oulgen/45/head -> origin/gh/oulgen/45/head 2025-08-14T21:25:14.0470972Z * [new branch] gh/oulgen/45/orig -> origin/gh/oulgen/45/orig 2025-08-14T21:25:14.0471103Z * [new branch] gh/oulgen/46/base -> origin/gh/oulgen/46/base 2025-08-14T21:25:14.0471244Z * [new branch] gh/oulgen/46/head -> origin/gh/oulgen/46/head 2025-08-14T21:25:14.0471374Z * [new branch] gh/oulgen/46/orig -> origin/gh/oulgen/46/orig 2025-08-14T21:25:14.0471538Z * [new branch] gh/oulgen/47/base -> origin/gh/oulgen/47/base 2025-08-14T21:25:14.0472497Z * [new branch] gh/oulgen/47/head -> origin/gh/oulgen/47/head 2025-08-14T21:25:14.0472657Z * [new branch] gh/oulgen/47/orig -> origin/gh/oulgen/47/orig 2025-08-14T21:25:14.0474418Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-08-14T21:25:14.0475684Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-08-14T21:25:14.0476002Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-08-14T21:25:14.0480529Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-08-14T21:25:14.0480781Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-08-14T21:25:14.0480914Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-08-14T21:25:14.0481052Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-08-14T21:25:14.0481205Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-08-14T21:25:14.0485268Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-08-14T21:25:14.0485614Z * [new branch] gh/qqaatw/29/base -> origin/gh/qqaatw/29/base 2025-08-14T21:25:14.0485760Z * [new branch] gh/qqaatw/29/head -> origin/gh/qqaatw/29/head 2025-08-14T21:25:14.0485902Z * [new branch] gh/qqaatw/29/orig -> origin/gh/qqaatw/29/orig 2025-08-14T21:25:14.0486125Z * [new branch] gh/raymo/cleanup-dynamo-logging -> origin/gh/raymo/cleanup-dynamo-logging 2025-08-14T21:25:14.0486283Z * [new branch] gh/raymo/refresh-script -> origin/gh/raymo/refresh-script 2025-08-14T21:25:14.0488937Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-08-14T21:25:14.0489165Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-08-14T21:25:14.0489289Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-08-14T21:25:14.0489409Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-08-14T21:25:14.0489566Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-08-14T21:25:14.0492787Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-08-14T21:25:14.0493048Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-08-14T21:25:14.0493189Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-08-14T21:25:14.0493316Z * [new branch] gh/rec/156/base -> origin/gh/rec/156/base 2025-08-14T21:25:14.0493443Z * [new branch] gh/rec/156/head -> origin/gh/rec/156/head 2025-08-14T21:25:14.0493583Z * [new branch] gh/rec/156/orig -> origin/gh/rec/156/orig 2025-08-14T21:25:14.0497530Z * [new branch] gh/rec/158/base -> origin/gh/rec/158/base 2025-08-14T21:25:14.0497779Z * [new branch] gh/rec/158/head -> origin/gh/rec/158/head 2025-08-14T21:25:14.0497918Z * [new branch] gh/rec/158/orig -> origin/gh/rec/158/orig 2025-08-14T21:25:14.0498168Z * [new branch] gh/rec/159/base -> origin/gh/rec/159/base 2025-08-14T21:25:14.0498295Z * [new branch] gh/rec/159/head -> origin/gh/rec/159/head 2025-08-14T21:25:14.0498415Z * [new branch] gh/rec/160/base -> origin/gh/rec/160/base 2025-08-14T21:25:14.0498538Z * [new branch] gh/rec/160/head -> origin/gh/rec/160/head 2025-08-14T21:25:14.0501264Z * [new branch] gh/rec/160/orig -> origin/gh/rec/160/orig 2025-08-14T21:25:14.0501610Z * [new branch] gh/rec/161/base -> origin/gh/rec/161/base 2025-08-14T21:25:14.0501830Z * [new branch] gh/rec/161/head -> origin/gh/rec/161/head 2025-08-14T21:25:14.0501950Z * [new branch] gh/rec/161/orig -> origin/gh/rec/161/orig 2025-08-14T21:25:14.0502080Z * [new branch] gh/rec/162/base -> origin/gh/rec/162/base 2025-08-14T21:25:14.0502206Z * [new branch] gh/rec/162/head -> origin/gh/rec/162/head 2025-08-14T21:25:14.0502563Z * [new branch] gh/rec/162/orig -> origin/gh/rec/162/orig 2025-08-14T21:25:14.0503378Z * [new branch] gh/rec/163/base -> origin/gh/rec/163/base 2025-08-14T21:25:14.0503796Z * [new branch] gh/rec/163/head -> origin/gh/rec/163/head 2025-08-14T21:25:14.0504862Z * [new branch] gh/rec/163/orig -> origin/gh/rec/163/orig 2025-08-14T21:25:14.0506115Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-08-14T21:25:14.0506406Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-08-14T21:25:14.0507365Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-08-14T21:25:14.0508594Z * [new branch] gh/robert-hardwick/1/base -> origin/gh/robert-hardwick/1/base 2025-08-14T21:25:14.0508995Z * [new branch] gh/robert-hardwick/1/head -> origin/gh/robert-hardwick/1/head 2025-08-14T21:25:14.0510049Z * [new branch] gh/robert-hardwick/1/orig -> origin/gh/robert-hardwick/1/orig 2025-08-14T21:25:14.0512796Z * [new branch] gh/robert-hardwick/2/base -> origin/gh/robert-hardwick/2/base 2025-08-14T21:25:14.0513053Z * [new branch] gh/robert-hardwick/2/head -> origin/gh/robert-hardwick/2/head 2025-08-14T21:25:14.0513221Z * [new branch] gh/robert-hardwick/2/orig -> origin/gh/robert-hardwick/2/orig 2025-08-14T21:25:14.0513377Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-08-14T21:25:14.0513530Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-08-14T21:25:14.0514079Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-08-14T21:25:14.0515248Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-08-14T21:25:14.0515503Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-08-14T21:25:14.0516826Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-08-14T21:25:14.0517799Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-08-14T21:25:14.0518041Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-08-14T21:25:14.0519462Z * [new branch] gh/rtimpe/10/base -> origin/gh/rtimpe/10/base 2025-08-14T21:25:14.0519730Z * [new branch] gh/rtimpe/10/head -> origin/gh/rtimpe/10/head 2025-08-14T21:25:14.0520703Z * [new branch] gh/rtimpe/10/orig -> origin/gh/rtimpe/10/orig 2025-08-14T21:25:14.0523913Z * [new branch] gh/rtimpe/11/base -> origin/gh/rtimpe/11/base 2025-08-14T21:25:14.0524099Z * [new branch] gh/rtimpe/11/head -> origin/gh/rtimpe/11/head 2025-08-14T21:25:14.0524433Z * [new branch] gh/rtimpe/11/orig -> origin/gh/rtimpe/11/orig 2025-08-14T21:25:14.0524557Z * [new branch] gh/rtimpe/12/base -> origin/gh/rtimpe/12/base 2025-08-14T21:25:14.0524681Z * [new branch] gh/rtimpe/12/head -> origin/gh/rtimpe/12/head 2025-08-14T21:25:14.0524849Z * [new branch] gh/rtimpe/12/orig -> origin/gh/rtimpe/12/orig 2025-08-14T21:25:14.0526222Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-08-14T21:25:14.0526561Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-08-14T21:25:14.0527080Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-08-14T21:25:14.0528868Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-08-14T21:25:14.0529208Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-08-14T21:25:14.0529378Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-08-14T21:25:14.0532408Z * [new branch] gh/rtimpe/5/base -> origin/gh/rtimpe/5/base 2025-08-14T21:25:14.0532750Z * [new branch] gh/rtimpe/5/head -> origin/gh/rtimpe/5/head 2025-08-14T21:25:14.0532905Z * [new branch] gh/rtimpe/5/orig -> origin/gh/rtimpe/5/orig 2025-08-14T21:25:14.0533078Z * [new branch] gh/rtimpe/6/base -> origin/gh/rtimpe/6/base 2025-08-14T21:25:14.0533435Z * [new branch] gh/rtimpe/6/head -> origin/gh/rtimpe/6/head 2025-08-14T21:25:14.0534008Z * [new branch] gh/rtimpe/6/orig -> origin/gh/rtimpe/6/orig 2025-08-14T21:25:14.0534387Z * [new branch] gh/rtimpe/7/base -> origin/gh/rtimpe/7/base 2025-08-14T21:25:14.0535058Z * [new branch] gh/rtimpe/7/head -> origin/gh/rtimpe/7/head 2025-08-14T21:25:14.0537597Z * [new branch] gh/rtimpe/7/orig -> origin/gh/rtimpe/7/orig 2025-08-14T21:25:14.0537932Z * [new branch] gh/rtimpe/8/base -> origin/gh/rtimpe/8/base 2025-08-14T21:25:14.0538123Z * [new branch] gh/rtimpe/8/head -> origin/gh/rtimpe/8/head 2025-08-14T21:25:14.0538249Z * [new branch] gh/rtimpe/8/orig -> origin/gh/rtimpe/8/orig 2025-08-14T21:25:14.0538567Z * [new branch] gh/rtimpe/9/base -> origin/gh/rtimpe/9/base 2025-08-14T21:25:14.0539596Z * [new branch] gh/rtimpe/9/head -> origin/gh/rtimpe/9/head 2025-08-14T21:25:14.0540192Z * [new branch] gh/rtimpe/9/orig -> origin/gh/rtimpe/9/orig 2025-08-14T21:25:14.0542789Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-08-14T21:25:14.0543141Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-08-14T21:25:14.0543383Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-08-14T21:25:14.0543548Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-08-14T21:25:14.0544242Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-08-14T21:25:14.0544779Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-08-14T21:25:14.0549468Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-08-14T21:25:14.0549811Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-08-14T21:25:14.0549995Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-08-14T21:25:14.0550183Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-08-14T21:25:14.0550596Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-08-14T21:25:14.0551260Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-08-14T21:25:14.0551606Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-08-14T21:25:14.0552086Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-08-14T21:25:14.0552377Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-08-14T21:25:14.0553245Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-08-14T21:25:14.0553900Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-08-14T21:25:14.0554583Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-08-14T21:25:14.0555841Z * [new branch] gh/sarckk/2/base -> origin/gh/sarckk/2/base 2025-08-14T21:25:14.0556327Z * [new branch] gh/sarckk/2/head -> origin/gh/sarckk/2/head 2025-08-14T21:25:14.0557162Z * [new branch] gh/sarckk/2/orig -> origin/gh/sarckk/2/orig 2025-08-14T21:25:14.0558483Z * [new branch] gh/seemethere/23/head -> origin/gh/seemethere/23/head 2025-08-14T21:25:14.0559608Z * [new branch] gh/seemethere/24/base -> origin/gh/seemethere/24/base 2025-08-14T21:25:14.0559774Z * [new branch] gh/seemethere/24/head -> origin/gh/seemethere/24/head 2025-08-14T21:25:14.0562119Z * [new branch] gh/seemethere/24/orig -> origin/gh/seemethere/24/orig 2025-08-14T21:25:14.0562335Z * [new branch] gh/seemethere/30/base -> origin/gh/seemethere/30/base 2025-08-14T21:25:14.0562684Z * [new branch] gh/seemethere/30/head -> origin/gh/seemethere/30/head 2025-08-14T21:25:14.0563880Z * [new branch] gh/seemethere/30/orig -> origin/gh/seemethere/30/orig 2025-08-14T21:25:14.0564049Z * [new branch] gh/seemethere/32/base -> origin/gh/seemethere/32/base 2025-08-14T21:25:14.0564201Z * [new branch] gh/seemethere/32/head -> origin/gh/seemethere/32/head 2025-08-14T21:25:14.0566777Z * [new branch] gh/seemethere/32/orig -> origin/gh/seemethere/32/orig 2025-08-14T21:25:14.0566967Z * [new branch] gh/seemethere/33/base -> origin/gh/seemethere/33/base 2025-08-14T21:25:14.0567112Z * [new branch] gh/seemethere/33/head -> origin/gh/seemethere/33/head 2025-08-14T21:25:14.0572225Z * [new branch] gh/seemethere/33/orig -> origin/gh/seemethere/33/orig 2025-08-14T21:25:14.0572417Z * [new branch] gh/seemethere/34/base -> origin/gh/seemethere/34/base 2025-08-14T21:25:14.0572588Z * [new branch] gh/seemethere/34/head -> origin/gh/seemethere/34/head 2025-08-14T21:25:14.0572754Z * [new branch] gh/seemethere/34/orig -> origin/gh/seemethere/34/orig 2025-08-14T21:25:14.0572887Z * [new branch] gh/seemethere/35/base -> origin/gh/seemethere/35/base 2025-08-14T21:25:14.0577071Z * [new branch] gh/seemethere/35/head -> origin/gh/seemethere/35/head 2025-08-14T21:25:14.0577263Z * [new branch] gh/seemethere/35/orig -> origin/gh/seemethere/35/orig 2025-08-14T21:25:14.0577416Z * [new branch] gh/seemethere/37/base -> origin/gh/seemethere/37/base 2025-08-14T21:25:14.0577569Z * [new branch] gh/seemethere/37/head -> origin/gh/seemethere/37/head 2025-08-14T21:25:14.0577720Z * [new branch] gh/seemethere/37/orig -> origin/gh/seemethere/37/orig 2025-08-14T21:25:14.0577855Z * [new branch] gh/seemethere/39/base -> origin/gh/seemethere/39/base 2025-08-14T21:25:14.0578017Z * [new branch] gh/seemethere/39/head -> origin/gh/seemethere/39/head 2025-08-14T21:25:14.0578331Z * [new branch] gh/seemethere/39/orig -> origin/gh/seemethere/39/orig 2025-08-14T21:25:14.0579290Z * [new branch] gh/seemethere/40/base -> origin/gh/seemethere/40/base 2025-08-14T21:25:14.0579535Z * [new branch] gh/seemethere/40/head -> origin/gh/seemethere/40/head 2025-08-14T21:25:14.0579680Z * [new branch] gh/seemethere/40/orig -> origin/gh/seemethere/40/orig 2025-08-14T21:25:14.0579829Z * [new branch] gh/seemethere/41/base -> origin/gh/seemethere/41/base 2025-08-14T21:25:14.0579980Z * [new branch] gh/seemethere/41/head -> origin/gh/seemethere/41/head 2025-08-14T21:25:14.0580120Z * [new branch] gh/seemethere/41/orig -> origin/gh/seemethere/41/orig 2025-08-14T21:25:14.0584271Z * [new branch] gh/seemethere/42/base -> origin/gh/seemethere/42/base 2025-08-14T21:25:14.0584925Z * [new branch] gh/seemethere/42/head -> origin/gh/seemethere/42/head 2025-08-14T21:25:14.0585271Z * [new branch] gh/seemethere/42/orig -> origin/gh/seemethere/42/orig 2025-08-14T21:25:14.0585475Z * [new branch] gh/seemethere/43/base -> origin/gh/seemethere/43/base 2025-08-14T21:25:14.0585773Z * [new branch] gh/seemethere/43/head -> origin/gh/seemethere/43/head 2025-08-14T21:25:14.0585939Z * [new branch] gh/seemethere/43/orig -> origin/gh/seemethere/43/orig 2025-08-14T21:25:14.0586175Z * [new branch] gh/seemethere/44/base -> origin/gh/seemethere/44/base 2025-08-14T21:25:14.0586339Z * [new branch] gh/seemethere/44/head -> origin/gh/seemethere/44/head 2025-08-14T21:25:14.0592164Z * [new branch] gh/seemethere/44/orig -> origin/gh/seemethere/44/orig 2025-08-14T21:25:14.0592539Z * [new branch] gh/seemethere/45/base -> origin/gh/seemethere/45/base 2025-08-14T21:25:14.0592705Z * [new branch] gh/seemethere/45/head -> origin/gh/seemethere/45/head 2025-08-14T21:25:14.0592856Z * [new branch] gh/seemethere/45/orig -> origin/gh/seemethere/45/orig 2025-08-14T21:25:14.0593013Z * [new branch] gh/seemethere/46/base -> origin/gh/seemethere/46/base 2025-08-14T21:25:14.0593162Z * [new branch] gh/seemethere/46/head -> origin/gh/seemethere/46/head 2025-08-14T21:25:14.0593318Z * [new branch] gh/seemethere/46/orig -> origin/gh/seemethere/46/orig 2025-08-14T21:25:14.0593462Z * [new branch] gh/seemethere/47/base -> origin/gh/seemethere/47/base 2025-08-14T21:25:14.0593603Z * [new branch] gh/seemethere/47/head -> origin/gh/seemethere/47/head 2025-08-14T21:25:14.0593756Z * [new branch] gh/seemethere/47/orig -> origin/gh/seemethere/47/orig 2025-08-14T21:25:14.0593903Z * [new branch] gh/seemethere/48/base -> origin/gh/seemethere/48/base 2025-08-14T21:25:14.0594056Z * [new branch] gh/seemethere/48/head -> origin/gh/seemethere/48/head 2025-08-14T21:25:14.0594206Z * [new branch] gh/seemethere/48/orig -> origin/gh/seemethere/48/orig 2025-08-14T21:25:14.0595479Z * [new branch] gh/seemethere/49/base -> origin/gh/seemethere/49/base 2025-08-14T21:25:14.0595780Z * [new branch] gh/seemethere/49/head -> origin/gh/seemethere/49/head 2025-08-14T21:25:14.0596876Z * [new branch] gh/seemethere/49/orig -> origin/gh/seemethere/49/orig 2025-08-14T21:25:14.0601511Z * [new branch] gh/seemethere/50/base -> origin/gh/seemethere/50/base 2025-08-14T21:25:14.0601690Z * [new branch] gh/seemethere/50/head -> origin/gh/seemethere/50/head 2025-08-14T21:25:14.0601845Z * [new branch] gh/seemethere/50/orig -> origin/gh/seemethere/50/orig 2025-08-14T21:25:14.0602001Z * [new branch] gh/seemethere/51/base -> origin/gh/seemethere/51/base 2025-08-14T21:25:14.0602293Z * [new branch] gh/seemethere/51/head -> origin/gh/seemethere/51/head 2025-08-14T21:25:14.0602446Z * [new branch] gh/seemethere/51/orig -> origin/gh/seemethere/51/orig 2025-08-14T21:25:14.0606528Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-08-14T21:25:14.0606805Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-08-14T21:25:14.0612903Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-08-14T21:25:14.0617709Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-08-14T21:25:14.0619879Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-08-14T21:25:14.0620069Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-08-14T21:25:14.0620216Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-08-14T21:25:14.0620365Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-08-14T21:25:14.0620500Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-08-14T21:25:14.0620642Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-08-14T21:25:14.0620773Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-08-14T21:25:14.0620906Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-08-14T21:25:14.0621048Z * [new branch] gh/seemethere/56/base -> origin/gh/seemethere/56/base 2025-08-14T21:25:14.0621180Z * [new branch] gh/seemethere/56/head -> origin/gh/seemethere/56/head 2025-08-14T21:25:14.0621489Z * [new branch] gh/seemethere/56/orig -> origin/gh/seemethere/56/orig 2025-08-14T21:25:14.0621638Z * [new branch] gh/seemethere/57/base -> origin/gh/seemethere/57/base 2025-08-14T21:25:14.0621774Z * [new branch] gh/seemethere/57/head -> origin/gh/seemethere/57/head 2025-08-14T21:25:14.0621917Z * [new branch] gh/seemethere/57/orig -> origin/gh/seemethere/57/orig 2025-08-14T21:25:14.0622052Z * [new branch] gh/seemethere/58/base -> origin/gh/seemethere/58/base 2025-08-14T21:25:14.0622194Z * [new branch] gh/seemethere/58/head -> origin/gh/seemethere/58/head 2025-08-14T21:25:14.0622326Z * [new branch] gh/seemethere/58/orig -> origin/gh/seemethere/58/orig 2025-08-14T21:25:14.0622458Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-08-14T21:25:14.0622598Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-08-14T21:25:14.0622734Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-08-14T21:25:14.0622895Z * [new branch] gh/seemethere/7/head -> origin/gh/seemethere/7/head 2025-08-14T21:25:14.0627765Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-08-14T21:25:14.0631353Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-08-14T21:25:14.0631620Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-08-14T21:25:14.0631800Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-08-14T21:25:14.0631986Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-08-14T21:25:14.0632156Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-08-14T21:25:14.0632324Z * [new branch] gh/shunting314/211/base -> origin/gh/shunting314/211/base 2025-08-14T21:25:14.0632676Z * [new branch] gh/shunting314/211/head -> origin/gh/shunting314/211/head 2025-08-14T21:25:14.0632822Z * [new branch] gh/shunting314/211/orig -> origin/gh/shunting314/211/orig 2025-08-14T21:25:14.0632983Z * [new branch] gh/shunting314/212/base -> origin/gh/shunting314/212/base 2025-08-14T21:25:14.0633133Z * [new branch] gh/shunting314/212/head -> origin/gh/shunting314/212/head 2025-08-14T21:25:14.0633281Z * [new branch] gh/shunting314/212/orig -> origin/gh/shunting314/212/orig 2025-08-14T21:25:14.0633439Z * [new branch] gh/shunting314/213/base -> origin/gh/shunting314/213/base 2025-08-14T21:25:14.0633595Z * [new branch] gh/shunting314/213/head -> origin/gh/shunting314/213/head 2025-08-14T21:25:14.0633761Z * [new branch] gh/shunting314/213/orig -> origin/gh/shunting314/213/orig 2025-08-14T21:25:14.0635028Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-08-14T21:25:14.0635192Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-08-14T21:25:14.0635345Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-08-14T21:25:14.0635490Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-08-14T21:25:14.0635645Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-08-14T21:25:14.0636675Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-08-14T21:25:14.0637129Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-08-14T21:25:14.0644486Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-08-14T21:25:14.0648716Z * [new branch] gh/sinhaanhsul/1/base -> origin/gh/sinhaanhsul/1/base 2025-08-14T21:25:14.0653393Z * [new branch] gh/sinhaanhsul/1/head -> origin/gh/sinhaanhsul/1/head 2025-08-14T21:25:14.0657216Z * [new branch] gh/skarjala/11/base -> origin/gh/skarjala/11/base 2025-08-14T21:25:14.0662299Z * [new branch] gh/skarjala/11/head -> origin/gh/skarjala/11/head 2025-08-14T21:25:14.0662497Z * [new branch] gh/skarjala/11/orig -> origin/gh/skarjala/11/orig 2025-08-14T21:25:14.0662630Z * [new branch] gh/skarjala/13/base -> origin/gh/skarjala/13/base 2025-08-14T21:25:14.0662759Z * [new branch] gh/skarjala/13/head -> origin/gh/skarjala/13/head 2025-08-14T21:25:14.0662895Z * [new branch] gh/skarjala/13/orig -> origin/gh/skarjala/13/orig 2025-08-14T21:25:14.0663022Z * [new branch] gh/skarjala/14/base -> origin/gh/skarjala/14/base 2025-08-14T21:25:14.0663169Z * [new branch] gh/skarjala/14/head -> origin/gh/skarjala/14/head 2025-08-14T21:25:14.0663303Z * [new branch] gh/skarjala/14/orig -> origin/gh/skarjala/14/orig 2025-08-14T21:25:14.0663427Z * [new branch] gh/skarjala/15/base -> origin/gh/skarjala/15/base 2025-08-14T21:25:14.0663556Z * [new branch] gh/skarjala/15/head -> origin/gh/skarjala/15/head 2025-08-14T21:25:14.0663679Z * [new branch] gh/skarjala/15/orig -> origin/gh/skarjala/15/orig 2025-08-14T21:25:14.0663813Z * [new branch] gh/skarjala/16/base -> origin/gh/skarjala/16/base 2025-08-14T21:25:14.0663936Z * [new branch] gh/skarjala/16/head -> origin/gh/skarjala/16/head 2025-08-14T21:25:14.0664075Z * [new branch] gh/skarjala/16/orig -> origin/gh/skarjala/16/orig 2025-08-14T21:25:14.0664275Z * [new branch] gh/skarjala/17/base -> origin/gh/skarjala/17/base 2025-08-14T21:25:14.0664406Z * [new branch] gh/skarjala/17/head -> origin/gh/skarjala/17/head 2025-08-14T21:25:14.0664701Z * [new branch] gh/skarjala/17/orig -> origin/gh/skarjala/17/orig 2025-08-14T21:25:14.0664825Z * [new branch] gh/skarjala/18/base -> origin/gh/skarjala/18/base 2025-08-14T21:25:14.0664949Z * [new branch] gh/skarjala/18/head -> origin/gh/skarjala/18/head 2025-08-14T21:25:14.0665080Z * [new branch] gh/skarjala/18/orig -> origin/gh/skarjala/18/orig 2025-08-14T21:25:14.0665204Z * [new branch] gh/skarjala/19/base -> origin/gh/skarjala/19/base 2025-08-14T21:25:14.0665330Z * [new branch] gh/skarjala/19/head -> origin/gh/skarjala/19/head 2025-08-14T21:25:14.0665463Z * [new branch] gh/skarjala/19/orig -> origin/gh/skarjala/19/orig 2025-08-14T21:25:14.0665606Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-08-14T21:25:14.0665750Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-08-14T21:25:14.0665888Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-08-14T21:25:14.0666025Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-08-14T21:25:14.0666161Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-08-14T21:25:14.0666288Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-08-14T21:25:14.0666430Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-08-14T21:25:14.0666557Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-08-14T21:25:14.0666683Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-08-14T21:25:14.0666815Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-08-14T21:25:14.0666981Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-08-14T21:25:14.0667316Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-08-14T21:25:14.0667469Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-08-14T21:25:14.0670808Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-08-14T21:25:14.0670999Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-08-14T21:25:14.0671145Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-08-14T21:25:14.0671301Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-08-14T21:25:14.0672009Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-08-14T21:25:14.0672544Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-08-14T21:25:14.0673186Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-08-14T21:25:14.0674134Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-08-14T21:25:14.0674743Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-08-14T21:25:14.0675497Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-08-14T21:25:14.0675966Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-08-14T21:25:14.0677430Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-08-14T21:25:14.0677718Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-08-14T21:25:14.0679108Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-08-14T21:25:14.0679388Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-08-14T21:25:14.0680061Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-08-14T21:25:14.0680918Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-08-14T21:25:14.0681892Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-08-14T21:25:14.0682191Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-08-14T21:25:14.0683193Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-08-14T21:25:14.0684104Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-08-14T21:25:14.0684696Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-08-14T21:25:14.0685432Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-08-14T21:25:14.0687149Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-08-14T21:25:14.0687428Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-08-14T21:25:14.0688310Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-08-14T21:25:14.0688889Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-08-14T21:25:14.0689535Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-08-14T21:25:14.0690063Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-08-14T21:25:14.0691196Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-08-14T21:25:14.0691593Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-08-14T21:25:14.0692637Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-08-14T21:25:14.0693192Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-08-14T21:25:14.0694174Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-08-14T21:25:14.0694583Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-08-14T21:25:14.0696262Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-08-14T21:25:14.0696663Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-08-14T21:25:14.0697555Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-08-14T21:25:14.0698840Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-08-14T21:25:14.0699100Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-08-14T21:25:14.0700184Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-08-14T21:25:14.0703057Z * [new branch] gh/soulitzer/362/base -> origin/gh/soulitzer/362/base 2025-08-14T21:25:14.0703243Z * [new branch] gh/soulitzer/362/head -> origin/gh/soulitzer/362/head 2025-08-14T21:25:14.0703391Z * [new branch] gh/soulitzer/362/orig -> origin/gh/soulitzer/362/orig 2025-08-14T21:25:14.0703547Z * [new branch] gh/soulitzer/372/base -> origin/gh/soulitzer/372/base 2025-08-14T21:25:14.0703753Z * [new branch] gh/soulitzer/372/head -> origin/gh/soulitzer/372/head 2025-08-14T21:25:14.0704313Z * [new branch] gh/soulitzer/372/orig -> origin/gh/soulitzer/372/orig 2025-08-14T21:25:14.0705755Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-08-14T21:25:14.0706314Z * [new branch] gh/swolchok/758/base -> origin/gh/swolchok/758/base 2025-08-14T21:25:14.0706930Z * [new branch] gh/swolchok/758/head -> origin/gh/swolchok/758/head 2025-08-14T21:25:14.0707590Z * [new branch] gh/swolchok/758/orig -> origin/gh/swolchok/758/orig 2025-08-14T21:25:14.0709071Z * [new branch] gh/swolchok/767/base -> origin/gh/swolchok/767/base 2025-08-14T21:25:14.0709674Z * [new branch] gh/swolchok/767/head -> origin/gh/swolchok/767/head 2025-08-14T21:25:14.0710867Z * [new branch] gh/swolchok/767/orig -> origin/gh/swolchok/767/orig 2025-08-14T21:25:14.0711474Z * [new branch] gh/swolchok/768/base -> origin/gh/swolchok/768/base 2025-08-14T21:25:14.0712364Z * [new branch] gh/swolchok/768/head -> origin/gh/swolchok/768/head 2025-08-14T21:25:14.0712955Z * [new branch] gh/swolchok/768/orig -> origin/gh/swolchok/768/orig 2025-08-14T21:25:14.0714257Z * [new branch] gh/swolchok/769/base -> origin/gh/swolchok/769/base 2025-08-14T21:25:14.0715244Z * [new branch] gh/swolchok/769/head -> origin/gh/swolchok/769/head 2025-08-14T21:25:14.0715650Z * [new branch] gh/swolchok/769/orig -> origin/gh/swolchok/769/orig 2025-08-14T21:25:14.0721350Z * [new branch] gh/swolchok/771/base -> origin/gh/swolchok/771/base 2025-08-14T21:25:14.0721527Z * [new branch] gh/swolchok/771/head -> origin/gh/swolchok/771/head 2025-08-14T21:25:14.0721679Z * [new branch] gh/swolchok/771/orig -> origin/gh/swolchok/771/orig 2025-08-14T21:25:14.0721813Z * [new branch] gh/swolchok/772/base -> origin/gh/swolchok/772/base 2025-08-14T21:25:14.0721960Z * [new branch] gh/swolchok/772/head -> origin/gh/swolchok/772/head 2025-08-14T21:25:14.0722089Z * [new branch] gh/swolchok/772/orig -> origin/gh/swolchok/772/orig 2025-08-14T21:25:14.0722421Z * [new branch] gh/swolchok/773/base -> origin/gh/swolchok/773/base 2025-08-14T21:25:14.0722582Z * [new branch] gh/swolchok/773/head -> origin/gh/swolchok/773/head 2025-08-14T21:25:14.0723024Z * [new branch] gh/swolchok/773/orig -> origin/gh/swolchok/773/orig 2025-08-14T21:25:14.0728698Z * [new branch] gh/swolchok/786/base -> origin/gh/swolchok/786/base 2025-08-14T21:25:14.0729041Z * [new branch] gh/swolchok/786/head -> origin/gh/swolchok/786/head 2025-08-14T21:25:14.0729195Z * [new branch] gh/swolchok/786/orig -> origin/gh/swolchok/786/orig 2025-08-14T21:25:14.0729334Z * [new branch] gh/swolchok/787/base -> origin/gh/swolchok/787/base 2025-08-14T21:25:14.0729594Z * [new branch] gh/swolchok/787/head -> origin/gh/swolchok/787/head 2025-08-14T21:25:14.0734447Z * [new branch] gh/swolchok/787/orig -> origin/gh/swolchok/787/orig 2025-08-14T21:25:14.0738603Z * [new branch] gh/syed-ahmed/2/base -> origin/gh/syed-ahmed/2/base 2025-08-14T21:25:14.0738885Z * [new branch] gh/syed-ahmed/2/head -> origin/gh/syed-ahmed/2/head 2025-08-14T21:25:14.0742891Z * [new branch] gh/syed-ahmed/2/orig -> origin/gh/syed-ahmed/2/orig 2025-08-14T21:25:14.0743506Z * [new branch] gh/syed-ahmed/3/base -> origin/gh/syed-ahmed/3/base 2025-08-14T21:25:14.0748065Z * [new branch] gh/syed-ahmed/3/head -> origin/gh/syed-ahmed/3/head 2025-08-14T21:25:14.0748224Z * [new branch] gh/syed-ahmed/3/orig -> origin/gh/syed-ahmed/3/orig 2025-08-14T21:25:14.0748352Z * [new branch] gh/syed-ahmed/4/base -> origin/gh/syed-ahmed/4/base 2025-08-14T21:25:14.0748489Z * [new branch] gh/syed-ahmed/4/head -> origin/gh/syed-ahmed/4/head 2025-08-14T21:25:14.0748635Z * [new branch] gh/syed-ahmed/4/orig -> origin/gh/syed-ahmed/4/orig 2025-08-14T21:25:14.0748790Z * [new branch] gh/teja-rao/3/base -> origin/gh/teja-rao/3/base 2025-08-14T21:25:14.0749112Z * [new branch] gh/teja-rao/3/head -> origin/gh/teja-rao/3/head 2025-08-14T21:25:14.0749238Z * [new branch] gh/teja-rao/3/orig -> origin/gh/teja-rao/3/orig 2025-08-14T21:25:14.0749375Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-08-14T21:25:14.0749503Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-08-14T21:25:14.0749627Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-08-14T21:25:14.0749782Z * [new branch] gh/titaiwangms/1/base -> origin/gh/titaiwangms/1/base 2025-08-14T21:25:14.0749922Z * [new branch] gh/titaiwangms/1/head -> origin/gh/titaiwangms/1/head 2025-08-14T21:25:14.0750067Z * [new branch] gh/titaiwangms/1/orig -> origin/gh/titaiwangms/1/orig 2025-08-14T21:25:14.0750206Z * [new branch] gh/titaiwangms/2/base -> origin/gh/titaiwangms/2/base 2025-08-14T21:25:14.0750344Z * [new branch] gh/titaiwangms/2/head -> origin/gh/titaiwangms/2/head 2025-08-14T21:25:14.0750480Z * [new branch] gh/titaiwangms/2/orig -> origin/gh/titaiwangms/2/orig 2025-08-14T21:25:14.0750613Z * [new branch] gh/titaiwangms/3/base -> origin/gh/titaiwangms/3/base 2025-08-14T21:25:14.0750754Z * [new branch] gh/titaiwangms/3/head -> origin/gh/titaiwangms/3/head 2025-08-14T21:25:14.0750885Z * [new branch] gh/titaiwangms/3/orig -> origin/gh/titaiwangms/3/orig 2025-08-14T21:25:14.0751019Z * [new branch] gh/titaiwangms/4/base -> origin/gh/titaiwangms/4/base 2025-08-14T21:25:14.0751158Z * [new branch] gh/titaiwangms/4/head -> origin/gh/titaiwangms/4/head 2025-08-14T21:25:14.0751340Z * [new branch] gh/titaiwangms/4/orig -> origin/gh/titaiwangms/4/orig 2025-08-14T21:25:14.0751482Z * [new branch] gh/titaiwangms/5/base -> origin/gh/titaiwangms/5/base 2025-08-14T21:25:14.0751629Z * [new branch] gh/titaiwangms/5/head -> origin/gh/titaiwangms/5/head 2025-08-14T21:25:14.0751764Z * [new branch] gh/titaiwangms/5/orig -> origin/gh/titaiwangms/5/orig 2025-08-14T21:25:14.0751905Z * [new branch] gh/titaiwangms/6/base -> origin/gh/titaiwangms/6/base 2025-08-14T21:25:14.0752039Z * [new branch] gh/titaiwangms/6/head -> origin/gh/titaiwangms/6/head 2025-08-14T21:25:14.0752197Z * [new branch] gh/titaiwangms/6/orig -> origin/gh/titaiwangms/6/orig 2025-08-14T21:25:14.0753264Z * [new branch] gh/titaiwangms/7/base -> origin/gh/titaiwangms/7/base 2025-08-14T21:25:14.0753951Z * [new branch] gh/titaiwangms/7/head -> origin/gh/titaiwangms/7/head 2025-08-14T21:25:14.0754202Z * [new branch] gh/titaiwangms/7/orig -> origin/gh/titaiwangms/7/orig 2025-08-14T21:25:14.0755465Z * [new branch] gh/titaiwangms/8/base -> origin/gh/titaiwangms/8/base 2025-08-14T21:25:14.0755725Z * [new branch] gh/titaiwangms/8/head -> origin/gh/titaiwangms/8/head 2025-08-14T21:25:14.0757015Z * [new branch] gh/titaiwangms/8/orig -> origin/gh/titaiwangms/8/orig 2025-08-14T21:25:14.0758911Z * [new branch] gh/tugsbayasgalan/1/base -> origin/gh/tugsbayasgalan/1/base 2025-08-14T21:25:14.0759070Z * [new branch] gh/tugsbayasgalan/1/head -> origin/gh/tugsbayasgalan/1/head 2025-08-14T21:25:14.0759222Z * [new branch] gh/tugsbayasgalan/1/orig -> origin/gh/tugsbayasgalan/1/orig 2025-08-14T21:25:14.0760959Z * [new branch] gh/v0i0/1/base -> origin/gh/v0i0/1/base 2025-08-14T21:25:14.0761119Z * [new branch] gh/v0i0/1/head -> origin/gh/v0i0/1/head 2025-08-14T21:25:14.0762152Z * [new branch] gh/v0i0/1/orig -> origin/gh/v0i0/1/orig 2025-08-14T21:25:14.0762531Z * [new branch] gh/v0i0/2/base -> origin/gh/v0i0/2/base 2025-08-14T21:25:14.0765922Z * [new branch] gh/v0i0/2/head -> origin/gh/v0i0/2/head 2025-08-14T21:25:14.0766059Z * [new branch] gh/v0i0/2/orig -> origin/gh/v0i0/2/orig 2025-08-14T21:25:14.0766243Z * [new branch] gh/v0i0/3/base -> origin/gh/v0i0/3/base 2025-08-14T21:25:14.0766936Z * [new branch] gh/v0i0/3/head -> origin/gh/v0i0/3/head 2025-08-14T21:25:14.0767312Z * [new branch] gh/v0i0/3/orig -> origin/gh/v0i0/3/orig 2025-08-14T21:25:14.0767926Z * [new branch] gh/v0i0/4/base -> origin/gh/v0i0/4/base 2025-08-14T21:25:14.0768359Z * [new branch] gh/v0i0/4/head -> origin/gh/v0i0/4/head 2025-08-14T21:25:14.0769788Z * [new branch] gh/v0i0/4/orig -> origin/gh/v0i0/4/orig 2025-08-14T21:25:14.0770036Z * [new branch] gh/v0i0/5/base -> origin/gh/v0i0/5/base 2025-08-14T21:25:14.0771309Z * [new branch] gh/v0i0/5/head -> origin/gh/v0i0/5/head 2025-08-14T21:25:14.0771457Z * [new branch] gh/v0i0/5/orig -> origin/gh/v0i0/5/orig 2025-08-14T21:25:14.0772723Z * [new branch] gh/v0i0/6/base -> origin/gh/v0i0/6/base 2025-08-14T21:25:14.0773360Z * [new branch] gh/v0i0/6/head -> origin/gh/v0i0/6/head 2025-08-14T21:25:14.0773508Z * [new branch] gh/v0i0/6/orig -> origin/gh/v0i0/6/orig 2025-08-14T21:25:14.0775225Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-08-14T21:25:14.0777258Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-08-14T21:25:14.0777668Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-08-14T21:25:14.0777839Z * [new branch] gh/wconstab/392/base -> origin/gh/wconstab/392/base 2025-08-14T21:25:14.0781404Z * [new branch] gh/wconstab/392/head -> origin/gh/wconstab/392/head 2025-08-14T21:25:14.0781715Z * [new branch] gh/wconstab/392/orig -> origin/gh/wconstab/392/orig 2025-08-14T21:25:14.0781864Z * [new branch] gh/wconstab/419/base -> origin/gh/wconstab/419/base 2025-08-14T21:25:14.0782009Z * [new branch] gh/wconstab/419/head -> origin/gh/wconstab/419/head 2025-08-14T21:25:14.0782146Z * [new branch] gh/wconstab/419/orig -> origin/gh/wconstab/419/orig 2025-08-14T21:25:14.0782275Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-08-14T21:25:14.0786460Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-08-14T21:25:14.0787042Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-08-14T21:25:14.0787314Z * [new branch] gh/wconstab/425/base -> origin/gh/wconstab/425/base 2025-08-14T21:25:14.0792051Z * [new branch] gh/wconstab/425/head -> origin/gh/wconstab/425/head 2025-08-14T21:25:14.0792241Z * [new branch] gh/wconstab/425/orig -> origin/gh/wconstab/425/orig 2025-08-14T21:25:14.0792558Z * [new branch] gh/wconstab/426/base -> origin/gh/wconstab/426/base 2025-08-14T21:25:14.0792696Z * [new branch] gh/wconstab/426/head -> origin/gh/wconstab/426/head 2025-08-14T21:25:14.0792834Z * [new branch] gh/wconstab/426/orig -> origin/gh/wconstab/426/orig 2025-08-14T21:25:14.0792965Z * [new branch] gh/wconstab/427/base -> origin/gh/wconstab/427/base 2025-08-14T21:25:14.0793114Z * [new branch] gh/wconstab/427/head -> origin/gh/wconstab/427/head 2025-08-14T21:25:14.0793285Z * [new branch] gh/wconstab/427/orig -> origin/gh/wconstab/427/orig 2025-08-14T21:25:14.0793591Z * [new branch] gh/wconstab/428/base -> origin/gh/wconstab/428/base 2025-08-14T21:25:14.0793736Z * [new branch] gh/wconstab/428/head -> origin/gh/wconstab/428/head 2025-08-14T21:25:14.0793881Z * [new branch] gh/wconstab/428/orig -> origin/gh/wconstab/428/orig 2025-08-14T21:25:14.0794147Z * [new branch] gh/wconstab/429/base -> origin/gh/wconstab/429/base 2025-08-14T21:25:14.0794301Z * [new branch] gh/wconstab/429/head -> origin/gh/wconstab/429/head 2025-08-14T21:25:14.0796649Z * [new branch] gh/wconstab/429/orig -> origin/gh/wconstab/429/orig 2025-08-14T21:25:14.0796828Z * [new branch] gh/wconstab/430/base -> origin/gh/wconstab/430/base 2025-08-14T21:25:14.0803422Z * [new branch] gh/wconstab/430/head -> origin/gh/wconstab/430/head 2025-08-14T21:25:14.0807618Z * [new branch] gh/wconstab/430/orig -> origin/gh/wconstab/430/orig 2025-08-14T21:25:14.0811954Z * [new branch] gh/wconstab/431/base -> origin/gh/wconstab/431/base 2025-08-14T21:25:14.0817270Z * [new branch] gh/wconstab/431/head -> origin/gh/wconstab/431/head 2025-08-14T21:25:14.0822100Z * [new branch] gh/wconstab/431/orig -> origin/gh/wconstab/431/orig 2025-08-14T21:25:14.0826306Z * [new branch] gh/wconstab/432/base -> origin/gh/wconstab/432/base 2025-08-14T21:25:14.0830396Z * [new branch] gh/wconstab/432/head -> origin/gh/wconstab/432/head 2025-08-14T21:25:14.0830588Z * [new branch] gh/wconstab/432/orig -> origin/gh/wconstab/432/orig 2025-08-14T21:25:14.0831020Z * [new branch] gh/wconstab/433/base -> origin/gh/wconstab/433/base 2025-08-14T21:25:14.0831411Z * [new branch] gh/wconstab/433/head -> origin/gh/wconstab/433/head 2025-08-14T21:25:14.0831567Z * [new branch] gh/wconstab/433/orig -> origin/gh/wconstab/433/orig 2025-08-14T21:25:14.0831710Z * [new branch] gh/wconstab/434/base -> origin/gh/wconstab/434/base 2025-08-14T21:25:14.0831844Z * [new branch] gh/wconstab/434/head -> origin/gh/wconstab/434/head 2025-08-14T21:25:14.0831973Z * [new branch] gh/wconstab/434/orig -> origin/gh/wconstab/434/orig 2025-08-14T21:25:14.0832111Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-08-14T21:25:14.0832241Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-08-14T21:25:14.0832376Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-08-14T21:25:14.0832503Z * [new branch] gh/wconstab/436/base -> origin/gh/wconstab/436/base 2025-08-14T21:25:14.0832635Z * [new branch] gh/wconstab/436/head -> origin/gh/wconstab/436/head 2025-08-14T21:25:14.0832773Z * [new branch] gh/wconstab/436/orig -> origin/gh/wconstab/436/orig 2025-08-14T21:25:14.0832902Z * [new branch] gh/wconstab/437/base -> origin/gh/wconstab/437/base 2025-08-14T21:25:14.0833038Z * [new branch] gh/wconstab/437/head -> origin/gh/wconstab/437/head 2025-08-14T21:25:14.0833166Z * [new branch] gh/wconstab/437/orig -> origin/gh/wconstab/437/orig 2025-08-14T21:25:14.0833292Z * [new branch] gh/wconstab/438/base -> origin/gh/wconstab/438/base 2025-08-14T21:25:14.0833423Z * [new branch] gh/wconstab/438/head -> origin/gh/wconstab/438/head 2025-08-14T21:25:14.0833551Z * [new branch] gh/wconstab/438/orig -> origin/gh/wconstab/438/orig 2025-08-14T21:25:14.0833685Z * [new branch] gh/wconstab/439/base -> origin/gh/wconstab/439/base 2025-08-14T21:25:14.0833817Z * [new branch] gh/wconstab/439/head -> origin/gh/wconstab/439/head 2025-08-14T21:25:14.0834107Z * [new branch] gh/wconstab/439/orig -> origin/gh/wconstab/439/orig 2025-08-14T21:25:14.0834244Z * [new branch] gh/wconstab/440/base -> origin/gh/wconstab/440/base 2025-08-14T21:25:14.0834374Z * [new branch] gh/wconstab/440/head -> origin/gh/wconstab/440/head 2025-08-14T21:25:14.0834543Z * [new branch] gh/wconstab/440/orig -> origin/gh/wconstab/440/orig 2025-08-14T21:25:14.0834699Z * [new branch] gh/wconstab/441/base -> origin/gh/wconstab/441/base 2025-08-14T21:25:14.0834844Z * [new branch] gh/wconstab/441/head -> origin/gh/wconstab/441/head 2025-08-14T21:25:14.0834992Z * [new branch] gh/wconstab/441/orig -> origin/gh/wconstab/441/orig 2025-08-14T21:25:14.0835132Z * [new branch] gh/wconstab/442/base -> origin/gh/wconstab/442/base 2025-08-14T21:25:14.0835280Z * [new branch] gh/wconstab/442/head -> origin/gh/wconstab/442/head 2025-08-14T21:25:14.0835442Z * [new branch] gh/wconstab/442/orig -> origin/gh/wconstab/442/orig 2025-08-14T21:25:14.0835592Z * [new branch] gh/weifengpy/27/base -> origin/gh/weifengpy/27/base 2025-08-14T21:25:14.0835744Z * [new branch] gh/weifengpy/27/head -> origin/gh/weifengpy/27/head 2025-08-14T21:25:14.0835891Z * [new branch] gh/weifengpy/27/orig -> origin/gh/weifengpy/27/orig 2025-08-14T21:25:14.0836231Z * [new branch] gh/weifengpy/30/base -> origin/gh/weifengpy/30/base 2025-08-14T21:25:14.0840272Z * [new branch] gh/weifengpy/30/head -> origin/gh/weifengpy/30/head 2025-08-14T21:25:14.0840425Z * [new branch] gh/weifengpy/30/orig -> origin/gh/weifengpy/30/orig 2025-08-14T21:25:14.0840690Z * [new branch] gh/weifengpy/31/base -> origin/gh/weifengpy/31/base 2025-08-14T21:25:14.0840865Z * [new branch] gh/weifengpy/31/head -> origin/gh/weifengpy/31/head 2025-08-14T21:25:14.0841017Z * [new branch] gh/weifengpy/31/orig -> origin/gh/weifengpy/31/orig 2025-08-14T21:25:14.0841184Z * [new branch] gh/weifengpy/32/base -> origin/gh/weifengpy/32/base 2025-08-14T21:25:14.0841325Z * [new branch] gh/weifengpy/32/head -> origin/gh/weifengpy/32/head 2025-08-14T21:25:14.0841545Z * [new branch] gh/weifengpy/32/orig -> origin/gh/weifengpy/32/orig 2025-08-14T21:25:14.0845266Z * [new branch] gh/weifengpy/33/base -> origin/gh/weifengpy/33/base 2025-08-14T21:25:14.0845475Z * [new branch] gh/weifengpy/33/head -> origin/gh/weifengpy/33/head 2025-08-14T21:25:14.0845640Z * [new branch] gh/weifengpy/33/orig -> origin/gh/weifengpy/33/orig 2025-08-14T21:25:14.0845821Z * [new branch] gh/williamwen42/196/base -> origin/gh/williamwen42/196/base 2025-08-14T21:25:14.0850904Z * [new branch] gh/williamwen42/196/head -> origin/gh/williamwen42/196/head 2025-08-14T21:25:14.0851101Z * [new branch] gh/williamwen42/196/orig -> origin/gh/williamwen42/196/orig 2025-08-14T21:25:14.0851268Z * [new branch] gh/williamwen42/209/base -> origin/gh/williamwen42/209/base 2025-08-14T21:25:14.0851443Z * [new branch] gh/williamwen42/209/head -> origin/gh/williamwen42/209/head 2025-08-14T21:25:14.0851599Z * [new branch] gh/williamwen42/209/orig -> origin/gh/williamwen42/209/orig 2025-08-14T21:25:14.0851791Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-08-14T21:25:14.0852422Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-08-14T21:25:14.0852643Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-08-14T21:25:14.0852940Z * [new branch] gh/williamwen42/252/base -> origin/gh/williamwen42/252/base 2025-08-14T21:25:14.0853114Z * [new branch] gh/williamwen42/252/head -> origin/gh/williamwen42/252/head 2025-08-14T21:25:14.0853280Z * [new branch] gh/williamwen42/252/orig -> origin/gh/williamwen42/252/orig 2025-08-14T21:25:14.0853740Z * [new branch] gh/williamwen42/256/base -> origin/gh/williamwen42/256/base 2025-08-14T21:25:14.0853907Z * [new branch] gh/williamwen42/256/head -> origin/gh/williamwen42/256/head 2025-08-14T21:25:14.0857474Z * [new branch] gh/williamwen42/256/orig -> origin/gh/williamwen42/256/orig 2025-08-14T21:25:14.0857720Z * [new branch] gh/williamwen42/258/base -> origin/gh/williamwen42/258/base 2025-08-14T21:25:14.0857864Z * [new branch] gh/williamwen42/258/head -> origin/gh/williamwen42/258/head 2025-08-14T21:25:14.0858032Z * [new branch] gh/williamwen42/258/orig -> origin/gh/williamwen42/258/orig 2025-08-14T21:25:14.0858187Z * [new branch] gh/williamwen42/260/base -> origin/gh/williamwen42/260/base 2025-08-14T21:25:14.0861180Z * [new branch] gh/williamwen42/260/head -> origin/gh/williamwen42/260/head 2025-08-14T21:25:14.0861324Z * [new branch] gh/williamwen42/260/orig -> origin/gh/williamwen42/260/orig 2025-08-14T21:25:14.0861463Z * [new branch] gh/williamwen42/261/base -> origin/gh/williamwen42/261/base 2025-08-14T21:25:14.0861608Z * [new branch] gh/williamwen42/261/head -> origin/gh/williamwen42/261/head 2025-08-14T21:25:14.0861751Z * [new branch] gh/williamwen42/261/orig -> origin/gh/williamwen42/261/orig 2025-08-14T21:25:14.0861896Z * [new branch] gh/williamwen42/262/base -> origin/gh/williamwen42/262/base 2025-08-14T21:25:14.0867679Z * [new branch] gh/williamwen42/262/head -> origin/gh/williamwen42/262/head 2025-08-14T21:25:14.0867926Z * [new branch] gh/williamwen42/262/orig -> origin/gh/williamwen42/262/orig 2025-08-14T21:25:14.0868111Z * [new branch] gh/williamwen42/263/base -> origin/gh/williamwen42/263/base 2025-08-14T21:25:14.0868271Z * [new branch] gh/williamwen42/263/head -> origin/gh/williamwen42/263/head 2025-08-14T21:25:14.0868463Z * [new branch] gh/williamwen42/263/orig -> origin/gh/williamwen42/263/orig 2025-08-14T21:25:14.0868625Z * [new branch] gh/williamwen42/264/base -> origin/gh/williamwen42/264/base 2025-08-14T21:25:14.0869508Z * [new branch] gh/williamwen42/264/head -> origin/gh/williamwen42/264/head 2025-08-14T21:25:14.0869801Z * [new branch] gh/williamwen42/264/orig -> origin/gh/williamwen42/264/orig 2025-08-14T21:25:14.0869960Z * [new branch] gh/williamwen42/265/base -> origin/gh/williamwen42/265/base 2025-08-14T21:25:14.0870105Z * [new branch] gh/williamwen42/265/head -> origin/gh/williamwen42/265/head 2025-08-14T21:25:14.0870250Z * [new branch] gh/williamwen42/265/orig -> origin/gh/williamwen42/265/orig 2025-08-14T21:25:14.0870394Z * [new branch] gh/williamwen42/266/base -> origin/gh/williamwen42/266/base 2025-08-14T21:25:14.0870539Z * [new branch] gh/williamwen42/266/head -> origin/gh/williamwen42/266/head 2025-08-14T21:25:14.0870707Z * [new branch] gh/williamwen42/266/orig -> origin/gh/williamwen42/266/orig 2025-08-14T21:25:14.0873186Z * [new branch] gh/williamwen42/267/base -> origin/gh/williamwen42/267/base 2025-08-14T21:25:14.0873375Z * [new branch] gh/williamwen42/267/head -> origin/gh/williamwen42/267/head 2025-08-14T21:25:14.0873522Z * [new branch] gh/williamwen42/267/orig -> origin/gh/williamwen42/267/orig 2025-08-14T21:25:14.0874212Z * [new branch] gh/williamwen42/268/base -> origin/gh/williamwen42/268/base 2025-08-14T21:25:14.0874818Z * [new branch] gh/williamwen42/268/head -> origin/gh/williamwen42/268/head 2025-08-14T21:25:14.0875528Z * [new branch] gh/williamwen42/268/orig -> origin/gh/williamwen42/268/orig 2025-08-14T21:25:14.0877101Z * [new branch] gh/williamwen42/269/base -> origin/gh/williamwen42/269/base 2025-08-14T21:25:14.0877308Z * [new branch] gh/williamwen42/269/head -> origin/gh/williamwen42/269/head 2025-08-14T21:25:14.0877520Z * [new branch] gh/williamwen42/269/orig -> origin/gh/williamwen42/269/orig 2025-08-14T21:25:14.0880657Z * [new branch] gh/williamwen42/270/base -> origin/gh/williamwen42/270/base 2025-08-14T21:25:14.0884788Z * [new branch] gh/williamwen42/270/head -> origin/gh/williamwen42/270/head 2025-08-14T21:25:14.0889909Z * [new branch] gh/williamwen42/270/orig -> origin/gh/williamwen42/270/orig 2025-08-14T21:25:14.0890131Z * [new branch] gh/williamwen42/271/base -> origin/gh/williamwen42/271/base 2025-08-14T21:25:14.0890306Z * [new branch] gh/williamwen42/271/head -> origin/gh/williamwen42/271/head 2025-08-14T21:25:14.0890451Z * [new branch] gh/williamwen42/271/orig -> origin/gh/williamwen42/271/orig 2025-08-14T21:25:14.0891113Z * [new branch] gh/williamwen42/272/base -> origin/gh/williamwen42/272/base 2025-08-14T21:25:14.0891320Z * [new branch] gh/williamwen42/272/head -> origin/gh/williamwen42/272/head 2025-08-14T21:25:14.0891482Z * [new branch] gh/williamwen42/272/orig -> origin/gh/williamwen42/272/orig 2025-08-14T21:25:14.0891644Z * [new branch] gh/williamwen42/273/base -> origin/gh/williamwen42/273/base 2025-08-14T21:25:14.0891809Z * [new branch] gh/williamwen42/273/head -> origin/gh/williamwen42/273/head 2025-08-14T21:25:14.0892123Z * [new branch] gh/williamwen42/273/orig -> origin/gh/williamwen42/273/orig 2025-08-14T21:25:14.0892277Z * [new branch] gh/williamwen42/274/base -> origin/gh/williamwen42/274/base 2025-08-14T21:25:14.0892429Z * [new branch] gh/williamwen42/274/head -> origin/gh/williamwen42/274/head 2025-08-14T21:25:14.0892571Z * [new branch] gh/williamwen42/274/orig -> origin/gh/williamwen42/274/orig 2025-08-14T21:25:14.0892716Z * [new branch] gh/williamwen42/275/base -> origin/gh/williamwen42/275/base 2025-08-14T21:25:14.0892884Z * [new branch] gh/williamwen42/275/head -> origin/gh/williamwen42/275/head 2025-08-14T21:25:14.0893040Z * [new branch] gh/williamwen42/276/base -> origin/gh/williamwen42/276/base 2025-08-14T21:25:14.0893206Z * [new branch] gh/williamwen42/276/head -> origin/gh/williamwen42/276/head 2025-08-14T21:25:14.0895235Z * [new branch] gh/williamwen42/276/orig -> origin/gh/williamwen42/276/orig 2025-08-14T21:25:14.0895770Z * [new branch] gh/williamwen42/277/base -> origin/gh/williamwen42/277/base 2025-08-14T21:25:14.0895979Z * [new branch] gh/williamwen42/277/head -> origin/gh/williamwen42/277/head 2025-08-14T21:25:14.0896153Z * [new branch] gh/williamwen42/277/orig -> origin/gh/williamwen42/277/orig 2025-08-14T21:25:14.0896310Z * [new branch] gh/williamwen42/278/base -> origin/gh/williamwen42/278/base 2025-08-14T21:25:14.0896493Z * [new branch] gh/williamwen42/278/head -> origin/gh/williamwen42/278/head 2025-08-14T21:25:14.0900383Z * [new branch] gh/williamwen42/278/orig -> origin/gh/williamwen42/278/orig 2025-08-14T21:25:14.0900576Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-08-14T21:25:14.0900724Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-08-14T21:25:14.0900904Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-08-14T21:25:14.0901206Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-08-14T21:25:14.0901339Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-08-14T21:25:14.0901474Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-08-14T21:25:14.0903006Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-08-14T21:25:14.0903347Z * [new branch] gh/xmfan/18/base -> origin/gh/xmfan/18/base 2025-08-14T21:25:14.0903492Z * [new branch] gh/xmfan/18/head -> origin/gh/xmfan/18/head 2025-08-14T21:25:14.0903623Z * [new branch] gh/xmfan/228/base -> origin/gh/xmfan/228/base 2025-08-14T21:25:14.0908166Z * [new branch] gh/xmfan/228/head -> origin/gh/xmfan/228/head 2025-08-14T21:25:14.0908504Z * [new branch] gh/xmfan/228/orig -> origin/gh/xmfan/228/orig 2025-08-14T21:25:14.0908818Z * [new branch] gh/xmfan/229/base -> origin/gh/xmfan/229/base 2025-08-14T21:25:14.0908965Z * [new branch] gh/xmfan/229/head -> origin/gh/xmfan/229/head 2025-08-14T21:25:14.0909087Z * [new branch] gh/xmfan/229/orig -> origin/gh/xmfan/229/orig 2025-08-14T21:25:14.0909207Z * [new branch] gh/xmfan/237/base -> origin/gh/xmfan/237/base 2025-08-14T21:25:14.0909335Z * [new branch] gh/xmfan/237/head -> origin/gh/xmfan/237/head 2025-08-14T21:25:14.0915953Z * [new branch] gh/xmfan/237/orig -> origin/gh/xmfan/237/orig 2025-08-14T21:25:14.0921189Z * [new branch] gh/xmfan/244/base -> origin/gh/xmfan/244/base 2025-08-14T21:25:14.0922773Z * [new branch] gh/xmfan/244/head -> origin/gh/xmfan/244/head 2025-08-14T21:25:14.0923335Z * [new branch] gh/xmfan/244/orig -> origin/gh/xmfan/244/orig 2025-08-14T21:25:14.0923628Z * [new branch] gh/xmfan/246/base -> origin/gh/xmfan/246/base 2025-08-14T21:25:14.0923789Z * [new branch] gh/xmfan/246/head -> origin/gh/xmfan/246/head 2025-08-14T21:25:14.0923923Z * [new branch] gh/xmfan/246/orig -> origin/gh/xmfan/246/orig 2025-08-14T21:25:14.0924046Z * [new branch] gh/xmfan/253/base -> origin/gh/xmfan/253/base 2025-08-14T21:25:14.0924296Z * [new branch] gh/xmfan/253/head -> origin/gh/xmfan/253/head 2025-08-14T21:25:14.0924448Z * [new branch] gh/xmfan/253/orig -> origin/gh/xmfan/253/orig 2025-08-14T21:25:14.0924570Z * [new branch] gh/xmfan/254/base -> origin/gh/xmfan/254/base 2025-08-14T21:25:14.0924701Z * [new branch] gh/xmfan/254/head -> origin/gh/xmfan/254/head 2025-08-14T21:25:14.0924955Z * [new branch] gh/xmfan/254/orig -> origin/gh/xmfan/254/orig 2025-08-14T21:25:14.0925084Z * [new branch] gh/xmfan/260/base -> origin/gh/xmfan/260/base 2025-08-14T21:25:14.0925342Z * [new branch] gh/xmfan/260/head -> origin/gh/xmfan/260/head 2025-08-14T21:25:14.0925514Z * [new branch] gh/xmfan/260/orig -> origin/gh/xmfan/260/orig 2025-08-14T21:25:14.0925655Z * [new branch] gh/xmfan/262/base -> origin/gh/xmfan/262/base 2025-08-14T21:25:14.0925775Z * [new branch] gh/xmfan/262/head -> origin/gh/xmfan/262/head 2025-08-14T21:25:14.0925894Z * [new branch] gh/xmfan/262/orig -> origin/gh/xmfan/262/orig 2025-08-14T21:25:14.0926026Z * [new branch] gh/xmfan/263/base -> origin/gh/xmfan/263/base 2025-08-14T21:25:14.0926147Z * [new branch] gh/xmfan/263/head -> origin/gh/xmfan/263/head 2025-08-14T21:25:14.0926278Z * [new branch] gh/xmfan/263/orig -> origin/gh/xmfan/263/orig 2025-08-14T21:25:14.0926576Z * [new branch] gh/xmfan/264/base -> origin/gh/xmfan/264/base 2025-08-14T21:25:14.0926696Z * [new branch] gh/xmfan/264/head -> origin/gh/xmfan/264/head 2025-08-14T21:25:14.0926829Z * [new branch] gh/xmfan/264/orig -> origin/gh/xmfan/264/orig 2025-08-14T21:25:14.0926959Z * [new branch] gh/xmfan/268/base -> origin/gh/xmfan/268/base 2025-08-14T21:25:14.0931469Z * [new branch] gh/xmfan/268/head -> origin/gh/xmfan/268/head 2025-08-14T21:25:14.0931796Z * [new branch] gh/xmfan/268/orig -> origin/gh/xmfan/268/orig 2025-08-14T21:25:14.0931947Z * [new branch] gh/xmfan/269/base -> origin/gh/xmfan/269/base 2025-08-14T21:25:14.0933547Z * [new branch] gh/xmfan/269/head -> origin/gh/xmfan/269/head 2025-08-14T21:25:14.0933876Z * [new branch] gh/xmfan/269/orig -> origin/gh/xmfan/269/orig 2025-08-14T21:25:14.0936384Z * [new branch] gh/xmfan/270/base -> origin/gh/xmfan/270/base 2025-08-14T21:25:14.0936718Z * [new branch] gh/xmfan/270/head -> origin/gh/xmfan/270/head 2025-08-14T21:25:14.0936866Z * [new branch] gh/xmfan/270/orig -> origin/gh/xmfan/270/orig 2025-08-14T21:25:14.0937080Z * [new branch] gh/xmfan/271/base -> origin/gh/xmfan/271/base 2025-08-14T21:25:14.0937588Z * [new branch] gh/xmfan/271/head -> origin/gh/xmfan/271/head 2025-08-14T21:25:14.0938540Z * [new branch] gh/xmfan/271/orig -> origin/gh/xmfan/271/orig 2025-08-14T21:25:14.0942439Z * [new branch] gh/xmfan/272/base -> origin/gh/xmfan/272/base 2025-08-14T21:25:14.0942762Z * [new branch] gh/xmfan/272/head -> origin/gh/xmfan/272/head 2025-08-14T21:25:14.0943057Z * [new branch] gh/xmfan/272/orig -> origin/gh/xmfan/272/orig 2025-08-14T21:25:14.0943201Z * [new branch] gh/xmfan/273/base -> origin/gh/xmfan/273/base 2025-08-14T21:25:14.0943324Z * [new branch] gh/xmfan/273/head -> origin/gh/xmfan/273/head 2025-08-14T21:25:14.0943585Z * [new branch] gh/xmfan/273/orig -> origin/gh/xmfan/273/orig 2025-08-14T21:25:14.0944480Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-08-14T21:25:14.0944626Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-08-14T21:25:14.0947462Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-08-14T21:25:14.0947753Z * [new branch] gh/xmfan/275/base -> origin/gh/xmfan/275/base 2025-08-14T21:25:14.0947949Z * [new branch] gh/xmfan/275/head -> origin/gh/xmfan/275/head 2025-08-14T21:25:14.0948173Z * [new branch] gh/xmfan/275/orig -> origin/gh/xmfan/275/orig 2025-08-14T21:25:14.0948326Z * [new branch] gh/xmfan/276/base -> origin/gh/xmfan/276/base 2025-08-14T21:25:14.0949466Z * [new branch] gh/xmfan/276/head -> origin/gh/xmfan/276/head 2025-08-14T21:25:14.0950040Z * [new branch] gh/xmfan/276/orig -> origin/gh/xmfan/276/orig 2025-08-14T21:25:14.0950506Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-08-14T21:25:14.0951744Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-08-14T21:25:14.0951917Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-08-14T21:25:14.0953386Z * [new branch] gh/xuanzhang816/12/base -> origin/gh/xuanzhang816/12/base 2025-08-14T21:25:14.0953659Z * [new branch] gh/xuanzhang816/12/head -> origin/gh/xuanzhang816/12/head 2025-08-14T21:25:14.0954499Z * [new branch] gh/xuanzhang816/12/orig -> origin/gh/xuanzhang816/12/orig 2025-08-14T21:25:14.0955847Z * [new branch] gh/xuanzhang816/14/base -> origin/gh/xuanzhang816/14/base 2025-08-14T21:25:14.0956359Z * [new branch] gh/xuanzhang816/14/head -> origin/gh/xuanzhang816/14/head 2025-08-14T21:25:14.0961340Z * [new branch] gh/xuanzhang816/14/orig -> origin/gh/xuanzhang816/14/orig 2025-08-14T21:25:14.0961495Z * [new branch] gh/xuanzhang816/18/base -> origin/gh/xuanzhang816/18/base 2025-08-14T21:25:14.0961642Z * [new branch] gh/xuanzhang816/18/head -> origin/gh/xuanzhang816/18/head 2025-08-14T21:25:14.0961778Z * [new branch] gh/xuanzhang816/18/orig -> origin/gh/xuanzhang816/18/orig 2025-08-14T21:25:14.0961917Z * [new branch] gh/xuanzhang816/19/base -> origin/gh/xuanzhang816/19/base 2025-08-14T21:25:14.0962094Z * [new branch] gh/xuanzhang816/19/head -> origin/gh/xuanzhang816/19/head 2025-08-14T21:25:14.0962251Z * [new branch] gh/xuanzhang816/19/orig -> origin/gh/xuanzhang816/19/orig 2025-08-14T21:25:14.0967739Z * [new branch] gh/xuanzhang816/20/base -> origin/gh/xuanzhang816/20/base 2025-08-14T21:25:14.0967947Z * [new branch] gh/xuanzhang816/20/head -> origin/gh/xuanzhang816/20/head 2025-08-14T21:25:14.0968118Z * [new branch] gh/xuanzhang816/20/orig -> origin/gh/xuanzhang816/20/orig 2025-08-14T21:25:14.0968275Z * [new branch] gh/xuanzhang816/21/base -> origin/gh/xuanzhang816/21/base 2025-08-14T21:25:14.0968430Z * [new branch] gh/xuanzhang816/21/head -> origin/gh/xuanzhang816/21/head 2025-08-14T21:25:14.0968582Z * [new branch] gh/xuanzhang816/21/orig -> origin/gh/xuanzhang816/21/orig 2025-08-14T21:25:14.0968743Z * [new branch] gh/xuanzhang816/22/base -> origin/gh/xuanzhang816/22/base 2025-08-14T21:25:14.0969092Z * [new branch] gh/xuanzhang816/22/head -> origin/gh/xuanzhang816/22/head 2025-08-14T21:25:14.0969277Z * [new branch] gh/xuanzhang816/22/orig -> origin/gh/xuanzhang816/22/orig 2025-08-14T21:25:14.0969429Z * [new branch] gh/xuanzhang816/23/base -> origin/gh/xuanzhang816/23/base 2025-08-14T21:25:14.0972532Z * [new branch] gh/xuanzhang816/23/head -> origin/gh/xuanzhang816/23/head 2025-08-14T21:25:14.0972866Z * [new branch] gh/xuanzhang816/23/orig -> origin/gh/xuanzhang816/23/orig 2025-08-14T21:25:14.0973059Z * [new branch] gh/xuanzhang816/24/base -> origin/gh/xuanzhang816/24/base 2025-08-14T21:25:14.0973270Z * [new branch] gh/xuanzhang816/24/head -> origin/gh/xuanzhang816/24/head 2025-08-14T21:25:14.0973431Z * [new branch] gh/xuanzhang816/24/orig -> origin/gh/xuanzhang816/24/orig 2025-08-14T21:25:14.0977876Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-08-14T21:25:14.0978181Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-08-14T21:25:14.0978339Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-08-14T21:25:14.0978580Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-08-14T21:25:14.0978732Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-08-14T21:25:14.0978941Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-08-14T21:25:14.0980229Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-08-14T21:25:14.0980395Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-08-14T21:25:14.0980616Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-08-14T21:25:14.0980776Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-08-14T21:25:14.0981161Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-08-14T21:25:14.0981320Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-08-14T21:25:14.0981553Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-08-14T21:25:14.0981694Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-08-14T21:25:14.0986886Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-08-14T21:25:14.0987071Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-08-14T21:25:14.0987207Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-08-14T21:25:14.0987353Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-08-14T21:25:14.0987505Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-08-14T21:25:14.0987659Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-08-14T21:25:14.0987802Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-08-14T21:25:14.0989140Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-08-14T21:25:14.0989555Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-08-14T21:25:14.0989727Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-08-14T21:25:14.0989877Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-08-14T21:25:14.0990014Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-08-14T21:25:14.0990154Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-08-14T21:25:14.0990669Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-08-14T21:25:14.0990838Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-08-14T21:25:14.0991495Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-08-14T21:25:14.0992054Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-08-14T21:25:14.0993087Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-08-14T21:25:14.0993682Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-08-14T21:25:14.0994637Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-08-14T21:25:14.0994889Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-08-14T21:25:14.0996375Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-08-14T21:25:14.0996873Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-08-14T21:25:14.0997868Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-08-14T21:25:14.0998543Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-08-14T21:25:14.0999544Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-08-14T21:25:14.0999771Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-08-14T21:25:14.1002393Z * [new branch] gh/yanbing-j/36/base -> origin/gh/yanbing-j/36/base 2025-08-14T21:25:14.1002595Z * [new branch] gh/yanbing-j/36/head -> origin/gh/yanbing-j/36/head 2025-08-14T21:25:14.1002742Z * [new branch] gh/yanbing-j/36/orig -> origin/gh/yanbing-j/36/orig 2025-08-14T21:25:14.1003498Z * [new branch] gh/yanbing-j/37/base -> origin/gh/yanbing-j/37/base 2025-08-14T21:25:14.1003860Z * [new branch] gh/yanbing-j/37/head -> origin/gh/yanbing-j/37/head 2025-08-14T21:25:14.1004142Z * [new branch] gh/yanbing-j/37/orig -> origin/gh/yanbing-j/37/orig 2025-08-14T21:25:14.1008598Z * [new branch] gh/yanbing-j/39/base -> origin/gh/yanbing-j/39/base 2025-08-14T21:25:14.1008956Z * [new branch] gh/yanbing-j/39/head -> origin/gh/yanbing-j/39/head 2025-08-14T21:25:14.1009112Z * [new branch] gh/yanbing-j/39/orig -> origin/gh/yanbing-j/39/orig 2025-08-14T21:25:14.1009282Z * [new branch] gh/yangw-dev/1/base -> origin/gh/yangw-dev/1/base 2025-08-14T21:25:14.1009434Z * [new branch] gh/yangw-dev/10/base -> origin/gh/yangw-dev/10/base 2025-08-14T21:25:14.1009589Z * [new branch] gh/yangw-dev/10/head -> origin/gh/yangw-dev/10/head 2025-08-14T21:25:14.1009789Z * [new branch] gh/yangw-dev/10/orig -> origin/gh/yangw-dev/10/orig 2025-08-14T21:25:14.1014460Z * [new branch] gh/yangw-dev/11/base -> origin/gh/yangw-dev/11/base 2025-08-14T21:25:14.1014674Z * [new branch] gh/yangw-dev/11/head -> origin/gh/yangw-dev/11/head 2025-08-14T21:25:14.1014819Z * [new branch] gh/yangw-dev/11/orig -> origin/gh/yangw-dev/11/orig 2025-08-14T21:25:14.1014949Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-08-14T21:25:14.1015088Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-08-14T21:25:14.1017544Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-08-14T21:25:14.1017877Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-08-14T21:25:14.1018031Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-08-14T21:25:14.1018338Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-08-14T21:25:14.1018490Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-08-14T21:25:14.1018620Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-08-14T21:25:14.1018745Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-08-14T21:25:14.1023103Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-08-14T21:25:14.1023295Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-08-14T21:25:14.1023452Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-08-14T21:25:14.1023601Z * [new branch] gh/yangw-dev/16/base -> origin/gh/yangw-dev/16/base 2025-08-14T21:25:14.1023760Z * [new branch] gh/yangw-dev/16/head -> origin/gh/yangw-dev/16/head 2025-08-14T21:25:14.1023936Z * [new branch] gh/yangw-dev/16/orig -> origin/gh/yangw-dev/16/orig 2025-08-14T21:25:14.1024088Z * [new branch] gh/yangw-dev/17/base -> origin/gh/yangw-dev/17/base 2025-08-14T21:25:14.1025819Z * [new branch] gh/yangw-dev/17/head -> origin/gh/yangw-dev/17/head 2025-08-14T21:25:14.1025975Z * [new branch] gh/yangw-dev/17/orig -> origin/gh/yangw-dev/17/orig 2025-08-14T21:25:14.1026128Z * [new branch] gh/yangw-dev/18/base -> origin/gh/yangw-dev/18/base 2025-08-14T21:25:14.1026270Z * [new branch] gh/yangw-dev/18/head -> origin/gh/yangw-dev/18/head 2025-08-14T21:25:14.1026395Z * [new branch] gh/yangw-dev/18/orig -> origin/gh/yangw-dev/18/orig 2025-08-14T21:25:14.1030153Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-08-14T21:25:14.1030638Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-08-14T21:25:14.1031032Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-08-14T21:25:14.1031210Z * [new branch] gh/yangw-dev/2/base -> origin/gh/yangw-dev/2/base 2025-08-14T21:25:14.1031406Z * [new branch] gh/yangw-dev/2/head -> origin/gh/yangw-dev/2/head 2025-08-14T21:25:14.1031563Z * [new branch] gh/yangw-dev/3/base -> origin/gh/yangw-dev/3/base 2025-08-14T21:25:14.1031727Z * [new branch] gh/yangw-dev/3/head -> origin/gh/yangw-dev/3/head 2025-08-14T21:25:14.1031870Z * [new branch] gh/yangw-dev/4/base -> origin/gh/yangw-dev/4/base 2025-08-14T21:25:14.1032016Z * [new branch] gh/yangw-dev/4/head -> origin/gh/yangw-dev/4/head 2025-08-14T21:25:14.1032373Z * [new branch] gh/yangw-dev/5/base -> origin/gh/yangw-dev/5/base 2025-08-14T21:25:14.1032546Z * [new branch] gh/yangw-dev/5/head -> origin/gh/yangw-dev/5/head 2025-08-14T21:25:14.1033814Z * [new branch] gh/yangw-dev/6/base -> origin/gh/yangw-dev/6/base 2025-08-14T21:25:14.1034080Z * [new branch] gh/yangw-dev/6/head -> origin/gh/yangw-dev/6/head 2025-08-14T21:25:14.1035320Z * [new branch] gh/yangw-dev/7/base -> origin/gh/yangw-dev/7/base 2025-08-14T21:25:14.1035582Z * [new branch] gh/yangw-dev/7/head -> origin/gh/yangw-dev/7/head 2025-08-14T21:25:14.1042870Z * [new branch] gh/yangw-dev/8/base -> origin/gh/yangw-dev/8/base 2025-08-14T21:25:14.1048005Z * [new branch] gh/yangw-dev/8/head -> origin/gh/yangw-dev/8/head 2025-08-14T21:25:14.1053068Z * [new branch] gh/yangw-dev/8/orig -> origin/gh/yangw-dev/8/orig 2025-08-14T21:25:14.1055001Z * [new branch] gh/yangw-dev/9/base -> origin/gh/yangw-dev/9/base 2025-08-14T21:25:14.1055337Z * [new branch] gh/yangw-dev/9/head -> origin/gh/yangw-dev/9/head 2025-08-14T21:25:14.1055493Z * [new branch] gh/yangw-dev/9/orig -> origin/gh/yangw-dev/9/orig 2025-08-14T21:25:14.1055815Z * [new branch] gh/ydwu4/233/base -> origin/gh/ydwu4/233/base 2025-08-14T21:25:14.1056011Z * [new branch] gh/ydwu4/233/head -> origin/gh/ydwu4/233/head 2025-08-14T21:25:14.1056152Z * [new branch] gh/ydwu4/233/orig -> origin/gh/ydwu4/233/orig 2025-08-14T21:25:14.1056298Z * [new branch] gh/ydwu4/246/base -> origin/gh/ydwu4/246/base 2025-08-14T21:25:14.1056441Z * [new branch] gh/ydwu4/246/head -> origin/gh/ydwu4/246/head 2025-08-14T21:25:14.1056576Z * [new branch] gh/ydwu4/246/orig -> origin/gh/ydwu4/246/orig 2025-08-14T21:25:14.1056728Z * [new branch] gh/ydwu4/253/base -> origin/gh/ydwu4/253/base 2025-08-14T21:25:14.1056877Z * [new branch] gh/ydwu4/253/head -> origin/gh/ydwu4/253/head 2025-08-14T21:25:14.1057013Z * [new branch] gh/ydwu4/253/orig -> origin/gh/ydwu4/253/orig 2025-08-14T21:25:14.1057150Z * [new branch] gh/ydwu4/255/base -> origin/gh/ydwu4/255/base 2025-08-14T21:25:14.1057288Z * [new branch] gh/ydwu4/255/head -> origin/gh/ydwu4/255/head 2025-08-14T21:25:14.1057422Z * [new branch] gh/ydwu4/255/orig -> origin/gh/ydwu4/255/orig 2025-08-14T21:25:14.1057561Z * [new branch] gh/ydwu4/259/base -> origin/gh/ydwu4/259/base 2025-08-14T21:25:14.1057704Z * [new branch] gh/ydwu4/259/head -> origin/gh/ydwu4/259/head 2025-08-14T21:25:14.1057843Z * [new branch] gh/ydwu4/259/orig -> origin/gh/ydwu4/259/orig 2025-08-14T21:25:14.1057983Z * [new branch] gh/ydwu4/262/base -> origin/gh/ydwu4/262/base 2025-08-14T21:25:14.1058117Z * [new branch] gh/ydwu4/262/head -> origin/gh/ydwu4/262/head 2025-08-14T21:25:14.1058496Z * [new branch] gh/ydwu4/262/orig -> origin/gh/ydwu4/262/orig 2025-08-14T21:25:14.1058644Z * [new branch] gh/ydwu4/263/base -> origin/gh/ydwu4/263/base 2025-08-14T21:25:14.1058793Z * [new branch] gh/ydwu4/263/head -> origin/gh/ydwu4/263/head 2025-08-14T21:25:14.1058931Z * [new branch] gh/ydwu4/263/orig -> origin/gh/ydwu4/263/orig 2025-08-14T21:25:14.1059079Z * [new branch] gh/ydwu4/269/base -> origin/gh/ydwu4/269/base 2025-08-14T21:25:14.1059205Z * [new branch] gh/ydwu4/269/head -> origin/gh/ydwu4/269/head 2025-08-14T21:25:14.1059331Z * [new branch] gh/ydwu4/269/orig -> origin/gh/ydwu4/269/orig 2025-08-14T21:25:14.1059479Z * [new branch] gh/ydwu4/270/base -> origin/gh/ydwu4/270/base 2025-08-14T21:25:14.1061202Z * [new branch] gh/ydwu4/270/head -> origin/gh/ydwu4/270/head 2025-08-14T21:25:14.1061541Z * [new branch] gh/ydwu4/270/orig -> origin/gh/ydwu4/270/orig 2025-08-14T21:25:14.1062030Z * [new branch] gh/ydwu4/272/base -> origin/gh/ydwu4/272/base 2025-08-14T21:25:14.1064236Z * [new branch] gh/ydwu4/272/head -> origin/gh/ydwu4/272/head 2025-08-14T21:25:14.1064419Z * [new branch] gh/ydwu4/272/orig -> origin/gh/ydwu4/272/orig 2025-08-14T21:25:14.1064556Z * [new branch] gh/ydwu4/275/base -> origin/gh/ydwu4/275/base 2025-08-14T21:25:14.1064987Z * [new branch] gh/ydwu4/275/head -> origin/gh/ydwu4/275/head 2025-08-14T21:25:14.1067388Z * [new branch] gh/ydwu4/275/orig -> origin/gh/ydwu4/275/orig 2025-08-14T21:25:14.1067717Z * [new branch] gh/ydwu4/276/base -> origin/gh/ydwu4/276/base 2025-08-14T21:25:14.1068069Z * [new branch] gh/ydwu4/276/head -> origin/gh/ydwu4/276/head 2025-08-14T21:25:14.1068282Z * [new branch] gh/ydwu4/276/orig -> origin/gh/ydwu4/276/orig 2025-08-14T21:25:14.1069848Z * [new branch] gh/ydwu4/277/base -> origin/gh/ydwu4/277/base 2025-08-14T21:25:14.1070463Z * [new branch] gh/ydwu4/277/head -> origin/gh/ydwu4/277/head 2025-08-14T21:25:14.1070624Z * [new branch] gh/ydwu4/277/orig -> origin/gh/ydwu4/277/orig 2025-08-14T21:25:14.1071876Z * [new branch] gh/ydwu4/278/base -> origin/gh/ydwu4/278/base 2025-08-14T21:25:14.1072032Z * [new branch] gh/ydwu4/278/head -> origin/gh/ydwu4/278/head 2025-08-14T21:25:14.1073133Z * [new branch] gh/ydwu4/278/orig -> origin/gh/ydwu4/278/orig 2025-08-14T21:25:14.1074246Z * [new branch] gh/ydwu4/279/base -> origin/gh/ydwu4/279/base 2025-08-14T21:25:14.1074801Z * [new branch] gh/ydwu4/279/head -> origin/gh/ydwu4/279/head 2025-08-14T21:25:14.1075633Z * [new branch] gh/ydwu4/279/orig -> origin/gh/ydwu4/279/orig 2025-08-14T21:25:14.1080138Z * [new branch] gh/ydwu4/280/base -> origin/gh/ydwu4/280/base 2025-08-14T21:25:14.1080461Z * [new branch] gh/ydwu4/280/head -> origin/gh/ydwu4/280/head 2025-08-14T21:25:14.1080612Z * [new branch] gh/ydwu4/280/orig -> origin/gh/ydwu4/280/orig 2025-08-14T21:25:14.1080746Z * [new branch] gh/ydwu4/281/base -> origin/gh/ydwu4/281/base 2025-08-14T21:25:14.1080990Z * [new branch] gh/ydwu4/281/head -> origin/gh/ydwu4/281/head 2025-08-14T21:25:14.1081129Z * [new branch] gh/ydwu4/281/orig -> origin/gh/ydwu4/281/orig 2025-08-14T21:25:14.1081652Z * [new branch] gh/ydwu4/282/base -> origin/gh/ydwu4/282/base 2025-08-14T21:25:14.1082762Z * [new branch] gh/ydwu4/282/head -> origin/gh/ydwu4/282/head 2025-08-14T21:25:14.1083163Z * [new branch] gh/ydwu4/282/orig -> origin/gh/ydwu4/282/orig 2025-08-14T21:25:14.1085692Z * [new branch] gh/ydwu4/283/base -> origin/gh/ydwu4/283/base 2025-08-14T21:25:14.1086028Z * [new branch] gh/ydwu4/283/head -> origin/gh/ydwu4/283/head 2025-08-14T21:25:14.1086187Z * [new branch] gh/ydwu4/283/orig -> origin/gh/ydwu4/283/orig 2025-08-14T21:25:14.1086408Z * [new branch] gh/ydwu4/284/base -> origin/gh/ydwu4/284/base 2025-08-14T21:25:14.1086560Z * [new branch] gh/ydwu4/284/head -> origin/gh/ydwu4/284/head 2025-08-14T21:25:14.1090830Z * [new branch] gh/ydwu4/284/orig -> origin/gh/ydwu4/284/orig 2025-08-14T21:25:14.1091166Z * [new branch] gh/ydwu4/285/base -> origin/gh/ydwu4/285/base 2025-08-14T21:25:14.1091331Z * [new branch] gh/ydwu4/285/head -> origin/gh/ydwu4/285/head 2025-08-14T21:25:14.1091555Z * [new branch] gh/ydwu4/285/orig -> origin/gh/ydwu4/285/orig 2025-08-14T21:25:14.1091894Z * [new branch] gh/ydwu4/286/base -> origin/gh/ydwu4/286/base 2025-08-14T21:25:14.1092044Z * [new branch] gh/ydwu4/286/head -> origin/gh/ydwu4/286/head 2025-08-14T21:25:14.1092646Z * [new branch] gh/ydwu4/286/orig -> origin/gh/ydwu4/286/orig 2025-08-14T21:25:14.1101813Z * [new branch] gh/ydwu4/287/base -> origin/gh/ydwu4/287/base 2025-08-14T21:25:14.1103950Z * [new branch] gh/ydwu4/287/head -> origin/gh/ydwu4/287/head 2025-08-14T21:25:14.1104234Z * [new branch] gh/ydwu4/287/orig -> origin/gh/ydwu4/287/orig 2025-08-14T21:25:14.1109995Z * [new branch] gh/ydwu4/288/base -> origin/gh/ydwu4/288/base 2025-08-14T21:25:14.1112398Z * [new branch] gh/ydwu4/288/head -> origin/gh/ydwu4/288/head 2025-08-14T21:25:14.1112603Z * [new branch] gh/ydwu4/288/orig -> origin/gh/ydwu4/288/orig 2025-08-14T21:25:14.1112753Z * [new branch] gh/ydwu4/289/base -> origin/gh/ydwu4/289/base 2025-08-14T21:25:14.1112886Z * [new branch] gh/ydwu4/289/head -> origin/gh/ydwu4/289/head 2025-08-14T21:25:14.1113015Z * [new branch] gh/ydwu4/289/orig -> origin/gh/ydwu4/289/orig 2025-08-14T21:25:14.1113153Z * [new branch] gh/ydwu4/290/base -> origin/gh/ydwu4/290/base 2025-08-14T21:25:14.1113282Z * [new branch] gh/ydwu4/290/head -> origin/gh/ydwu4/290/head 2025-08-14T21:25:14.1113412Z * [new branch] gh/ydwu4/290/orig -> origin/gh/ydwu4/290/orig 2025-08-14T21:25:14.1113550Z * [new branch] gh/ydwu4/291/base -> origin/gh/ydwu4/291/base 2025-08-14T21:25:14.1113681Z * [new branch] gh/ydwu4/291/head -> origin/gh/ydwu4/291/head 2025-08-14T21:25:14.1113821Z * [new branch] gh/ydwu4/291/orig -> origin/gh/ydwu4/291/orig 2025-08-14T21:25:14.1113947Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-08-14T21:25:14.1114073Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-08-14T21:25:14.1114207Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-08-14T21:25:14.1114333Z * [new branch] gh/ydwu4/293/base -> origin/gh/ydwu4/293/base 2025-08-14T21:25:14.1114471Z * [new branch] gh/ydwu4/293/head -> origin/gh/ydwu4/293/head 2025-08-14T21:25:14.1114600Z * [new branch] gh/ydwu4/293/orig -> origin/gh/ydwu4/293/orig 2025-08-14T21:25:14.1114728Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-08-14T21:25:14.1114864Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-08-14T21:25:14.1115050Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-08-14T21:25:14.1115176Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-08-14T21:25:14.1115306Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-08-14T21:25:14.1115435Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-08-14T21:25:14.1115570Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-08-14T21:25:14.1115697Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-08-14T21:25:14.1115824Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-08-14T21:25:14.1115962Z * [new branch] gh/ydwu4/297/base -> origin/gh/ydwu4/297/base 2025-08-14T21:25:14.1116291Z * [new branch] gh/ydwu4/297/head -> origin/gh/ydwu4/297/head 2025-08-14T21:25:14.1116438Z * [new branch] gh/ydwu4/297/orig -> origin/gh/ydwu4/297/orig 2025-08-14T21:25:14.1116566Z * [new branch] gh/ydwu4/298/base -> origin/gh/ydwu4/298/base 2025-08-14T21:25:14.1116881Z * [new branch] gh/ydwu4/298/head -> origin/gh/ydwu4/298/head 2025-08-14T21:25:14.1123106Z * [new branch] gh/ydwu4/298/orig -> origin/gh/ydwu4/298/orig 2025-08-14T21:25:14.1123457Z * [new branch] gh/ydwu4/299/base -> origin/gh/ydwu4/299/base 2025-08-14T21:25:14.1123630Z * [new branch] gh/ydwu4/299/head -> origin/gh/ydwu4/299/head 2025-08-14T21:25:14.1123788Z * [new branch] gh/ydwu4/299/orig -> origin/gh/ydwu4/299/orig 2025-08-14T21:25:14.1124052Z * [new branch] gh/ydwu4/300/base -> origin/gh/ydwu4/300/base 2025-08-14T21:25:14.1124354Z * [new branch] gh/ydwu4/300/head -> origin/gh/ydwu4/300/head 2025-08-14T21:25:14.1124928Z * [new branch] gh/ydwu4/300/orig -> origin/gh/ydwu4/300/orig 2025-08-14T21:25:14.1125092Z * [new branch] gh/ydwu4/301/base -> origin/gh/ydwu4/301/base 2025-08-14T21:25:14.1125227Z * [new branch] gh/ydwu4/301/head -> origin/gh/ydwu4/301/head 2025-08-14T21:25:14.1125351Z * [new branch] gh/ydwu4/301/orig -> origin/gh/ydwu4/301/orig 2025-08-14T21:25:14.1125474Z * [new branch] gh/ydwu4/302/base -> origin/gh/ydwu4/302/base 2025-08-14T21:25:14.1131599Z * [new branch] gh/ydwu4/302/head -> origin/gh/ydwu4/302/head 2025-08-14T21:25:14.1131910Z * [new branch] gh/ydwu4/302/orig -> origin/gh/ydwu4/302/orig 2025-08-14T21:25:14.1132063Z * [new branch] gh/ydwu4/303/base -> origin/gh/ydwu4/303/base 2025-08-14T21:25:14.1132200Z * [new branch] gh/ydwu4/303/head -> origin/gh/ydwu4/303/head 2025-08-14T21:25:14.1132468Z * [new branch] gh/ydwu4/303/orig -> origin/gh/ydwu4/303/orig 2025-08-14T21:25:14.1132603Z * [new branch] gh/ydwu4/304/base -> origin/gh/ydwu4/304/base 2025-08-14T21:25:14.1132794Z * [new branch] gh/ydwu4/304/head -> origin/gh/ydwu4/304/head 2025-08-14T21:25:14.1133406Z * [new branch] gh/ydwu4/304/orig -> origin/gh/ydwu4/304/orig 2025-08-14T21:25:14.1135451Z * [new branch] gh/ydwu4/305/base -> origin/gh/ydwu4/305/base 2025-08-14T21:25:14.1135622Z * [new branch] gh/ydwu4/305/head -> origin/gh/ydwu4/305/head 2025-08-14T21:25:14.1135764Z * [new branch] gh/ydwu4/305/orig -> origin/gh/ydwu4/305/orig 2025-08-14T21:25:14.1135893Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-08-14T21:25:14.1136034Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-08-14T21:25:14.1136387Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-08-14T21:25:14.1136512Z * [new branch] gh/ydwu4/307/base -> origin/gh/ydwu4/307/base 2025-08-14T21:25:14.1136647Z * [new branch] gh/ydwu4/307/head -> origin/gh/ydwu4/307/head 2025-08-14T21:25:14.1139383Z * [new branch] gh/ydwu4/307/orig -> origin/gh/ydwu4/307/orig 2025-08-14T21:25:14.1139532Z * [new branch] gh/ydwu4/308/base -> origin/gh/ydwu4/308/base 2025-08-14T21:25:14.1139769Z * [new branch] gh/ydwu4/308/head -> origin/gh/ydwu4/308/head 2025-08-14T21:25:14.1139907Z * [new branch] gh/ydwu4/308/orig -> origin/gh/ydwu4/308/orig 2025-08-14T21:25:14.1140113Z * [new branch] gh/ydwu4/309/base -> origin/gh/ydwu4/309/base 2025-08-14T21:25:14.1140257Z * [new branch] gh/ydwu4/309/head -> origin/gh/ydwu4/309/head 2025-08-14T21:25:14.1144013Z * [new branch] gh/ydwu4/309/orig -> origin/gh/ydwu4/309/orig 2025-08-14T21:25:14.1144163Z * [new branch] gh/ydwu4/310/base -> origin/gh/ydwu4/310/base 2025-08-14T21:25:14.1144283Z * [new branch] gh/ydwu4/310/head -> origin/gh/ydwu4/310/head 2025-08-14T21:25:14.1144417Z * [new branch] gh/ydwu4/310/orig -> origin/gh/ydwu4/310/orig 2025-08-14T21:25:14.1144541Z * [new branch] gh/ydwu4/311/base -> origin/gh/ydwu4/311/base 2025-08-14T21:25:14.1144670Z * [new branch] gh/ydwu4/311/head -> origin/gh/ydwu4/311/head 2025-08-14T21:25:14.1144913Z * [new branch] gh/ydwu4/311/orig -> origin/gh/ydwu4/311/orig 2025-08-14T21:25:14.1151238Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-08-14T21:25:14.1151571Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-08-14T21:25:14.1151744Z * [new branch] gh/yf225/171/base -> origin/gh/yf225/171/base 2025-08-14T21:25:14.1151873Z * [new branch] gh/yf225/171/head -> origin/gh/yf225/171/head 2025-08-14T21:25:14.1152012Z * [new branch] gh/yf225/171/orig -> origin/gh/yf225/171/orig 2025-08-14T21:25:14.1152141Z * [new branch] gh/yf225/172/base -> origin/gh/yf225/172/base 2025-08-14T21:25:14.1152266Z * [new branch] gh/yf225/172/head -> origin/gh/yf225/172/head 2025-08-14T21:25:14.1152400Z * [new branch] gh/yf225/172/orig -> origin/gh/yf225/172/orig 2025-08-14T21:25:14.1152538Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-08-14T21:25:14.1152673Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-08-14T21:25:14.1153007Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-08-14T21:25:14.1154727Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-08-14T21:25:14.1155415Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-08-14T21:25:14.1156902Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-08-14T21:25:14.1157069Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-08-14T21:25:14.1157221Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-08-14T21:25:14.1159816Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-08-14T21:25:14.1160390Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-08-14T21:25:14.1160606Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-08-14T21:25:14.1160956Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-08-14T21:25:14.1163534Z * [new branch] gh/ysiraichi/79/base -> origin/gh/ysiraichi/79/base 2025-08-14T21:25:14.1163840Z * [new branch] gh/ysiraichi/79/head -> origin/gh/ysiraichi/79/head 2025-08-14T21:25:14.1167521Z * [new branch] gh/ysiraichi/79/orig -> origin/gh/ysiraichi/79/orig 2025-08-14T21:25:14.1167698Z * [new branch] gh/ysiraichi/81/base -> origin/gh/ysiraichi/81/base 2025-08-14T21:25:14.1167849Z * [new branch] gh/ysiraichi/81/head -> origin/gh/ysiraichi/81/head 2025-08-14T21:25:14.1167995Z * [new branch] gh/ysiraichi/81/orig -> origin/gh/ysiraichi/81/orig 2025-08-14T21:25:14.1168134Z * [new branch] gh/ysiraichi/84/base -> origin/gh/ysiraichi/84/base 2025-08-14T21:25:14.1172622Z * [new branch] gh/ysiraichi/84/head -> origin/gh/ysiraichi/84/head 2025-08-14T21:25:14.1172956Z * [new branch] gh/ysiraichi/84/orig -> origin/gh/ysiraichi/84/orig 2025-08-14T21:25:14.1173099Z * [new branch] gh/ysiraichi/85/base -> origin/gh/ysiraichi/85/base 2025-08-14T21:25:14.1173241Z * [new branch] gh/ysiraichi/85/head -> origin/gh/ysiraichi/85/head 2025-08-14T21:25:14.1173373Z * [new branch] gh/ysiraichi/85/orig -> origin/gh/ysiraichi/85/orig 2025-08-14T21:25:14.1173515Z * [new branch] gh/ysiraichi/86/base -> origin/gh/ysiraichi/86/base 2025-08-14T21:25:14.1173643Z * [new branch] gh/ysiraichi/86/head -> origin/gh/ysiraichi/86/head 2025-08-14T21:25:14.1176359Z * [new branch] gh/ysiraichi/86/orig -> origin/gh/ysiraichi/86/orig 2025-08-14T21:25:14.1176511Z * [new branch] gh/ysiraichi/87/base -> origin/gh/ysiraichi/87/base 2025-08-14T21:25:14.1176924Z * [new branch] gh/ysiraichi/87/head -> origin/gh/ysiraichi/87/head 2025-08-14T21:25:14.1177086Z * [new branch] gh/ysiraichi/87/orig -> origin/gh/ysiraichi/87/orig 2025-08-14T21:25:14.1180671Z * [new branch] gh/ysiraichi/88/base -> origin/gh/ysiraichi/88/base 2025-08-14T21:25:14.1180964Z * [new branch] gh/ysiraichi/88/head -> origin/gh/ysiraichi/88/head 2025-08-14T21:25:14.1181129Z * [new branch] gh/ysiraichi/88/orig -> origin/gh/ysiraichi/88/orig 2025-08-14T21:25:14.1181297Z * [new branch] gh/yuguo68/1/base -> origin/gh/yuguo68/1/base 2025-08-14T21:25:14.1181430Z * [new branch] gh/yuguo68/1/head -> origin/gh/yuguo68/1/head 2025-08-14T21:25:14.1181565Z * [new branch] gh/yuguo68/1/orig -> origin/gh/yuguo68/1/orig 2025-08-14T21:25:14.1187110Z * [new branch] gh/yuguo68/2/base -> origin/gh/yuguo68/2/base 2025-08-14T21:25:14.1187395Z * [new branch] gh/yuguo68/2/head -> origin/gh/yuguo68/2/head 2025-08-14T21:25:14.1193102Z * [new branch] gh/yuguo68/2/orig -> origin/gh/yuguo68/2/orig 2025-08-14T21:25:14.1193304Z * [new branch] gh/zhxchen17/25/base -> origin/gh/zhxchen17/25/base 2025-08-14T21:25:14.1193459Z * [new branch] gh/zhxchen17/25/head -> origin/gh/zhxchen17/25/head 2025-08-14T21:25:14.1193607Z * [new branch] gh/zhxchen17/25/orig -> origin/gh/zhxchen17/25/orig 2025-08-14T21:25:14.1193758Z * [new branch] gh/zhxchen17/31/base -> origin/gh/zhxchen17/31/base 2025-08-14T21:25:14.1193898Z * [new branch] gh/zhxchen17/31/head -> origin/gh/zhxchen17/31/head 2025-08-14T21:25:14.1194049Z * [new branch] gh/zhxchen17/31/orig -> origin/gh/zhxchen17/31/orig 2025-08-14T21:25:14.1194184Z * [new branch] gh/zhxchen17/33/base -> origin/gh/zhxchen17/33/base 2025-08-14T21:25:14.1194527Z * [new branch] gh/zhxchen17/33/head -> origin/gh/zhxchen17/33/head 2025-08-14T21:25:14.1194665Z * [new branch] gh/zhxchen17/33/orig -> origin/gh/zhxchen17/33/orig 2025-08-14T21:25:14.1194799Z * [new branch] gh/zhxchen17/34/base -> origin/gh/zhxchen17/34/base 2025-08-14T21:25:14.1194943Z * [new branch] gh/zhxchen17/34/head -> origin/gh/zhxchen17/34/head 2025-08-14T21:25:14.1195080Z * [new branch] gh/zhxchen17/35/base -> origin/gh/zhxchen17/35/base 2025-08-14T21:25:14.1203148Z * [new branch] gh/zhxchen17/35/head -> origin/gh/zhxchen17/35/head 2025-08-14T21:25:14.1205300Z * [new branch] gh/zhxchen17/36/base -> origin/gh/zhxchen17/36/base 2025-08-14T21:25:14.1205575Z * [new branch] gh/zhxchen17/36/head -> origin/gh/zhxchen17/36/head 2025-08-14T21:25:14.1212492Z * [new branch] gh/zhxchen17/36/orig -> origin/gh/zhxchen17/36/orig 2025-08-14T21:25:14.1216063Z * [new branch] gh/zklaus/1/base -> origin/gh/zklaus/1/base 2025-08-14T21:25:14.1221624Z * [new branch] gh/zklaus/1/head -> origin/gh/zklaus/1/head 2025-08-14T21:25:14.1224802Z * [new branch] gh/zklaus/1/orig -> origin/gh/zklaus/1/orig 2025-08-14T21:25:14.1225272Z * [new branch] gh/zklaus/10/base -> origin/gh/zklaus/10/base 2025-08-14T21:25:14.1225497Z * [new branch] gh/zklaus/10/head -> origin/gh/zklaus/10/head 2025-08-14T21:25:14.1225660Z * [new branch] gh/zklaus/10/orig -> origin/gh/zklaus/10/orig 2025-08-14T21:25:14.1225813Z * [new branch] gh/zklaus/11/base -> origin/gh/zklaus/11/base 2025-08-14T21:25:14.1225959Z * [new branch] gh/zklaus/11/head -> origin/gh/zklaus/11/head 2025-08-14T21:25:14.1226309Z * [new branch] gh/zklaus/11/orig -> origin/gh/zklaus/11/orig 2025-08-14T21:25:14.1226462Z * [new branch] gh/zklaus/12/base -> origin/gh/zklaus/12/base 2025-08-14T21:25:14.1226600Z * [new branch] gh/zklaus/12/head -> origin/gh/zklaus/12/head 2025-08-14T21:25:14.1226738Z * [new branch] gh/zklaus/12/orig -> origin/gh/zklaus/12/orig 2025-08-14T21:25:14.1226869Z * [new branch] gh/zklaus/14/base -> origin/gh/zklaus/14/base 2025-08-14T21:25:14.1227014Z * [new branch] gh/zklaus/14/head -> origin/gh/zklaus/14/head 2025-08-14T21:25:14.1227157Z * [new branch] gh/zklaus/14/orig -> origin/gh/zklaus/14/orig 2025-08-14T21:25:14.1227299Z * [new branch] gh/zklaus/15/base -> origin/gh/zklaus/15/base 2025-08-14T21:25:14.1227444Z * [new branch] gh/zklaus/15/head -> origin/gh/zklaus/15/head 2025-08-14T21:25:14.1227588Z * [new branch] gh/zklaus/15/orig -> origin/gh/zklaus/15/orig 2025-08-14T21:25:14.1227719Z * [new branch] gh/zklaus/16/base -> origin/gh/zklaus/16/base 2025-08-14T21:25:14.1227856Z * [new branch] gh/zklaus/16/head -> origin/gh/zklaus/16/head 2025-08-14T21:25:14.1227999Z * [new branch] gh/zklaus/16/orig -> origin/gh/zklaus/16/orig 2025-08-14T21:25:14.1228147Z * [new branch] gh/zklaus/17/base -> origin/gh/zklaus/17/base 2025-08-14T21:25:14.1228288Z * [new branch] gh/zklaus/17/head -> origin/gh/zklaus/17/head 2025-08-14T21:25:14.1228433Z * [new branch] gh/zklaus/17/orig -> origin/gh/zklaus/17/orig 2025-08-14T21:25:14.1228609Z * [new branch] gh/zklaus/18/base -> origin/gh/zklaus/18/base 2025-08-14T21:25:14.1228753Z * [new branch] gh/zklaus/18/head -> origin/gh/zklaus/18/head 2025-08-14T21:25:14.1228902Z * [new branch] gh/zklaus/18/orig -> origin/gh/zklaus/18/orig 2025-08-14T21:25:14.1229096Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-08-14T21:25:14.1229239Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-08-14T21:25:14.1229385Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-08-14T21:25:14.1229529Z * [new branch] gh/zklaus/7/base -> origin/gh/zklaus/7/base 2025-08-14T21:25:14.1229664Z * [new branch] gh/zklaus/7/head -> origin/gh/zklaus/7/head 2025-08-14T21:25:14.1229811Z * [new branch] gh/zklaus/7/orig -> origin/gh/zklaus/7/orig 2025-08-14T21:25:14.1229988Z * [new branch] gh/zklaus/9/base -> origin/gh/zklaus/9/base 2025-08-14T21:25:14.1230141Z * [new branch] gh/zklaus/9/head -> origin/gh/zklaus/9/head 2025-08-14T21:25:14.1230285Z * [new branch] gh/zklaus/9/orig -> origin/gh/zklaus/9/orig 2025-08-14T21:25:14.1230437Z * [new branch] gh/zou3519/1175/base -> origin/gh/zou3519/1175/base 2025-08-14T21:25:14.1230588Z * [new branch] gh/zou3519/1175/head -> origin/gh/zou3519/1175/head 2025-08-14T21:25:14.1230730Z * [new branch] gh/zou3519/1175/orig -> origin/gh/zou3519/1175/orig 2025-08-14T21:25:14.1230884Z * [new branch] gh/zou3519/1177/base -> origin/gh/zou3519/1177/base 2025-08-14T21:25:14.1231020Z * [new branch] gh/zou3519/1177/head -> origin/gh/zou3519/1177/head 2025-08-14T21:25:14.1231158Z * [new branch] gh/zou3519/1177/orig -> origin/gh/zou3519/1177/orig 2025-08-14T21:25:14.1231299Z * [new branch] gh/zou3519/1187/base -> origin/gh/zou3519/1187/base 2025-08-14T21:25:14.1231468Z * [new branch] gh/zou3519/1187/head -> origin/gh/zou3519/1187/head 2025-08-14T21:25:14.1231652Z * [new branch] gh/zou3519/1187/orig -> origin/gh/zou3519/1187/orig 2025-08-14T21:25:14.1231795Z * [new branch] gh/zou3519/1188/base -> origin/gh/zou3519/1188/base 2025-08-14T21:25:14.1231977Z * [new branch] gh/zou3519/1188/head -> origin/gh/zou3519/1188/head 2025-08-14T21:25:14.1232135Z * [new branch] gh/zou3519/1188/orig -> origin/gh/zou3519/1188/orig 2025-08-14T21:25:14.1232277Z * [new branch] gh/zou3519/1189/base -> origin/gh/zou3519/1189/base 2025-08-14T21:25:14.1232423Z * [new branch] gh/zou3519/1189/head -> origin/gh/zou3519/1189/head 2025-08-14T21:25:14.1232560Z * [new branch] gh/zou3519/1189/orig -> origin/gh/zou3519/1189/orig 2025-08-14T21:25:14.1237026Z * [new branch] gh/zou3519/1190/base -> origin/gh/zou3519/1190/base 2025-08-14T21:25:14.1237228Z * [new branch] gh/zou3519/1190/head -> origin/gh/zou3519/1190/head 2025-08-14T21:25:14.1237381Z * [new branch] gh/zou3519/1190/orig -> origin/gh/zou3519/1190/orig 2025-08-14T21:25:14.1237530Z * [new branch] gh/zou3519/1191/base -> origin/gh/zou3519/1191/base 2025-08-14T21:25:14.1237679Z * [new branch] gh/zou3519/1191/head -> origin/gh/zou3519/1191/head 2025-08-14T21:25:14.1237826Z * [new branch] gh/zou3519/1191/orig -> origin/gh/zou3519/1191/orig 2025-08-14T21:25:14.1238474Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-08-14T21:25:14.1238918Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-08-14T21:25:14.1239902Z * [new branch] gh/zpcore/10/base -> origin/gh/zpcore/10/base 2025-08-14T21:25:14.1240476Z * [new branch] gh/zpcore/10/head -> origin/gh/zpcore/10/head 2025-08-14T21:25:14.1241654Z * [new branch] gh/zpcore/10/orig -> origin/gh/zpcore/10/orig 2025-08-14T21:25:14.1246229Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-08-14T21:25:14.1246555Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-08-14T21:25:14.1246689Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-08-14T21:25:14.1246821Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-08-14T21:25:14.1246948Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-08-14T21:25:14.1247083Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-08-14T21:25:14.1247224Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-08-14T21:25:14.1247624Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-08-14T21:25:14.1248008Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-08-14T21:25:14.1248956Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-08-14T21:25:14.1249390Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-08-14T21:25:14.1252242Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-08-14T21:25:14.1252580Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-08-14T21:25:14.1252727Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-08-14T21:25:14.1252866Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-08-14T21:25:14.1253136Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-08-14T21:25:14.1253639Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-08-14T21:25:14.1254481Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-08-14T21:25:14.1259318Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-08-14T21:25:14.1259693Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-08-14T21:25:14.1259941Z * [new branch] gh/zpcore/9/head -> origin/gh/zpcore/9/head 2025-08-14T21:25:14.1260189Z * [new branch] gh/zpcore/9/orig -> origin/gh/zpcore/9/orig 2025-08-14T21:25:14.1260339Z * [new branch] google-main -> origin/google-main 2025-08-14T21:25:14.1261034Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-08-14T21:25:14.1261225Z * [new branch] guangyey/host_alloc -> origin/guangyey/host_alloc 2025-08-14T21:25:14.1261552Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-08-14T21:25:14.1262049Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-08-14T21:25:14.1269640Z * [new branch] haozhe/bf16-dynamic-shape -> origin/haozhe/bf16-dynamic-shape 2025-08-14T21:25:14.1271791Z * [new branch] hc_baseline -> origin/hc_baseline 2025-08-14T21:25:14.1271997Z * [new branch] headeronlyScalarType -> origin/headeronlyScalarType 2025-08-14T21:25:14.1272125Z * [new branch] hf_update -> origin/hf_update 2025-08-14T21:25:14.1272277Z * [new branch] hhh_decomp_mul -> origin/hhh_decomp_mul 2025-08-14T21:25:14.1272406Z * [new branch] hhh_rand -> origin/hhh_rand 2025-08-14T21:25:14.1272536Z * [new branch] hoy/mmsplitk -> origin/hoy/mmsplitk 2025-08-14T21:25:14.1272689Z * [new branch] hoy/triton-PR3973 -> origin/hoy/triton-PR3973 2025-08-14T21:25:14.1272909Z * [new branch] hoy/triton-coalescing-baseline -> origin/hoy/triton-coalescing-baseline 2025-08-14T21:25:14.1273239Z * [new branch] hoy/triton-coalescing-min -> origin/hoy/triton-coalescing-min 2025-08-14T21:25:14.1273414Z * [new branch] hoy/triton-coalescing-new -> origin/hoy/triton-coalescing-new 2025-08-14T21:25:14.1273574Z * [new branch] hoy/triton-coalescing-vec -> origin/hoy/triton-coalescing-vec 2025-08-14T21:25:14.1273727Z * [new branch] inductordecompfix -> origin/inductordecompfix 2025-08-14T21:25:14.1273846Z * [new branch] inline -> origin/inline 2025-08-14T21:25:14.1273971Z * [new branch] inlining -> origin/inlining 2025-08-14T21:25:14.1274120Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-08-14T21:25:14.1274246Z * [new branch] int8_sdpa -> origin/int8_sdpa 2025-08-14T21:25:14.1274696Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-08-14T21:25:14.1275879Z * [new branch] issue#58739 -> origin/issue#58739 2025-08-14T21:25:14.1276205Z * [new branch] issue-154849 -> origin/issue-154849 2025-08-14T21:25:14.1281516Z * [new branch] ivanov/cherry-pick-ckpt-fixes -> origin/ivanov/cherry-pick-ckpt-fixes 2025-08-14T21:25:14.1281914Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-08-14T21:25:14.1282230Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-08-14T21:25:14.1282436Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-08-14T21:25:14.1282678Z * [new branch] justinchu/attention-tests -> origin/justinchu/attention-tests 2025-08-14T21:25:14.1283354Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-08-14T21:25:14.1288562Z * [new branch] justinchuby/JitScalarType -> origin/justinchuby/JitScalarType 2025-08-14T21:25:14.1290786Z * [new branch] justinchuby/dynamo-true -> origin/justinchuby/dynamo-true 2025-08-14T21:25:14.1290975Z * [new branch] justinchuby/opset-20 -> origin/justinchuby/opset-20 2025-08-14T21:25:14.1291128Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-08-14T21:25:14.1291251Z * [new branch] kainan_test -> origin/kainan_test 2025-08-14T21:25:14.1291453Z * [new branch] leslie/enable_poc_reduction_fusion -> origin/leslie/enable_poc_reduction_fusion 2025-08-14T21:25:14.1291639Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-08-14T21:25:14.1291816Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-08-14T21:25:14.1292001Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-08-14T21:25:14.1292162Z * [new branch] liaoxuan/tags_issue -> origin/liaoxuan/tags_issue 2025-08-14T21:25:14.1292344Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-08-14T21:25:14.1292488Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-08-14T21:25:14.1293160Z * [new branch] lintbuilddocker -> origin/lintbuilddocker 2025-08-14T21:25:14.1293329Z * [new branch] llama4-stable -> origin/llama4-stable 2025-08-14T21:25:14.1293553Z * [new branch] logdetfix -> origin/logdetfix 2025-08-14T21:25:14.1294482Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-08-14T21:25:14.1298042Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-08-14T21:25:14.1298414Z * [new branch] lucaskabela/fix_157452 -> origin/lucaskabela/fix_157452 2025-08-14T21:25:14.1299000Z * [new branch] lucaskabela/fix_circular_import_158120 -> origin/lucaskabela/fix_circular_import_158120 2025-08-14T21:25:14.1299311Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-08-14T21:25:14.1299550Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-08-14T21:25:14.1299867Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-08-14T21:25:14.1300068Z * [new branch] lucaskabela/issue_120648 -> origin/lucaskabela/issue_120648 2025-08-14T21:25:14.1302359Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-08-14T21:25:14.1302762Z * [new branch] lucaskabela/registry_fix -> origin/lucaskabela/registry_fix 2025-08-14T21:25:14.1303143Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-08-14T21:25:14.1303453Z * [new branch] lucaskabela/type_guards -> origin/lucaskabela/type_guards 2025-08-14T21:25:14.1303660Z * [new branch] lucaskabela/typing-misc -> origin/lucaskabela/typing-misc 2025-08-14T21:25:14.1305160Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-08-14T21:25:14.1305462Z * [new branch] lucaskabela/typing_bytecode_analysis_transform -> origin/lucaskabela/typing_bytecode_analysis_transform 2025-08-14T21:25:14.1305952Z * [new branch] lucaskabela/typing_cache_files -> origin/lucaskabela/typing_cache_files 2025-08-14T21:25:14.1306628Z * [new branch] lucaskabela/typing_compile_autograd -> origin/lucaskabela/typing_compile_autograd 2025-08-14T21:25:14.1307489Z * [new branch] lucaskabela/typing_debug_utils.py -> origin/lucaskabela/typing_debug_utils.py 2025-08-14T21:25:14.1307990Z * [new branch] lucaskabela/typing_decorators -> origin/lucaskabela/typing_decorators 2025-08-14T21:25:14.1308622Z * [new branch] lucaskabela/typing_eval_frame -> origin/lucaskabela/typing_eval_frame 2025-08-14T21:25:14.1309680Z * [new branch] lucaskabela/typing_for_codegen -> origin/lucaskabela/typing_for_codegen 2025-08-14T21:25:14.1310149Z * [new branch] lucaskabela/typing_output_graph -> origin/lucaskabela/typing_output_graph 2025-08-14T21:25:14.1311093Z * [new branch] lucaskabela/typing_side_effects -> origin/lucaskabela/typing_side_effects 2025-08-14T21:25:14.1311698Z * [new branch] lucaskabela/typing_source_guard -> origin/lucaskabela/typing_source_guard 2025-08-14T21:25:14.1312750Z * [new branch] lucaskabela/typing_trace_rules -> origin/lucaskabela/typing_trace_rules 2025-08-14T21:25:14.1312950Z * [new branch] lucaskabela/typing_utils.py -> origin/lucaskabela/typing_utils.py 2025-08-14T21:25:14.1314196Z * [new branch] lucaskabela/typing_utils_improvements -> origin/lucaskabela/typing_utils_improvements 2025-08-14T21:25:14.1314644Z * [new branch] main -> origin/main 2025-08-14T21:25:14.1317003Z * [new branch] main-enable-b200-distributed-tests -> origin/main-enable-b200-distributed-tests 2025-08-14T21:25:14.1317164Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-08-14T21:25:14.1325668Z * [new branch] malfet-patch-10 -> origin/malfet-patch-10 2025-08-14T21:25:14.1325859Z * [new branch] malfet-patch-11 -> origin/malfet-patch-11 2025-08-14T21:25:14.1326004Z * [new branch] malfet-patch-13 -> origin/malfet-patch-13 2025-08-14T21:25:14.1326150Z * [new branch] malfet-patch-14 -> origin/malfet-patch-14 2025-08-14T21:25:14.1326316Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-08-14T21:25:14.1326677Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-08-14T21:25:14.1326821Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-08-14T21:25:14.1326962Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-08-14T21:25:14.1327103Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-08-14T21:25:14.1327238Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-08-14T21:25:14.1327369Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-08-14T21:25:14.1332403Z * [new branch] malfet-patch-9 -> origin/malfet-patch-9 2025-08-14T21:25:14.1332627Z * [new branch] malfet/delete-upsteam-cuda -> origin/malfet/delete-upsteam-cuda 2025-08-14T21:25:14.1332999Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-08-14T21:25:14.1333229Z * [new branch] manuel/fix_multidim_boolean_indexing -> origin/manuel/fix_multidim_boolean_indexing 2025-08-14T21:25:14.1333389Z * [new branch] manuel/np_empty_ellipsis -> origin/manuel/np_empty_ellipsis 2025-08-14T21:25:14.1333600Z * [new branch] manuel/test-ops-common-allow-mps -> origin/manuel/test-ops-common-allow-mps 2025-08-14T21:25:14.1333760Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-08-14T21:25:14.1333909Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-08-14T21:25:14.1334042Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-08-14T21:25:14.1334186Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-08-14T21:25:14.1334576Z * [new branch] mlazos/backup-test-branch -> origin/mlazos/backup-test-branch 2025-08-14T21:25:14.1338813Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-08-14T21:25:14.1344279Z * [new branch] mlazos/baseline -> origin/mlazos/baseline 2025-08-14T21:25:14.1344547Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-08-14T21:25:14.1345145Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-08-14T21:25:14.1345324Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-08-14T21:25:14.1345475Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-08-14T21:25:14.1345611Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-08-14T21:25:14.1345734Z * [new branch] mlazos/ck2 -> origin/mlazos/ck2 2025-08-14T21:25:14.1345911Z * [new branch] mlazos/combokernels -> origin/mlazos/combokernels 2025-08-14T21:25:14.1346067Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-08-14T21:25:14.1346234Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-08-14T21:25:14.1346443Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-08-14T21:25:14.1346601Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-08-14T21:25:14.1346782Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-08-14T21:25:14.1346923Z * [new branch] mlazos/data-gather -> origin/mlazos/data-gather 2025-08-14T21:25:14.1347134Z * [new branch] mlazos/data-ptrs2 -> origin/mlazos/data-ptrs2 2025-08-14T21:25:14.1347281Z * [new branch] mlazos/data-ptrs3 -> origin/mlazos/data-ptrs3 2025-08-14T21:25:14.1347455Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-08-14T21:25:14.1347727Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-08-14T21:25:14.1347865Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-08-14T21:25:14.1348145Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-08-14T21:25:14.1348338Z * [new branch] mlazos/disable-closures -> origin/mlazos/disable-closures 2025-08-14T21:25:14.1348503Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-08-14T21:25:14.1350680Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-08-14T21:25:14.1350870Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-08-14T21:25:14.1351003Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-08-14T21:25:14.1353243Z * [new branch] mlazos/exp_disable -> origin/mlazos/exp_disable 2025-08-14T21:25:14.1353523Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-08-14T21:25:14.1354045Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-08-14T21:25:14.1354219Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-08-14T21:25:14.1354538Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-08-14T21:25:14.1355689Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-08-14T21:25:14.1355893Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-08-14T21:25:14.1360790Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-08-14T21:25:14.1361147Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-08-14T21:25:14.1361558Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-08-14T21:25:14.1361846Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-08-14T21:25:14.1362006Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-08-14T21:25:14.1362239Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-08-14T21:25:14.1362902Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-08-14T21:25:14.1363244Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-08-14T21:25:14.1363388Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-08-14T21:25:14.1363669Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-08-14T21:25:14.1363926Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-08-14T21:25:14.1364982Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-08-14T21:25:14.1365199Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-08-14T21:25:14.1370824Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-08-14T21:25:14.1370994Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-08-14T21:25:14.1371117Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-08-14T21:25:14.1371239Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-08-14T21:25:14.1371366Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-08-14T21:25:14.1371485Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-08-14T21:25:14.1371599Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-08-14T21:25:14.1371722Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-08-14T21:25:14.1371851Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-08-14T21:25:14.1372156Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-08-14T21:25:14.1381170Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-08-14T21:25:14.1384487Z * [new branch] mlazos/hop-modes -> origin/mlazos/hop-modes 2025-08-14T21:25:14.1384842Z * [new branch] mlazos/init-per-param -> origin/mlazos/init-per-param 2025-08-14T21:25:14.1385097Z * [new branch] mlazos/init_per_param -> origin/mlazos/init_per_param 2025-08-14T21:25:14.1385265Z * [new branch] mlazos/less-guards -> origin/mlazos/less-guards 2025-08-14T21:25:14.1385451Z * [new branch] mlazos/lr-composibility -> origin/mlazos/lr-composibility 2025-08-14T21:25:14.1385686Z * [new branch] mlazos/main -> origin/mlazos/main 2025-08-14T21:25:14.1386388Z * [new branch] mlazos/main-test-enablement -> origin/mlazos/main-test-enablement 2025-08-14T21:25:14.1386718Z * [new branch] mlazos/main2 -> origin/mlazos/main2 2025-08-14T21:25:14.1386896Z * [new branch] mlazos/mcg -> origin/mlazos/mcg 2025-08-14T21:25:14.1387038Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-08-14T21:25:14.1387263Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-08-14T21:25:14.1387427Z * [new branch] mlazos/mlazos/ck2 -> origin/mlazos/mlazos/ck2 2025-08-14T21:25:14.1387698Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-08-14T21:25:14.1388335Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-08-14T21:25:14.1388536Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-08-14T21:25:14.1388839Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-08-14T21:25:14.1389007Z * [new branch] mlazos/more-tests -> origin/mlazos/more-tests 2025-08-14T21:25:14.1389140Z * [new branch] mlazos/nested-dc -> origin/mlazos/nested-dc 2025-08-14T21:25:14.1389274Z * [new branch] mlazos/no-cpp -> origin/mlazos/no-cpp 2025-08-14T21:25:14.1389489Z * [new branch] mlazos/no-init-group-handling -> origin/mlazos/no-init-group-handling 2025-08-14T21:25:14.1389615Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-08-14T21:25:14.1389771Z * [new branch] mlazos/opt-bench-exp2 -> origin/mlazos/opt-bench-exp2 2025-08-14T21:25:14.1390088Z * [new branch] mlazos/opt-incr -> origin/mlazos/opt-incr 2025-08-14T21:25:14.1390740Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-08-14T21:25:14.1390938Z * [new branch] mlazos/proxy-opt -> origin/mlazos/proxy-opt 2025-08-14T21:25:14.1392812Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-08-14T21:25:14.1393010Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-08-14T21:25:14.1393212Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-08-14T21:25:14.1393355Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-08-14T21:25:14.1393588Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-08-14T21:25:14.1393838Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-08-14T21:25:14.1394100Z * [new branch] mlazos/sub-param-fix -> origin/mlazos/sub-param-fix 2025-08-14T21:25:14.1395137Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-08-14T21:25:14.1395383Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-08-14T21:25:14.1400454Z * [new branch] mlazos/test -> origin/mlazos/test 2025-08-14T21:25:14.1405400Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-08-14T21:25:14.1405609Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-08-14T21:25:14.1405767Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-08-14T21:25:14.1405911Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-08-14T21:25:14.1406055Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-08-14T21:25:14.1406189Z * [new branch] mlazos/topo-fix -> origin/mlazos/topo-fix 2025-08-14T21:25:14.1406351Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-08-14T21:25:14.1406504Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-08-14T21:25:14.1406643Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-08-14T21:25:14.1406780Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-08-14T21:25:14.1406928Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-08-14T21:25:14.1407065Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-08-14T21:25:14.1407202Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-08-14T21:25:14.1407331Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-08-14T21:25:14.1407638Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-08-14T21:25:14.1408251Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-08-14T21:25:14.1408474Z * [new branch] modify-setupvllm -> origin/modify-setupvllm 2025-08-14T21:25:14.1409155Z * [new branch] move-theme-out-docker -> origin/move-theme-out-docker 2025-08-14T21:25:14.1412217Z * [new branch] mps-linear-1d -> origin/mps-linear-1d 2025-08-14T21:25:14.1412533Z * [new branch] msaroufim/be1 -> origin/msaroufim/be1 2025-08-14T21:25:14.1412800Z * [new branch] msaroufim/cn_path -> origin/msaroufim/cn_path 2025-08-14T21:25:14.1413063Z * [new branch] msaroufim/dtensorfusedadam -> origin/msaroufim/dtensorfusedadam 2025-08-14T21:25:14.1413298Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-08-14T21:25:14.1419048Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-08-14T21:25:14.1419381Z * [new branch] muon_dev -> origin/muon_dev 2025-08-14T21:25:14.1419668Z * [new branch] new-modifiy-setupvllm -> origin/new-modifiy-setupvllm 2025-08-14T21:25:14.1419895Z * [new branch] new-setupvllm -> origin/new-setupvllm 2025-08-14T21:25:14.1420066Z * [new branch] newtest-base -> origin/newtest-base 2025-08-14T21:25:14.1420205Z * [new branch] ngimel/cat_perf -> origin/ngimel/cat_perf 2025-08-14T21:25:14.1420361Z * [new branch] ngimel/cudamoduleload -> origin/ngimel/cudamoduleload 2025-08-14T21:25:14.1420559Z * [new branch] ngimel/fabric_driver_version -> origin/ngimel/fabric_driver_version 2025-08-14T21:25:14.1420697Z * [new branch] ngimel/fabric_symm -> origin/ngimel/fabric_symm 2025-08-14T21:25:14.1420826Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-08-14T21:25:14.1421118Z * [new branch] ngimel/grouped_mm_checks -> origin/ngimel/grouped_mm_checks 2025-08-14T21:25:14.1422358Z * [new branch] ngimel/guardfabric -> origin/ngimel/guardfabric 2025-08-14T21:25:14.1422590Z * [new branch] ngimel/index_None -> origin/ngimel/index_None 2025-08-14T21:25:14.1426403Z * [new branch] ngimel/modeguard -> origin/ngimel/modeguard 2025-08-14T21:25:14.1426582Z * [new branch] ngimel/multicast_fix -> origin/ngimel/multicast_fix 2025-08-14T21:25:14.1426741Z * [new branch] ngimel/unbind_multimem -> origin/ngimel/unbind_multimem 2025-08-14T21:25:14.1426860Z * [new branch] nightly -> origin/nightly 2025-08-14T21:25:14.1427022Z * [new branch] nmacchioni-patch-10 -> origin/nmacchioni-patch-10 2025-08-14T21:25:14.1427317Z * [new branch] nmacchioni-patch-7 -> origin/nmacchioni-patch-7 2025-08-14T21:25:14.1427806Z * [new branch] nmacchioni-patch-8 -> origin/nmacchioni-patch-8 2025-08-14T21:25:14.1429391Z * [new branch] nmacchioni-patch-9 -> origin/nmacchioni-patch-9 2025-08-14T21:25:14.1432024Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-08-14T21:25:14.1432677Z * [new branch] nweidia/enable-B200-inductor-nightly-ci -> origin/nweidia/enable-B200-inductor-nightly-ci 2025-08-14T21:25:14.1432835Z * [new branch] one-off -> origin/one-off 2025-08-14T21:25:14.1432994Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-08-14T21:25:14.1436337Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-08-14T21:25:14.1436644Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-08-14T21:25:14.1436827Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-08-14T21:25:14.1437176Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-08-14T21:25:14.1437345Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-08-14T21:25:14.1440820Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-08-14T21:25:14.1445621Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-08-14T21:25:14.1451139Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-08-14T21:25:14.1456752Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-08-14T21:25:14.1456927Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-08-14T21:25:14.1457076Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-08-14T21:25:14.1457209Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-08-14T21:25:14.1457392Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-08-14T21:25:14.1457534Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-08-14T21:25:14.1457676Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-08-14T21:25:14.1457823Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-08-14T21:25:14.1457960Z * [new branch] oulgen/fx_graph -> origin/oulgen/fx_graph 2025-08-14T21:25:14.1458099Z * [new branch] padded-tensor -> origin/padded-tensor 2025-08-14T21:25:14.1458235Z * [new branch] parallel_cat -> origin/parallel_cat 2025-08-14T21:25:14.1458350Z * [new branch] pca2 -> origin/pca2 2025-08-14T21:25:14.1458503Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-08-14T21:25:14.1458722Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-08-14T21:25:14.1459045Z * [new branch] pianpwk/dde_repeat_cat -> origin/pianpwk/dde_repeat_cat 2025-08-14T21:25:14.1459245Z * [new branch] pianpwk/draft_export_normalize -> origin/pianpwk/draft_export_normalize 2025-08-14T21:25:14.1459404Z * [new branch] pianpwk/dynamic_source_dim -> origin/pianpwk/dynamic_source_dim 2025-08-14T21:25:14.1459579Z * [new branch] pianpwk/invalidate_fake_memo -> origin/pianpwk/invalidate_fake_memo 2025-08-14T21:25:14.1459746Z * [new branch] pianpwk/lru_cache_bound_sympy -> origin/pianpwk/lru_cache_bound_sympy 2025-08-14T21:25:14.1459887Z * [new branch] pianpwk/max_1_strides -> origin/pianpwk/max_1_strides 2025-08-14T21:25:14.1460035Z * [new branch] pianpwk/nonzero_memo -> origin/pianpwk/nonzero_memo 2025-08-14T21:25:14.1460238Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-08-14T21:25:14.1460412Z * [new branch] pianpwk/oblivious_should_swap -> origin/pianpwk/oblivious_should_swap 2025-08-14T21:25:14.1460585Z * [new branch] pianpwk/oblivious_slice_forward -> origin/pianpwk/oblivious_slice_forward 2025-08-14T21:25:14.1460733Z * [new branch] pianpwk/oblivious_where -> origin/pianpwk/oblivious_where 2025-08-14T21:25:14.1461075Z * [new branch] pianpwk/param_static_pgo -> origin/pianpwk/param_static_pgo 2025-08-14T21:25:14.1461779Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-08-14T21:25:14.1461995Z * [new branch] pianpwk/remove_guard_fail_break -> origin/pianpwk/remove_guard_fail_break 2025-08-14T21:25:14.1462263Z * [new branch] pianpwk/slice_fresh_symbols -> origin/pianpwk/slice_fresh_symbols 2025-08-14T21:25:14.1462459Z * [new branch] pianpwk/sym_sym -> origin/pianpwk/sym_sym 2025-08-14T21:25:14.1462621Z * [new branch] pianpwk/test_slice_fake_impl -> origin/pianpwk/test_slice_fake_impl 2025-08-14T21:25:14.1462794Z * [new branch] pianpwk/unbacked_channels_last -> origin/pianpwk/unbacked_channels_last 2025-08-14T21:25:14.1462964Z * [new branch] pianpwk/unbacked_safe_conv1d -> origin/pianpwk/unbacked_safe_conv1d 2025-08-14T21:25:14.1463139Z * [new branch] pianpwk/unbacked_sdpa_flash -> origin/pianpwk/unbacked_sdpa_flash 2025-08-14T21:25:14.1463302Z * [new branch] pianpwk/unbacked_should_swap -> origin/pianpwk/unbacked_should_swap 2025-08-14T21:25:14.1463601Z * [new branch] pianpwk/unbacked_should_swap_2 -> origin/pianpwk/unbacked_should_swap_2 2025-08-14T21:25:14.1463794Z * [new branch] pianpwk/unbacked_slice_binding -> origin/pianpwk/unbacked_slice_binding 2025-08-14T21:25:14.1464086Z * [new branch] pianpwk/unbacked_slice_forward -> origin/pianpwk/unbacked_slice_forward 2025-08-14T21:25:14.1469361Z * [new branch] pianpwk/verbose_tensor_guards -> origin/pianpwk/verbose_tensor_guards 2025-08-14T21:25:14.1469714Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-08-14T21:25:14.1470010Z * [new branch] pianpwk/whitelist_optimizer -> origin/pianpwk/whitelist_optimizer 2025-08-14T21:25:14.1470269Z * [new branch] pin-torchao -> origin/pin-torchao 2025-08-14T21:25:14.1471540Z * [new branch] piz/fall_back_missing_0705 -> origin/piz/fall_back_missing_0705 2025-08-14T21:25:14.1472134Z * [new branch] piz/fall_back_missing_0716 -> origin/piz/fall_back_missing_0716 2025-08-14T21:25:14.1472336Z * [new branch] piz/fill_dist_cost_0702-3 -> origin/piz/fill_dist_cost_0702-3 2025-08-14T21:25:14.1472488Z * [new branch] piz/fill_dist_cost_0702-4 -> origin/piz/fill_dist_cost_0702-4 2025-08-14T21:25:14.1472660Z * [new branch] piz/fill_dist_cost_0702-5 -> origin/piz/fill_dist_cost_0702-5 2025-08-14T21:25:14.1472957Z * [new branch] piz/fix_sort_ -> origin/piz/fix_sort_ 2025-08-14T21:25:14.1473118Z * [new branch] piz/improve_scatter_0808 -> origin/piz/improve_scatter_0808 2025-08-14T21:25:14.1473265Z * [new branch] pool-separate -> origin/pool-separate 2025-08-14T21:25:14.1473387Z * [new branch] pr-156087 -> origin/pr-156087 2025-08-14T21:25:14.1473521Z * [new branch] pr/131860 -> origin/pr/131860 2025-08-14T21:25:14.1473665Z * [new branch] predispatch_to -> origin/predispatch_to 2025-08-14T21:25:14.1474233Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-08-14T21:25:14.1475358Z * [new branch] pt2e-cache-model-device -> origin/pt2e-cache-model-device 2025-08-14T21:25:14.1475662Z * [new branch] pull-latest-theme -> origin/pull-latest-theme 2025-08-14T21:25:14.1477195Z * [new branch] pyobjectslot -> origin/pyobjectslot 2025-08-14T21:25:14.1477519Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-08-14T21:25:14.1480562Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-08-14T21:25:14.1480880Z * [new branch] quint-bits -> origin/quint-bits 2025-08-14T21:25:14.1481024Z * [new branch] release/1.10 -> origin/release/1.10 2025-08-14T21:25:14.1481715Z * [new branch] release/1.11 -> origin/release/1.11 2025-08-14T21:25:14.1482480Z * [new branch] release/1.12 -> origin/release/1.12 2025-08-14T21:25:14.1486549Z * [new branch] release/1.13 -> origin/release/1.13 2025-08-14T21:25:14.1486847Z * [new branch] release/1.4 -> origin/release/1.4 2025-08-14T21:25:14.1486983Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-08-14T21:25:14.1487109Z * [new branch] release/1.5 -> origin/release/1.5 2025-08-14T21:25:14.1487223Z * [new branch] release/1.6 -> origin/release/1.6 2025-08-14T21:25:14.1492652Z * [new branch] release/1.7 -> origin/release/1.7 2025-08-14T21:25:14.1492803Z * [new branch] release/1.8 -> origin/release/1.8 2025-08-14T21:25:14.1492916Z * [new branch] release/1.9 -> origin/release/1.9 2025-08-14T21:25:14.1493032Z * [new branch] release/2.0 -> origin/release/2.0 2025-08-14T21:25:14.1493141Z * [new branch] release/2.1 -> origin/release/2.1 2025-08-14T21:25:14.1493249Z * [new branch] release/2.2 -> origin/release/2.2 2025-08-14T21:25:14.1493379Z * [new branch] release/2.3 -> origin/release/2.3 2025-08-14T21:25:14.1499541Z * [new branch] release/2.4 -> origin/release/2.4 2025-08-14T21:25:14.1499721Z * [new branch] release/2.5 -> origin/release/2.5 2025-08-14T21:25:14.1499885Z * [new branch] release/2.6 -> origin/release/2.6 2025-08-14T21:25:14.1500037Z * [new branch] release/2.7 -> origin/release/2.7 2025-08-14T21:25:14.1500168Z * [new branch] release/2.8 -> origin/release/2.8 2025-08-14T21:25:14.1500293Z * [new branch] release_notes -> origin/release_notes 2025-08-14T21:25:14.1500497Z * [new branch] remove-actionable-label -> origin/remove-actionable-label 2025-08-14T21:25:14.1500628Z * [new branch] remove-ao -> origin/remove-ao 2025-08-14T21:25:14.1500885Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-08-14T21:25:14.1501297Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-08-14T21:25:14.1501544Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-08-14T21:25:14.1504190Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-08-14T21:25:14.1504432Z * [new branch] replace-pytorch-labs-20250812-204125 -> origin/replace-pytorch-labs-20250812-204125 2025-08-14T21:25:14.1504682Z * [new branch] replace-pytorch-labs-20250812-205624 -> origin/replace-pytorch-labs-20250812-205624 2025-08-14T21:25:14.1504973Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-08-14T21:25:14.1505216Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-08-14T21:25:14.1509160Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-08-14T21:25:14.1509610Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-08-14T21:25:14.1509858Z * [new branch] revert-direct-updates -> origin/revert-direct-updates 2025-08-14T21:25:14.1510078Z * [new branch] rocm-monitoring -> origin/rocm-monitoring 2025-08-14T21:25:14.1510350Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-08-14T21:25:14.1510644Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-08-14T21:25:14.1510813Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-08-14T21:25:14.1511136Z * [new branch] rzou/njt -> origin/rzou/njt 2025-08-14T21:25:14.1511410Z * [new branch] rzou/operator -> origin/rzou/operator 2025-08-14T21:25:14.1511558Z * [new branch] rzou/pca -> origin/rzou/pca 2025-08-14T21:25:14.1511788Z * [new branch] rzou/pipe_split -> origin/rzou/pipe_split 2025-08-14T21:25:14.1511945Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-08-14T21:25:14.1512164Z * [new branch] rzou/setup_context -> origin/rzou/setup_context 2025-08-14T21:25:14.1513573Z * [new branch] sanchitintel/refactor_aten_int8_woq_gemm -> origin/sanchitintel/refactor_aten_int8_woq_gemm 2025-08-14T21:25:14.1513936Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-08-14T21:25:14.1515061Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-08-14T21:25:14.1515491Z * [new branch] save -> origin/save 2025-08-14T21:25:14.1517117Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-08-14T21:25:14.1520154Z * [new branch] seemethere-patch-1 -> origin/seemethere-patch-1 2025-08-14T21:25:14.1520426Z * [new branch] setup-torchci -> origin/setup-torchci 2025-08-14T21:25:14.1526055Z * [new branch] setupvllm -> origin/setupvllm 2025-08-14T21:25:14.1531676Z * [new branch] share_and_pin_fork -> origin/share_and_pin_fork 2025-08-14T21:25:14.1536978Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-08-14T21:25:14.1542507Z * [new branch] shikaili_fp8_allgather -> origin/shikaili_fp8_allgather 2025-08-14T21:25:14.1547443Z * [new branch] shoumikhin-patch-12 -> origin/shoumikhin-patch-12 2025-08-14T21:25:14.1549475Z * [new branch] simplify-fq-per-channel -> origin/simplify-fq-per-channel 2025-08-14T21:25:14.1549644Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-08-14T21:25:14.1549788Z * [new branch] sqzhang/flight4 -> origin/sqzhang/flight4 2025-08-14T21:25:14.1549940Z * [new branch] sqzhang/flight4plus -> origin/sqzhang/flight4plus 2025-08-14T21:25:14.1550099Z * [new branch] sraikund/record_funct_test -> origin/sraikund/record_funct_test 2025-08-14T21:25:14.1550228Z * [new branch] sraikund16/test -> origin/sraikund16/test 2025-08-14T21:25:14.1550421Z * [new branch] stablize-compilation-time -> origin/stablize-compilation-time 2025-08-14T21:25:14.1550570Z * [new branch] standalone-templates -> origin/standalone-templates 2025-08-14T21:25:14.1550734Z * [new branch] standalone_package_weights -> origin/standalone_package_weights 2025-08-14T21:25:14.1550876Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-08-14T21:25:14.1551010Z * [new branch] step2vllmsetup -> origin/step2vllmsetup 2025-08-14T21:25:14.1551130Z * [new branch] subgraph_fuse -> origin/subgraph_fuse 2025-08-14T21:25:14.1551288Z * [new branch] support-uv-in-collect_env -> origin/support-uv-in-collect_env 2025-08-14T21:25:14.1551443Z * [new branch] suryasub/fix-nccl-hang -> origin/suryasub/fix-nccl-hang 2025-08-14T21:25:14.1551559Z * [new branch] sve-poc -> origin/sve-poc 2025-08-14T21:25:14.1551695Z * [new branch] svekars-patch-1 -> origin/svekars-patch-1 2025-08-14T21:25:14.1551839Z * [new branch] svekars-patch-2 -> origin/svekars-patch-2 2025-08-14T21:25:14.1552016Z * [new branch] switch-bn -> origin/switch-bn 2025-08-14T21:25:14.1552194Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-08-14T21:25:14.1552374Z * [new branch] tenpercent/ck_inductor_gfx950 -> origin/tenpercent/ck_inductor_gfx950 2025-08-14T21:25:14.1552522Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-08-14T21:25:14.1552727Z * [new branch] test-half-migration-internally -> origin/test-half-migration-internally 2025-08-14T21:25:14.1552867Z * [new branch] test-internal-et -> origin/test-internal-et 2025-08-14T21:25:14.1553028Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-08-14T21:25:14.1553211Z * [new branch] test-myst-markdown-docstring -> origin/test-myst-markdown-docstring 2025-08-14T21:25:14.1553332Z * [new branch] test-old -> origin/test-old 2025-08-14T21:25:14.1553535Z * [new branch] test-vec-migration-internally -> origin/test-vec-migration-internally 2025-08-14T21:25:14.1553668Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-08-14T21:25:14.1553802Z * [new branch] test/inductor -> origin/test/inductor 2025-08-14T21:25:14.1553948Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-08-14T21:25:14.1554076Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-08-14T21:25:14.1554238Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-08-14T21:25:14.1554390Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-08-14T21:25:14.1554518Z * [new branch] trackMonitor -> origin/trackMonitor 2025-08-14T21:25:14.1554666Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-08-14T21:25:14.1554827Z * [new branch] tree_vec_base -> origin/tree_vec_base 2025-08-14T21:25:14.1554963Z * [new branch] triton-update -> origin/triton-update 2025-08-14T21:25:14.1555085Z * [new branch] triton_kernel -> origin/triton_kernel 2025-08-14T21:25:14.1555224Z * [new branch] triton_kernel_perf -> origin/triton_kernel_perf 2025-08-14T21:25:14.1555367Z * [new branch] try-runllm -> origin/try-runllm 2025-08-14T21:25:14.1555489Z * [new branch] type_dec -> origin/type_dec 2025-08-14T21:25:14.1555669Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-08-14T21:25:14.1555920Z * [new branch] update-audio-commit-hash/16307312222-1661-1 -> origin/update-audio-commit-hash/16307312222-1661-1 2025-08-14T21:25:14.1556367Z * [new branch] update-audio-commit-hash/16431348808-1673-1 -> origin/update-audio-commit-hash/16431348808-1673-1 2025-08-14T21:25:14.1556617Z * [new branch] update-audio-commit-hash/16510774365-1683-1 -> origin/update-audio-commit-hash/16510774365-1683-1 2025-08-14T21:25:14.1556845Z * [new branch] update-audio-commit-hash/16583472358-1693-1 -> origin/update-audio-commit-hash/16583472358-1693-1 2025-08-14T21:25:14.1557084Z * [new branch] update-audio-commit-hash/16663082088-1700-1 -> origin/update-audio-commit-hash/16663082088-1700-1 2025-08-14T21:25:14.1557316Z * [new branch] update-audio-commit-hash/16737365217-1704-1 -> origin/update-audio-commit-hash/16737365217-1704-1 2025-08-14T21:25:14.1557949Z * [new branch] update-audio-commit-hash/16791960928-1711-1 -> origin/update-audio-commit-hash/16791960928-1711-1 2025-08-14T21:25:14.1560758Z * [new branch] update-audio-commit-hash/16818882925-1712-1 -> origin/update-audio-commit-hash/16818882925-1712-1 2025-08-14T21:25:14.1561386Z * [new branch] update-audio-commit-hash/16895560422-1720-1 -> origin/update-audio-commit-hash/16895560422-1720-1 2025-08-14T21:25:14.1561803Z * [new branch] update-audio-commit-hash/16924174496-1738-1 -> origin/update-audio-commit-hash/16924174496-1738-1 2025-08-14T21:25:14.1562134Z * [new branch] update-dynamic-shapes-doc -> origin/update-dynamic-shapes-doc 2025-08-14T21:25:14.1562830Z * [new branch] update-executorch-commit-hash/15694981040-1626-1 -> origin/update-executorch-commit-hash/15694981040-1626-1 2025-08-14T21:25:14.1563133Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-08-14T21:25:14.1563725Z * [new branch] update-vision-commit-hash/15336342773-1607-1 -> origin/update-vision-commit-hash/15336342773-1607-1 2025-08-14T21:25:14.1565233Z * [new branch] update-vllm-commit-hash/16431348808-1673-1 -> origin/update-vllm-commit-hash/16431348808-1673-1 2025-08-14T21:25:14.1565644Z * [new branch] update-vllm-commit-hash/16484773233-1682-1 -> origin/update-vllm-commit-hash/16484773233-1682-1 2025-08-14T21:25:14.1565892Z * [new branch] update-vllm-commit-hash/16510774365-1683-1 -> origin/update-vllm-commit-hash/16510774365-1683-1 2025-08-14T21:25:14.1566411Z * [new branch] update-vllm-commit-hash/16534031105-1684-1 -> origin/update-vllm-commit-hash/16534031105-1684-1 2025-08-14T21:25:14.1568127Z * [new branch] update-vllm-commit-hash/16545403308-1687-1 -> origin/update-vllm-commit-hash/16545403308-1687-1 2025-08-14T21:25:14.1568578Z * [new branch] update-vllm-commit-hash/16557202787-1688-1 -> origin/update-vllm-commit-hash/16557202787-1688-1 2025-08-14T21:25:14.1568968Z * [new branch] update-vllm-commit-hash/16583472358-1693-1 -> origin/update-vllm-commit-hash/16583472358-1693-1 2025-08-14T21:25:14.1569366Z * [new branch] update-vllm-commit-hash/16663082088-1700-1 -> origin/update-vllm-commit-hash/16663082088-1700-1 2025-08-14T21:25:14.1571079Z * [new branch] update-vllm-commit-hash/16737365217-1704-1 -> origin/update-vllm-commit-hash/16737365217-1704-1 2025-08-14T21:25:14.1571499Z * [new branch] update-vllm-commit-hash/16843157111-1713-1 -> origin/update-vllm-commit-hash/16843157111-1713-1 2025-08-14T21:25:14.1571838Z * [new branch] update-vllm-commit-hash/16855312394-1714-1 -> origin/update-vllm-commit-hash/16855312394-1714-1 2025-08-14T21:25:14.1572138Z * [new branch] update-vllm-commit-hash/16924174496-1738-1 -> origin/update-vllm-commit-hash/16924174496-1738-1 2025-08-14T21:25:14.1572438Z * [new branch] update-vllm-commit-hash/16952608705-1745-1 -> origin/update-vllm-commit-hash/16952608705-1745-1 2025-08-14T21:25:14.1573395Z * [new branch] update-xla-commit-hash/16260974441-194-1 -> origin/update-xla-commit-hash/16260974441-194-1 2025-08-14T21:25:14.1573842Z * [new branch] update-xla-commit-hash/16717126778-197-1 -> origin/update-xla-commit-hash/16717126778-197-1 2025-08-14T21:25:14.1574370Z * [new branch] update-xla-commit-hash/16873912760-198-1 -> origin/update-xla-commit-hash/16873912760-198-1 2025-08-14T21:25:14.1576912Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-08-14T21:25:14.1577257Z * [new branch] update_executorch_pin -> origin/update_executorch_pin 2025-08-14T21:25:14.1577504Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-08-14T21:25:14.1577741Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-08-14T21:25:14.1577985Z * [new branch] update_slow_tests_1752478971 -> origin/update_slow_tests_1752478971 2025-08-14T21:25:14.1578753Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-08-14T21:25:14.1581650Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-08-14T21:25:14.1582001Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-08-14T21:25:14.1582218Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-08-14T21:25:14.1582349Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-08-14T21:25:14.1584005Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-08-14T21:25:14.1584295Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-08-14T21:25:14.1586434Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-08-14T21:25:14.1586738Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-08-14T21:25:14.1586866Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-08-14T21:25:14.1587093Z * [new branch] validate_fn -> origin/validate_fn 2025-08-14T21:25:14.1588378Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-08-14T21:25:14.1588572Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-08-14T21:25:14.1591424Z * [new branch] viable/strict -> origin/viable/strict 2025-08-14T21:25:14.1591587Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-08-14T21:25:14.1591706Z * [new branch] vllmpin -> origin/vllmpin 2025-08-14T21:25:14.1591827Z * [new branch] vllmpintest -> origin/vllmpintest 2025-08-14T21:25:14.1593970Z * [new branch] wdvr-patch-1 -> origin/wdvr-patch-1 2025-08-14T21:25:14.1594142Z * [new branch] wdvr-patch-2 -> origin/wdvr-patch-2 2025-08-14T21:25:14.1594331Z * [new branch] wdvr/conda_devcontainer -> origin/wdvr/conda_devcontainer 2025-08-14T21:25:14.1595825Z * [new branch] wdvr/fix_logging_test -> origin/wdvr/fix_logging_test 2025-08-14T21:25:14.1595966Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-08-14T21:25:14.1596346Z * [new branch] weight_sharing_cpp -> origin/weight_sharing_cpp 2025-08-14T21:25:14.1597646Z * [new branch] whc/flight -> origin/whc/flight 2025-08-14T21:25:14.1597995Z * [new branch] whc/flight4 -> origin/whc/flight4 2025-08-14T21:25:14.1600249Z * [new branch] whc/flight51 -> origin/whc/flight51 2025-08-14T21:25:14.1600408Z * [new branch] whc/flight53 -> origin/whc/flight53 2025-08-14T21:25:14.1600542Z * [new branch] whc/p2phang -> origin/whc/p2phang 2025-08-14T21:25:14.1600725Z * [new branch] whc/stage2 -> origin/whc/stage2 2025-08-14T21:25:14.1602605Z * [new branch] whc/uneven -> origin/whc/uneven 2025-08-14T21:25:14.1602779Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-08-14T21:25:14.1603180Z * [new branch] win_warnings -> origin/win_warnings 2025-08-14T21:25:14.1603726Z * [new branch] workonoldcommit -> origin/workonoldcommit 2025-08-14T21:25:14.1605017Z * [new branch] wwen/programming-model-2.8 -> origin/wwen/programming-model-2.8 2025-08-14T21:25:14.1605517Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-08-14T21:25:14.1606759Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-08-14T21:25:14.1607034Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-08-14T21:25:14.1607600Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-08-14T21:25:14.1607977Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-08-14T21:25:14.1609054Z * [new branch] xmfan/ca_api -> origin/xmfan/ca_api 2025-08-14T21:25:14.1609182Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-08-14T21:25:14.1614502Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-08-14T21:25:14.1614874Z * [new branch] xmfan/ca_cudagraphs -> origin/xmfan/ca_cudagraphs 2025-08-14T21:25:14.1615903Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-08-14T21:25:14.1616146Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-08-14T21:25:14.1617154Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-08-14T21:25:14.1617417Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-08-14T21:25:14.1618374Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-08-14T21:25:14.1618544Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-08-14T21:25:14.1619609Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-08-14T21:25:14.1620038Z * [new branch] xmfan/ca_mem_base -> origin/xmfan/ca_mem_base 2025-08-14T21:25:14.1620887Z * [new branch] xmfan/ca_mem_fix -> origin/xmfan/ca_mem_fix 2025-08-14T21:25:14.1623959Z * [new branch] xmfan/ca_memory_fix -> origin/xmfan/ca_memory_fix 2025-08-14T21:25:14.1624128Z * [new branch] xmfan/ca_memory_fix_rebased -> origin/xmfan/ca_memory_fix_rebased 2025-08-14T21:25:14.1624283Z * [new branch] xmfan/ca_memory_fix_rebased2 -> origin/xmfan/ca_memory_fix_rebased2 2025-08-14T21:25:14.1624436Z * [new branch] xmfan/ca_move_to_cuda -> origin/xmfan/ca_move_to_cuda 2025-08-14T21:25:14.1624747Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-08-14T21:25:14.1625014Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-08-14T21:25:14.1626677Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-08-14T21:25:14.1626862Z * [new branch] xmfan/ca_scalar -> origin/xmfan/ca_scalar 2025-08-14T21:25:14.1627321Z * [new branch] xmfan/ca_subclass_mem_fix -> origin/xmfan/ca_subclass_mem_fix 2025-08-14T21:25:14.1628670Z * [new branch] xmfan/ca_warm_mem -> origin/xmfan/ca_warm_mem 2025-08-14T21:25:14.1628848Z * [new branch] xmfan/ca_warm_mem_base -> origin/xmfan/ca_warm_mem_base 2025-08-14T21:25:14.1629239Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-08-14T21:25:14.1632289Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-08-14T21:25:14.1632717Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-08-14T21:25:14.1632861Z * [new branch] xmfan/cacu_may27 -> origin/xmfan/cacu_may27 2025-08-14T21:25:14.1633021Z * [new branch] xmfan/circular_dep -> origin/xmfan/circular_dep 2025-08-14T21:25:14.1633218Z * [new branch] xmfan/compiled_autograd_feb_29 -> origin/xmfan/compiled_autograd_feb_29 2025-08-14T21:25:14.1633469Z * [new branch] xmfan/compiled_autograd_graph_breaks -> origin/xmfan/compiled_autograd_graph_breaks 2025-08-14T21:25:14.1634117Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-08-14T21:25:14.1635541Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-08-14T21:25:14.1636002Z * [new branch] xmfan/issue_123374 -> origin/xmfan/issue_123374 2025-08-14T21:25:14.1636988Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-08-14T21:25:14.1640479Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-08-14T21:25:14.1640691Z * [new branch] xmfan/segfault_test -> origin/xmfan/segfault_test 2025-08-14T21:25:14.1640851Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-08-14T21:25:14.1640992Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-08-14T21:25:14.1641113Z * [new branch] xmfan/test -> origin/xmfan/test 2025-08-14T21:25:14.1641303Z * [new branch] y-do-we-have-7-build-systems -> origin/y-do-we-have-7-build-systems 2025-08-14T21:25:14.1642849Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-08-14T21:25:14.1643056Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-08-14T21:25:14.1647628Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-08-14T21:25:14.1648228Z * [new branch] yihan_quantization -> origin/yihan_quantization 2025-08-14T21:25:14.1648557Z * [new branch] yiming/add_nativert_benchmark -> origin/yiming/add_nativert_benchmark 2025-08-14T21:25:14.1648771Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-08-14T21:25:14.1648981Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-08-14T21:25:14.1649171Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-08-14T21:25:14.1649307Z * [new branch] zainr/fixlint -> origin/zainr/fixlint 2025-08-14T21:25:14.1656122Z * [new branch] zainr/git-push-v2 -> origin/zainr/git-push-v2 2025-08-14T21:25:14.1661056Z * [new branch] zainr/lint-py3.9 -> origin/zainr/lint-py3.9 2025-08-14T21:25:14.1665778Z * [new branch] zainr/mypy15-claude -> origin/zainr/mypy15-claude 2025-08-14T21:25:14.1670928Z * [new branch] zainr/pre-push-hooks -> origin/zainr/pre-push-hooks 2025-08-14T21:25:14.1672858Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-08-14T21:25:14.1673128Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-08-14T21:25:14.1673355Z * [new branch] zainr/unstable -> origin/zainr/unstable 2025-08-14T21:25:14.1673506Z * [new branch] zainr/unstable-xla -> origin/zainr/unstable-xla 2025-08-14T21:25:14.1673661Z * [new branch] zainr/uv-pip-fix -> origin/zainr/uv-pip-fix 2025-08-14T21:25:14.1673950Z * [new branch] zainr/vs-aarch64 -> origin/zainr/vs-aarch64 2025-08-14T21:25:14.1674117Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-08-14T21:25:14.1674354Z * [new branch] zb2p -> origin/zb2p 2025-08-14T21:25:14.1674517Z * [new branch] zdevito-patch-1 -> origin/zdevito-patch-1 2025-08-14T21:25:14.1674791Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-08-14T21:25:14.1674944Z * [new branch] zhxchen17/nativert/0 -> origin/zhxchen17/nativert/0 2025-08-14T21:25:14.1675237Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-08-14T21:25:14.1675408Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-08-14T21:25:14.1676282Z * [new branch] zxiiro/bazel -> origin/zxiiro/bazel 2025-08-14T21:25:14.1676700Z * [new branch] zxiiro/get-hardware -> origin/zxiiro/get-hardware 2025-08-14T21:25:14.1676881Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-08-14T21:25:14.1677021Z * [new branch] zxiiro/test -> origin/zxiiro/test 2025-08-14T21:25:14.1677351Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-08-14T21:25:14.1677477Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-08-14T21:25:14.1677599Z * [new tag] ciflow/binaries/138996 -> ciflow/binaries/138996 2025-08-14T21:25:14.1677717Z * [new tag] ciflow/binaries/143959 -> ciflow/binaries/143959 2025-08-14T21:25:14.1677835Z * [new tag] ciflow/binaries/154595 -> ciflow/binaries/154595 2025-08-14T21:25:14.1677949Z * [new tag] ciflow/binaries/156049 -> ciflow/binaries/156049 2025-08-14T21:25:14.1678064Z * [new tag] ciflow/binaries/156712 -> ciflow/binaries/156712 2025-08-14T21:25:14.1678181Z * [new tag] ciflow/binaries/157432 -> ciflow/binaries/157432 2025-08-14T21:25:14.1678293Z * [new tag] ciflow/binaries/157685 -> ciflow/binaries/157685 2025-08-14T21:25:14.1678409Z * [new tag] ciflow/binaries/157689 -> ciflow/binaries/157689 2025-08-14T21:25:14.1678520Z * [new tag] ciflow/binaries/158104 -> ciflow/binaries/158104 2025-08-14T21:25:14.1678630Z * [new tag] ciflow/binaries/158623 -> ciflow/binaries/158623 2025-08-14T21:25:14.1678748Z * [new tag] ciflow/binaries/159827 -> ciflow/binaries/159827 2025-08-14T21:25:14.1678860Z * [new tag] ciflow/binaries/159869 -> ciflow/binaries/159869 2025-08-14T21:25:14.1678979Z * [new tag] ciflow/binaries/160593 -> ciflow/binaries/160593 2025-08-14T21:25:14.1679175Z * [new tag] ciflow/binaries_libtorch/143959 -> ciflow/binaries_libtorch/143959 2025-08-14T21:25:14.1679316Z * [new tag] ciflow/binaries_libtorch/156049 -> ciflow/binaries_libtorch/156049 2025-08-14T21:25:14.1679460Z * [new tag] ciflow/binaries_libtorch/157432 -> ciflow/binaries_libtorch/157432 2025-08-14T21:25:14.1679591Z * [new tag] ciflow/binaries_wheel/143959 -> ciflow/binaries_wheel/143959 2025-08-14T21:25:14.1679719Z * [new tag] ciflow/binaries_wheel/156049 -> ciflow/binaries_wheel/156049 2025-08-14T21:25:14.1679856Z * [new tag] ciflow/binaries_wheel/157432 -> ciflow/binaries_wheel/157432 2025-08-14T21:25:14.1679984Z * [new tag] ciflow/binaries_wheel/158733 -> ciflow/binaries_wheel/158733 2025-08-14T21:25:14.1680117Z * [new tag] ciflow/binaries_wheel/160301 -> ciflow/binaries_wheel/160301 2025-08-14T21:25:14.1680253Z * [new tag] ciflow/binaries_wheel/160496 -> ciflow/binaries_wheel/160496 2025-08-14T21:25:14.1680405Z * [new tag] ciflow/h100-distributed/156703 -> ciflow/h100-distributed/156703 2025-08-14T21:25:14.1680549Z * [new tag] ciflow/h100-symm-mem/151845 -> ciflow/h100-symm-mem/151845 2025-08-14T21:25:14.1680676Z * [new tag] ciflow/h100-symm-mem/155923 -> ciflow/h100-symm-mem/155923 2025-08-14T21:25:14.1680807Z * [new tag] ciflow/h100-symm-mem/157635 -> ciflow/h100-symm-mem/157635 2025-08-14T21:25:14.1680929Z * [new tag] ciflow/h100-symm-mem/159118 -> ciflow/h100-symm-mem/159118 2025-08-14T21:25:14.1681051Z * [new tag] ciflow/h100-symm-mem/159562 -> ciflow/h100-symm-mem/159562 2025-08-14T21:25:14.1681189Z * [new tag] ciflow/h100-symm-mem/159889 -> ciflow/h100-symm-mem/159889 2025-08-14T21:25:14.1681300Z * [new tag] ciflow/h100/159158 -> ciflow/h100/159158 2025-08-14T21:25:14.1681752Z * [new tag] ciflow/h100/160450 -> ciflow/h100/160450 2025-08-14T21:25:14.1682168Z * [new tag] ciflow/h100/160480 -> ciflow/h100/160480 2025-08-14T21:25:14.1684267Z * [new tag] ciflow/h100/160614 -> ciflow/h100/160614 2025-08-14T21:25:14.1684715Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/151845 -> ciflow/inductor-perf-test-nightly-rocm/151845 2025-08-14T21:25:14.1685039Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/160538 -> ciflow/inductor-perf-test-nightly-rocm/160538 2025-08-14T21:25:14.1685414Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/156599 -> ciflow/inductor-perf-test-nightly-x86-zen/156599 2025-08-14T21:25:14.1685675Z * [new tag] ciflow/inductor-periodic/160406 -> ciflow/inductor-periodic/160406 2025-08-14T21:25:14.1685852Z * [new tag] ciflow/inductor-periodic/160538 -> ciflow/inductor-periodic/160538 2025-08-14T21:25:14.1686192Z * [new tag] ciflow/inductor-rocm/151845 -> ciflow/inductor-rocm/151845 2025-08-14T21:25:14.1686615Z * [new tag] ciflow/inductor-rocm/159158 -> ciflow/inductor-rocm/159158 2025-08-14T21:25:14.1687872Z * [new tag] ciflow/inductor-rocm/160073 -> ciflow/inductor-rocm/160073 2025-08-14T21:25:14.1688192Z * [new tag] ciflow/inductor-rocm/160538 -> ciflow/inductor-rocm/160538 2025-08-14T21:25:14.1688347Z * [new tag] ciflow/inductor/134881 -> ciflow/inductor/134881 2025-08-14T21:25:14.1688569Z * [new tag] ciflow/inductor/137400 -> ciflow/inductor/137400 2025-08-14T21:25:14.1691547Z * [new tag] ciflow/inductor/144516 -> ciflow/inductor/144516 2025-08-14T21:25:14.1691853Z * [new tag] ciflow/inductor/146506 -> ciflow/inductor/146506 2025-08-14T21:25:14.1692022Z * [new tag] ciflow/inductor/147360 -> ciflow/inductor/147360 2025-08-14T21:25:14.1692317Z * [new tag] ciflow/inductor/147990 -> ciflow/inductor/147990 2025-08-14T21:25:14.1692585Z * [new tag] ciflow/inductor/148180 -> ciflow/inductor/148180 2025-08-14T21:25:14.1692723Z * [new tag] ciflow/inductor/148328 -> ciflow/inductor/148328 2025-08-14T21:25:14.1693298Z * [new tag] ciflow/inductor/148484 -> ciflow/inductor/148484 2025-08-14T21:25:14.1693463Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-08-14T21:25:14.1693582Z * [new tag] ciflow/inductor/150302 -> ciflow/inductor/150302 2025-08-14T21:25:14.1693701Z * [new tag] ciflow/inductor/151845 -> ciflow/inductor/151845 2025-08-14T21:25:14.1693832Z * [new tag] ciflow/inductor/152198 -> ciflow/inductor/152198 2025-08-14T21:25:14.1698910Z * [new tag] ciflow/inductor/152624 -> ciflow/inductor/152624 2025-08-14T21:25:14.1699245Z * [new tag] ciflow/inductor/153966 -> ciflow/inductor/153966 2025-08-14T21:25:14.1699497Z * [new tag] ciflow/inductor/154193 -> ciflow/inductor/154193 2025-08-14T21:25:14.1699638Z * [new tag] ciflow/inductor/154650 -> ciflow/inductor/154650 2025-08-14T21:25:14.1699841Z * [new tag] ciflow/inductor/154694 -> ciflow/inductor/154694 2025-08-14T21:25:14.1699977Z * [new tag] ciflow/inductor/155072 -> ciflow/inductor/155072 2025-08-14T21:25:14.1700101Z * [new tag] ciflow/inductor/155152 -> ciflow/inductor/155152 2025-08-14T21:25:14.1700346Z * [new tag] ciflow/inductor/155153 -> ciflow/inductor/155153 2025-08-14T21:25:14.1700488Z * [new tag] ciflow/inductor/155154 -> ciflow/inductor/155154 2025-08-14T21:25:14.1700687Z * [new tag] ciflow/inductor/155501 -> ciflow/inductor/155501 2025-08-14T21:25:14.1700983Z * [new tag] ciflow/inductor/155502 -> ciflow/inductor/155502 2025-08-14T21:25:14.1701112Z * [new tag] ciflow/inductor/155503 -> ciflow/inductor/155503 2025-08-14T21:25:14.1703732Z * [new tag] ciflow/inductor/155504 -> ciflow/inductor/155504 2025-08-14T21:25:14.1703880Z * [new tag] ciflow/inductor/155557 -> ciflow/inductor/155557 2025-08-14T21:25:14.1704083Z * [new tag] ciflow/inductor/155608 -> ciflow/inductor/155608 2025-08-14T21:25:14.1704222Z * [new tag] ciflow/inductor/155923 -> ciflow/inductor/155923 2025-08-14T21:25:14.1704413Z * [new tag] ciflow/inductor/155928 -> ciflow/inductor/155928 2025-08-14T21:25:14.1704546Z * [new tag] ciflow/inductor/155958 -> ciflow/inductor/155958 2025-08-14T21:25:14.1704745Z * [new tag] ciflow/inductor/156049 -> ciflow/inductor/156049 2025-08-14T21:25:14.1704881Z * [new tag] ciflow/inductor/156851 -> ciflow/inductor/156851 2025-08-14T21:25:14.1705085Z * [new tag] ciflow/inductor/156967 -> ciflow/inductor/156967 2025-08-14T21:25:14.1705201Z * [new tag] ciflow/inductor/157148 -> ciflow/inductor/157148 2025-08-14T21:25:14.1705450Z * [new tag] ciflow/inductor/157149 -> ciflow/inductor/157149 2025-08-14T21:25:14.1705579Z * [new tag] ciflow/inductor/157152 -> ciflow/inductor/157152 2025-08-14T21:25:14.1708543Z * [new tag] ciflow/inductor/157542 -> ciflow/inductor/157542 2025-08-14T21:25:14.1708866Z * [new tag] ciflow/inductor/157572 -> ciflow/inductor/157572 2025-08-14T21:25:14.1708990Z * [new tag] ciflow/inductor/157635 -> ciflow/inductor/157635 2025-08-14T21:25:14.1709260Z * [new tag] ciflow/inductor/157685 -> ciflow/inductor/157685 2025-08-14T21:25:14.1709384Z * [new tag] ciflow/inductor/157686 -> ciflow/inductor/157686 2025-08-14T21:25:14.1709617Z * [new tag] ciflow/inductor/157689 -> ciflow/inductor/157689 2025-08-14T21:25:14.1709746Z * [new tag] ciflow/inductor/157699 -> ciflow/inductor/157699 2025-08-14T21:25:14.1709862Z * [new tag] ciflow/inductor/157743 -> ciflow/inductor/157743 2025-08-14T21:25:14.1709984Z * [new tag] ciflow/inductor/157944 -> ciflow/inductor/157944 2025-08-14T21:25:14.1710102Z * [new tag] ciflow/inductor/157971 -> ciflow/inductor/157971 2025-08-14T21:25:14.1710224Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-08-14T21:25:14.1710354Z * [new tag] ciflow/inductor/158061 -> ciflow/inductor/158061 2025-08-14T21:25:14.1710478Z * [new tag] ciflow/inductor/158091 -> ciflow/inductor/158091 2025-08-14T21:25:14.1710605Z * [new tag] ciflow/inductor/158097 -> ciflow/inductor/158097 2025-08-14T21:25:14.1710724Z * [new tag] ciflow/inductor/158098 -> ciflow/inductor/158098 2025-08-14T21:25:14.1710840Z * [new tag] ciflow/inductor/158104 -> ciflow/inductor/158104 2025-08-14T21:25:14.1710969Z * [new tag] ciflow/inductor/158168 -> ciflow/inductor/158168 2025-08-14T21:25:14.1711573Z * [new tag] ciflow/inductor/158250 -> ciflow/inductor/158250 2025-08-14T21:25:14.1711716Z * [new tag] ciflow/inductor/158321 -> ciflow/inductor/158321 2025-08-14T21:25:14.1712613Z * [new tag] ciflow/inductor/158609 -> ciflow/inductor/158609 2025-08-14T21:25:14.1713035Z * [new tag] ciflow/inductor/158647 -> ciflow/inductor/158647 2025-08-14T21:25:14.1713210Z * [new tag] ciflow/inductor/158914 -> ciflow/inductor/158914 2025-08-14T21:25:14.1715092Z * [new tag] ciflow/inductor/158932 -> ciflow/inductor/158932 2025-08-14T21:25:14.1715286Z * [new tag] ciflow/inductor/158987 -> ciflow/inductor/158987 2025-08-14T21:25:14.1715431Z * [new tag] ciflow/inductor/159009 -> ciflow/inductor/159009 2025-08-14T21:25:14.1715557Z * [new tag] ciflow/inductor/159010 -> ciflow/inductor/159010 2025-08-14T21:25:14.1715692Z * [new tag] ciflow/inductor/159093 -> ciflow/inductor/159093 2025-08-14T21:25:14.1716246Z * [new tag] ciflow/inductor/159158 -> ciflow/inductor/159158 2025-08-14T21:25:14.1716997Z * [new tag] ciflow/inductor/159197 -> ciflow/inductor/159197 2025-08-14T21:25:14.1717491Z * [new tag] ciflow/inductor/159274 -> ciflow/inductor/159274 2025-08-14T21:25:14.1717948Z * [new tag] ciflow/inductor/159281 -> ciflow/inductor/159281 2025-08-14T21:25:14.1718191Z * [new tag] ciflow/inductor/159329 -> ciflow/inductor/159329 2025-08-14T21:25:14.1721329Z * [new tag] ciflow/inductor/159361 -> ciflow/inductor/159361 2025-08-14T21:25:14.1721672Z * [new tag] ciflow/inductor/159365 -> ciflow/inductor/159365 2025-08-14T21:25:14.1721858Z * [new tag] ciflow/inductor/159366 -> ciflow/inductor/159366 2025-08-14T21:25:14.1722009Z * [new tag] ciflow/inductor/159367 -> ciflow/inductor/159367 2025-08-14T21:25:14.1722233Z * [new tag] ciflow/inductor/159368 -> ciflow/inductor/159368 2025-08-14T21:25:14.1722372Z * [new tag] ciflow/inductor/159473 -> ciflow/inductor/159473 2025-08-14T21:25:14.1722597Z * [new tag] ciflow/inductor/159483 -> ciflow/inductor/159483 2025-08-14T21:25:14.1723170Z * [new tag] ciflow/inductor/159508 -> ciflow/inductor/159508 2025-08-14T21:25:14.1723336Z * [new tag] ciflow/inductor/159523 -> ciflow/inductor/159523 2025-08-14T21:25:14.1723478Z * [new tag] ciflow/inductor/159678 -> ciflow/inductor/159678 2025-08-14T21:25:14.1723846Z * [new tag] ciflow/inductor/159691 -> ciflow/inductor/159691 2025-08-14T21:25:14.1729552Z * [new tag] ciflow/inductor/159778 -> ciflow/inductor/159778 2025-08-14T21:25:14.1729879Z * [new tag] ciflow/inductor/159786 -> ciflow/inductor/159786 2025-08-14T21:25:14.1730057Z * [new tag] ciflow/inductor/159817 -> ciflow/inductor/159817 2025-08-14T21:25:14.1730217Z * [new tag] ciflow/inductor/159842 -> ciflow/inductor/159842 2025-08-14T21:25:14.1730366Z * [new tag] ciflow/inductor/159864 -> ciflow/inductor/159864 2025-08-14T21:25:14.1730590Z * [new tag] ciflow/inductor/159865 -> ciflow/inductor/159865 2025-08-14T21:25:14.1730738Z * [new tag] ciflow/inductor/159869 -> ciflow/inductor/159869 2025-08-14T21:25:14.1730971Z * [new tag] ciflow/inductor/159875 -> ciflow/inductor/159875 2025-08-14T21:25:14.1731095Z * [new tag] ciflow/inductor/159889 -> ciflow/inductor/159889 2025-08-14T21:25:14.1731228Z * [new tag] ciflow/inductor/159902 -> ciflow/inductor/159902 2025-08-14T21:25:14.1731592Z * [new tag] ciflow/inductor/159923 -> ciflow/inductor/159923 2025-08-14T21:25:14.1731745Z * [new tag] ciflow/inductor/159944 -> ciflow/inductor/159944 2025-08-14T21:25:14.1732985Z * [new tag] ciflow/inductor/160004 -> ciflow/inductor/160004 2025-08-14T21:25:14.1733137Z * [new tag] ciflow/inductor/160080 -> ciflow/inductor/160080 2025-08-14T21:25:14.1733253Z * [new tag] ciflow/inductor/160108 -> ciflow/inductor/160108 2025-08-14T21:25:14.1733373Z * [new tag] ciflow/inductor/160109 -> ciflow/inductor/160109 2025-08-14T21:25:14.1733635Z * [new tag] ciflow/inductor/160111 -> ciflow/inductor/160111 2025-08-14T21:25:14.1733762Z * [new tag] ciflow/inductor/160113 -> ciflow/inductor/160113 2025-08-14T21:25:14.1733886Z * [new tag] ciflow/inductor/160127 -> ciflow/inductor/160127 2025-08-14T21:25:14.1734029Z * [new tag] ciflow/inductor/160131 -> ciflow/inductor/160131 2025-08-14T21:25:14.1734156Z * [new tag] ciflow/inductor/160132 -> ciflow/inductor/160132 2025-08-14T21:25:14.1734265Z * [new tag] ciflow/inductor/160136 -> ciflow/inductor/160136 2025-08-14T21:25:14.1734377Z * [new tag] ciflow/inductor/160138 -> ciflow/inductor/160138 2025-08-14T21:25:14.1734503Z * [new tag] ciflow/inductor/160151 -> ciflow/inductor/160151 2025-08-14T21:25:14.1734614Z * [new tag] ciflow/inductor/160152 -> ciflow/inductor/160152 2025-08-14T21:25:14.1734735Z * [new tag] ciflow/inductor/160154 -> ciflow/inductor/160154 2025-08-14T21:25:14.1734847Z * [new tag] ciflow/inductor/160156 -> ciflow/inductor/160156 2025-08-14T21:25:14.1734969Z * [new tag] ciflow/inductor/160161 -> ciflow/inductor/160161 2025-08-14T21:25:14.1735124Z * [new tag] ciflow/inductor/160166 -> ciflow/inductor/160166 2025-08-14T21:25:14.1735532Z * [new tag] ciflow/inductor/160168 -> ciflow/inductor/160168 2025-08-14T21:25:14.1736026Z * [new tag] ciflow/inductor/160174 -> ciflow/inductor/160174 2025-08-14T21:25:14.1736441Z * [new tag] ciflow/inductor/160181 -> ciflow/inductor/160181 2025-08-14T21:25:14.1737267Z * [new tag] ciflow/inductor/160183 -> ciflow/inductor/160183 2025-08-14T21:25:14.1737405Z * [new tag] ciflow/inductor/160190 -> ciflow/inductor/160190 2025-08-14T21:25:14.1740681Z * [new tag] ciflow/inductor/160198 -> ciflow/inductor/160198 2025-08-14T21:25:14.1741132Z * [new tag] ciflow/inductor/160201 -> ciflow/inductor/160201 2025-08-14T21:25:14.1741293Z * [new tag] ciflow/inductor/160209 -> ciflow/inductor/160209 2025-08-14T21:25:14.1741839Z * [new tag] ciflow/inductor/160218 -> ciflow/inductor/160218 2025-08-14T21:25:14.1742003Z * [new tag] ciflow/inductor/160239 -> ciflow/inductor/160239 2025-08-14T21:25:14.1742113Z * [new tag] ciflow/inductor/160250 -> ciflow/inductor/160250 2025-08-14T21:25:14.1742232Z * [new tag] ciflow/inductor/160253 -> ciflow/inductor/160253 2025-08-14T21:25:14.1742343Z * [new tag] ciflow/inductor/160266 -> ciflow/inductor/160266 2025-08-14T21:25:14.1742453Z * [new tag] ciflow/inductor/160282 -> ciflow/inductor/160282 2025-08-14T21:25:14.1742702Z * [new tag] ciflow/inductor/160298 -> ciflow/inductor/160298 2025-08-14T21:25:14.1742936Z * [new tag] ciflow/inductor/160301 -> ciflow/inductor/160301 2025-08-14T21:25:14.1743149Z * [new tag] ciflow/inductor/160310 -> ciflow/inductor/160310 2025-08-14T21:25:14.1749111Z * [new tag] ciflow/inductor/160323 -> ciflow/inductor/160323 2025-08-14T21:25:14.1751498Z * [new tag] ciflow/inductor/160324 -> ciflow/inductor/160324 2025-08-14T21:25:14.1751796Z * [new tag] ciflow/inductor/160325 -> ciflow/inductor/160325 2025-08-14T21:25:14.1751943Z * [new tag] ciflow/inductor/160326 -> ciflow/inductor/160326 2025-08-14T21:25:14.1752071Z * [new tag] ciflow/inductor/160327 -> ciflow/inductor/160327 2025-08-14T21:25:14.1752269Z * [new tag] ciflow/inductor/160328 -> ciflow/inductor/160328 2025-08-14T21:25:14.1752536Z * [new tag] ciflow/inductor/160329 -> ciflow/inductor/160329 2025-08-14T21:25:14.1752796Z * [new tag] ciflow/inductor/160351 -> ciflow/inductor/160351 2025-08-14T21:25:14.1752928Z * [new tag] ciflow/inductor/160353 -> ciflow/inductor/160353 2025-08-14T21:25:14.1753042Z * [new tag] ciflow/inductor/160362 -> ciflow/inductor/160362 2025-08-14T21:25:14.1753168Z * [new tag] ciflow/inductor/160363 -> ciflow/inductor/160363 2025-08-14T21:25:14.1753626Z * [new tag] ciflow/inductor/160364 -> ciflow/inductor/160364 2025-08-14T21:25:14.1753783Z * [new tag] ciflow/inductor/160365 -> ciflow/inductor/160365 2025-08-14T21:25:14.1753915Z * [new tag] ciflow/inductor/160366 -> ciflow/inductor/160366 2025-08-14T21:25:14.1754039Z * [new tag] ciflow/inductor/160367 -> ciflow/inductor/160367 2025-08-14T21:25:14.1754163Z * [new tag] ciflow/inductor/160368 -> ciflow/inductor/160368 2025-08-14T21:25:14.1754327Z * [new tag] ciflow/inductor/160369 -> ciflow/inductor/160369 2025-08-14T21:25:14.1754470Z * [new tag] ciflow/inductor/160371 -> ciflow/inductor/160371 2025-08-14T21:25:14.1754596Z * [new tag] ciflow/inductor/160374 -> ciflow/inductor/160374 2025-08-14T21:25:14.1754717Z * [new tag] ciflow/inductor/160375 -> ciflow/inductor/160375 2025-08-14T21:25:14.1754835Z * [new tag] ciflow/inductor/160377 -> ciflow/inductor/160377 2025-08-14T21:25:14.1754972Z * [new tag] ciflow/inductor/160380 -> ciflow/inductor/160380 2025-08-14T21:25:14.1755096Z * [new tag] ciflow/inductor/160381 -> ciflow/inductor/160381 2025-08-14T21:25:14.1755225Z * [new tag] ciflow/inductor/160383 -> ciflow/inductor/160383 2025-08-14T21:25:14.1756275Z * [new tag] ciflow/inductor/160394 -> ciflow/inductor/160394 2025-08-14T21:25:14.1756778Z * [new tag] ciflow/inductor/160401 -> ciflow/inductor/160401 2025-08-14T21:25:14.1757132Z * [new tag] ciflow/inductor/160402 -> ciflow/inductor/160402 2025-08-14T21:25:14.1757266Z * [new tag] ciflow/inductor/160403 -> ciflow/inductor/160403 2025-08-14T21:25:14.1761633Z * [new tag] ciflow/inductor/160424 -> ciflow/inductor/160424 2025-08-14T21:25:14.1761957Z * [new tag] ciflow/inductor/160426 -> ciflow/inductor/160426 2025-08-14T21:25:14.1762151Z * [new tag] ciflow/inductor/160431 -> ciflow/inductor/160431 2025-08-14T21:25:14.1762312Z * [new tag] ciflow/inductor/160448 -> ciflow/inductor/160448 2025-08-14T21:25:14.1762438Z * [new tag] ciflow/inductor/160450 -> ciflow/inductor/160450 2025-08-14T21:25:14.1762564Z * [new tag] ciflow/inductor/160455 -> ciflow/inductor/160455 2025-08-14T21:25:14.1762844Z * [new tag] ciflow/inductor/160456 -> ciflow/inductor/160456 2025-08-14T21:25:14.1763458Z * [new tag] ciflow/inductor/160461 -> ciflow/inductor/160461 2025-08-14T21:25:14.1763631Z * [new tag] ciflow/inductor/160462 -> ciflow/inductor/160462 2025-08-14T21:25:14.1763755Z * [new tag] ciflow/inductor/160467 -> ciflow/inductor/160467 2025-08-14T21:25:14.1763883Z * [new tag] ciflow/inductor/160470 -> ciflow/inductor/160470 2025-08-14T21:25:14.1764002Z * [new tag] ciflow/inductor/160473 -> ciflow/inductor/160473 2025-08-14T21:25:14.1764133Z * [new tag] ciflow/inductor/160476 -> ciflow/inductor/160476 2025-08-14T21:25:14.1764316Z * [new tag] ciflow/inductor/160480 -> ciflow/inductor/160480 2025-08-14T21:25:14.1764753Z * [new tag] ciflow/inductor/160481 -> ciflow/inductor/160481 2025-08-14T21:25:14.1765335Z * [new tag] ciflow/inductor/160482 -> ciflow/inductor/160482 2025-08-14T21:25:14.1765594Z * [new tag] ciflow/inductor/160483 -> ciflow/inductor/160483 2025-08-14T21:25:14.1766373Z * [new tag] ciflow/inductor/160485 -> ciflow/inductor/160485 2025-08-14T21:25:14.1766729Z * [new tag] ciflow/inductor/160486 -> ciflow/inductor/160486 2025-08-14T21:25:14.1767259Z * [new tag] ciflow/inductor/160503 -> ciflow/inductor/160503 2025-08-14T21:25:14.1767683Z * [new tag] ciflow/inductor/160510 -> ciflow/inductor/160510 2025-08-14T21:25:14.1768045Z * [new tag] ciflow/inductor/160527 -> ciflow/inductor/160527 2025-08-14T21:25:14.1768479Z * [new tag] ciflow/inductor/160530 -> ciflow/inductor/160530 2025-08-14T21:25:14.1770992Z * [new tag] ciflow/inductor/160531 -> ciflow/inductor/160531 2025-08-14T21:25:14.1771327Z * [new tag] ciflow/inductor/160538 -> ciflow/inductor/160538 2025-08-14T21:25:14.1771541Z * [new tag] ciflow/inductor/160539 -> ciflow/inductor/160539 2025-08-14T21:25:14.1773760Z * [new tag] ciflow/inductor/160540 -> ciflow/inductor/160540 2025-08-14T21:25:14.1774056Z * [new tag] ciflow/inductor/160548 -> ciflow/inductor/160548 2025-08-14T21:25:14.1774242Z * [new tag] ciflow/inductor/160561 -> ciflow/inductor/160561 2025-08-14T21:25:14.1774519Z * [new tag] ciflow/inductor/160576 -> ciflow/inductor/160576 2025-08-14T21:25:14.1774690Z * [new tag] ciflow/inductor/160578 -> ciflow/inductor/160578 2025-08-14T21:25:14.1774931Z * [new tag] ciflow/inductor/160580 -> ciflow/inductor/160580 2025-08-14T21:25:14.1775259Z * [new tag] ciflow/inductor/160583 -> ciflow/inductor/160583 2025-08-14T21:25:14.1775858Z * [new tag] ciflow/inductor/160589 -> ciflow/inductor/160589 2025-08-14T21:25:14.1780094Z * [new tag] ciflow/inductor/160590 -> ciflow/inductor/160590 2025-08-14T21:25:14.1780402Z * [new tag] ciflow/inductor/160592 -> ciflow/inductor/160592 2025-08-14T21:25:14.1780566Z * [new tag] ciflow/inductor/160596 -> ciflow/inductor/160596 2025-08-14T21:25:14.1780701Z * [new tag] ciflow/inductor/160601 -> ciflow/inductor/160601 2025-08-14T21:25:14.1780896Z * [new tag] ciflow/inductor/160607 -> ciflow/inductor/160607 2025-08-14T21:25:14.1781023Z * [new tag] ciflow/inductor/160608 -> ciflow/inductor/160608 2025-08-14T21:25:14.1781224Z * [new tag] ciflow/inductor/160611 -> ciflow/inductor/160611 2025-08-14T21:25:14.1781964Z * [new tag] ciflow/inductor/160614 -> ciflow/inductor/160614 2025-08-14T21:25:14.1782155Z * [new tag] ciflow/inductor/160616 -> ciflow/inductor/160616 2025-08-14T21:25:14.1782314Z * [new tag] ciflow/inductor/160619 -> ciflow/inductor/160619 2025-08-14T21:25:14.1782456Z * [new tag] ciflow/inductor/160625 -> ciflow/inductor/160625 2025-08-14T21:25:14.1782584Z * [new tag] ciflow/inductor/160635 -> ciflow/inductor/160635 2025-08-14T21:25:14.1782716Z * [new tag] ciflow/inductor/160649 -> ciflow/inductor/160649 2025-08-14T21:25:14.1782841Z * [new tag] ciflow/inductor/160658 -> ciflow/inductor/160658 2025-08-14T21:25:14.1782963Z * [new tag] ciflow/inductor/160662 -> ciflow/inductor/160662 2025-08-14T21:25:14.1783115Z * [new tag] ciflow/inductor/160668 -> ciflow/inductor/160668 2025-08-14T21:25:14.1783265Z * [new tag] ciflow/inductor/160669 -> ciflow/inductor/160669 2025-08-14T21:25:14.1783392Z * [new tag] ciflow/inductor/160670 -> ciflow/inductor/160670 2025-08-14T21:25:14.1783671Z * [new tag] ciflow/inductor/160671 -> ciflow/inductor/160671 2025-08-14T21:25:14.1783825Z * [new tag] ciflow/inductor/160677 -> ciflow/inductor/160677 2025-08-14T21:25:14.1783966Z * [new tag] ciflow/inductor/160679 -> ciflow/inductor/160679 2025-08-14T21:25:14.1784101Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-08-14T21:25:14.1784439Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-08-14T21:25:14.1784620Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-08-14T21:25:14.1785090Z * [new tag] ciflow/linux-aarch64/147855 -> ciflow/linux-aarch64/147855 2025-08-14T21:25:14.1785577Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-08-14T21:25:14.1786019Z * [new tag] ciflow/linux-aarch64/159737 -> ciflow/linux-aarch64/159737 2025-08-14T21:25:14.1786355Z * [new tag] ciflow/linux-aarch64/160078 -> ciflow/linux-aarch64/160078 2025-08-14T21:25:14.1786835Z * [new tag] ciflow/linux-aarch64/160299 -> ciflow/linux-aarch64/160299 2025-08-14T21:25:14.1792062Z * [new tag] ciflow/linux-aarch64/160301 -> ciflow/linux-aarch64/160301 2025-08-14T21:25:14.1792383Z * [new tag] ciflow/mps/155923 -> ciflow/mps/155923 2025-08-14T21:25:14.1792520Z * [new tag] ciflow/mps/157553 -> ciflow/mps/157553 2025-08-14T21:25:14.1792726Z * [new tag] ciflow/mps/157635 -> ciflow/mps/157635 2025-08-14T21:25:14.1792855Z * [new tag] ciflow/mps/160541 -> ciflow/mps/160541 2025-08-14T21:25:14.1793001Z * [new tag] ciflow/nightly/156049 -> ciflow/nightly/156049 2025-08-14T21:25:14.1793264Z * [new tag] ciflow/nightly/158104 -> ciflow/nightly/158104 2025-08-14T21:25:14.1793968Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-08-14T21:25:14.1794363Z * [new tag] ciflow/periodic-rocm-mi300/139971 -> ciflow/periodic-rocm-mi300/139971 2025-08-14T21:25:14.1794538Z * [new tag] ciflow/periodic-rocm-mi300/160073 -> ciflow/periodic-rocm-mi300/160073 2025-08-14T21:25:14.1794702Z * [new tag] ciflow/periodic-rocm-mi300/160538 -> ciflow/periodic-rocm-mi300/160538 2025-08-14T21:25:14.1794855Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-08-14T21:25:14.1794992Z * [new tag] ciflow/periodic/131296 -> ciflow/periodic/131296 2025-08-14T21:25:14.1795127Z * [new tag] ciflow/periodic/139971 -> ciflow/periodic/139971 2025-08-14T21:25:14.1795249Z * [new tag] ciflow/periodic/143959 -> ciflow/periodic/143959 2025-08-14T21:25:14.1795383Z * [new tag] ciflow/periodic/154595 -> ciflow/periodic/154595 2025-08-14T21:25:14.1795516Z * [new tag] ciflow/periodic/156703 -> ciflow/periodic/156703 2025-08-14T21:25:14.1795638Z * [new tag] ciflow/periodic/160201 -> ciflow/periodic/160201 2025-08-14T21:25:14.1796188Z * [new tag] ciflow/periodic/160424 -> ciflow/periodic/160424 2025-08-14T21:25:14.1796733Z * [new tag] ciflow/periodic/160538 -> ciflow/periodic/160538 2025-08-14T21:25:14.1797966Z * [new tag] ciflow/periodic/1febab2a89302464f6c7d69cfbef7a24c421ea65 -> ciflow/periodic/1febab2a89302464f6c7d69cfbef7a24c421ea65 2025-08-14T21:25:14.1798138Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-08-14T21:25:14.1801720Z * [new tag] ciflow/periodic/2ee22e435131369a7e4f8cc4732579acc29a941b -> ciflow/periodic/2ee22e435131369a7e4f8cc4732579acc29a941b 2025-08-14T21:25:14.1802059Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-08-14T21:25:14.1802200Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-08-14T21:25:14.1802326Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-08-14T21:25:14.1802605Z * [new tag] ciflow/periodic/4a773e1e867f28a8ff0b15203e5cd9548f74fcee -> ciflow/periodic/4a773e1e867f28a8ff0b15203e5cd9548f74fcee 2025-08-14T21:25:14.1802872Z * [new tag] ciflow/periodic/5f5f508aa836a46dfe88857fb223049616b94e93 -> ciflow/periodic/5f5f508aa836a46dfe88857fb223049616b94e93 2025-08-14T21:25:14.1803189Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-08-14T21:25:14.1803928Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-08-14T21:25:14.1804123Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-08-14T21:25:14.1804297Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-08-14T21:25:14.1805576Z * [new tag] ciflow/periodic/d7114f05b10de8e6de81ffc567d63944c3117d51 -> ciflow/periodic/d7114f05b10de8e6de81ffc567d63944c3117d51 2025-08-14T21:25:14.1805864Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-08-14T21:25:14.1806355Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-08-14T21:25:14.1808059Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-08-14T21:25:14.1808388Z * [new tag] ciflow/rocm-mi300/151360 -> ciflow/rocm-mi300/151360 2025-08-14T21:25:14.1808539Z * [new tag] ciflow/rocm-mi300/159158 -> ciflow/rocm-mi300/159158 2025-08-14T21:25:14.1808947Z * [new tag] ciflow/rocm-mi300/160073 -> ciflow/rocm-mi300/160073 2025-08-14T21:25:14.1809435Z * [new tag] ciflow/rocm-mi300/160468 -> ciflow/rocm-mi300/160468 2025-08-14T21:25:14.1809588Z * [new tag] ciflow/rocm-mi300/160538 -> ciflow/rocm-mi300/160538 2025-08-14T21:25:14.1810257Z * [new tag] ciflow/rocm-mi355/160215 -> ciflow/rocm-mi355/160215 2025-08-14T21:25:14.1810886Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-08-14T21:25:14.1811068Z * [new tag] ciflow/rocm/151360 -> ciflow/rocm/151360 2025-08-14T21:25:14.1811739Z * [new tag] ciflow/rocm/151845 -> ciflow/rocm/151845 2025-08-14T21:25:14.1811872Z * [new tag] ciflow/rocm/154864 -> ciflow/rocm/154864 2025-08-14T21:25:14.1815383Z * [new tag] ciflow/rocm/156491 -> ciflow/rocm/156491 2025-08-14T21:25:14.1815684Z * [new tag] ciflow/rocm/158219 -> ciflow/rocm/158219 2025-08-14T21:25:14.1815836Z * [new tag] ciflow/rocm/158220 -> ciflow/rocm/158220 2025-08-14T21:25:14.1815963Z * [new tag] ciflow/rocm/158224 -> ciflow/rocm/158224 2025-08-14T21:25:14.1816086Z * [new tag] ciflow/rocm/159158 -> ciflow/rocm/159158 2025-08-14T21:25:14.1816192Z * [new tag] ciflow/rocm/160215 -> ciflow/rocm/160215 2025-08-14T21:25:14.1816297Z * [new tag] ciflow/rocm/160468 -> ciflow/rocm/160468 2025-08-14T21:25:14.1816537Z * [new tag] ciflow/rocm/160538 -> ciflow/rocm/160538 2025-08-14T21:25:14.1816659Z * [new tag] ciflow/s390/143959 -> ciflow/s390/143959 2025-08-14T21:25:14.1816787Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-08-14T21:25:14.1817115Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-08-14T21:25:14.1818209Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-08-14T21:25:14.1818621Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-08-14T21:25:14.1818774Z * [new tag] ciflow/slow/154595 -> ciflow/slow/154595 2025-08-14T21:25:14.1819251Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-08-14T21:25:14.1821584Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-08-14T21:25:14.1821735Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-08-14T21:25:14.1821856Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-08-14T21:25:14.1821981Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-08-14T21:25:14.1822333Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-08-14T21:25:14.1822867Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-08-14T21:25:14.1823443Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-08-14T21:25:14.1824570Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-08-14T21:25:14.1824815Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-08-14T21:25:14.1825156Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-08-14T21:25:14.1827829Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-08-14T21:25:14.1828130Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-08-14T21:25:14.1828260Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-08-14T21:25:14.1828669Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-08-14T21:25:14.1828929Z * [new tag] ciflow/trunk/131296 -> ciflow/trunk/131296 2025-08-14T21:25:14.1829595Z * [new tag] ciflow/trunk/137400 -> ciflow/trunk/137400 2025-08-14T21:25:14.1829771Z * [new tag] ciflow/trunk/138996 -> ciflow/trunk/138996 2025-08-14T21:25:14.1829897Z * [new tag] ciflow/trunk/139971 -> ciflow/trunk/139971 2025-08-14T21:25:14.1830010Z * [new tag] ciflow/trunk/147360 -> ciflow/trunk/147360 2025-08-14T21:25:14.1830400Z * [new tag] ciflow/trunk/147855 -> ciflow/trunk/147855 2025-08-14T21:25:14.1830687Z * [new tag] ciflow/trunk/148180 -> ciflow/trunk/148180 2025-08-14T21:25:14.1831311Z * [new tag] ciflow/trunk/148328 -> ciflow/trunk/148328 2025-08-14T21:25:14.1831877Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-08-14T21:25:14.1832360Z * [new tag] ciflow/trunk/150282 -> ciflow/trunk/150282 2025-08-14T21:25:14.1832665Z * [new tag] ciflow/trunk/150302 -> ciflow/trunk/150302 2025-08-14T21:25:14.1833274Z * [new tag] ciflow/trunk/151845 -> ciflow/trunk/151845 2025-08-14T21:25:14.1833755Z * [new tag] ciflow/trunk/152624 -> ciflow/trunk/152624 2025-08-14T21:25:14.1834523Z * [new tag] ciflow/trunk/154193 -> ciflow/trunk/154193 2025-08-14T21:25:14.1834824Z * [new tag] ciflow/trunk/154595 -> ciflow/trunk/154595 2025-08-14T21:25:14.1835352Z * [new tag] ciflow/trunk/154650 -> ciflow/trunk/154650 2025-08-14T21:25:14.1836589Z * [new tag] ciflow/trunk/154694 -> ciflow/trunk/154694 2025-08-14T21:25:14.1836737Z * [new tag] ciflow/trunk/155958 -> ciflow/trunk/155958 2025-08-14T21:25:14.1837062Z * [new tag] ciflow/trunk/156049 -> ciflow/trunk/156049 2025-08-14T21:25:14.1840235Z * [new tag] ciflow/trunk/156703 -> ciflow/trunk/156703 2025-08-14T21:25:14.1840419Z * [new tag] ciflow/trunk/156851 -> ciflow/trunk/156851 2025-08-14T21:25:14.1840543Z * [new tag] ciflow/trunk/157148 -> ciflow/trunk/157148 2025-08-14T21:25:14.1840664Z * [new tag] ciflow/trunk/157152 -> ciflow/trunk/157152 2025-08-14T21:25:14.1840778Z * [new tag] ciflow/trunk/157432 -> ciflow/trunk/157432 2025-08-14T21:25:14.1840890Z * [new tag] ciflow/trunk/157685 -> ciflow/trunk/157685 2025-08-14T21:25:14.1841183Z * [new tag] ciflow/trunk/157689 -> ciflow/trunk/157689 2025-08-14T21:25:14.1841522Z * [new tag] ciflow/trunk/157699 -> ciflow/trunk/157699 2025-08-14T21:25:14.1841647Z * [new tag] ciflow/trunk/157813 -> ciflow/trunk/157813 2025-08-14T21:25:14.1841997Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-08-14T21:25:14.1842521Z * [new tag] ciflow/trunk/158091 -> ciflow/trunk/158091 2025-08-14T21:25:14.1842924Z * [new tag] ciflow/trunk/158104 -> ciflow/trunk/158104 2025-08-14T21:25:14.1843497Z * [new tag] ciflow/trunk/158219 -> ciflow/trunk/158219 2025-08-14T21:25:14.1843861Z * [new tag] ciflow/trunk/158220 -> ciflow/trunk/158220 2025-08-14T21:25:14.1844396Z * [new tag] ciflow/trunk/158224 -> ciflow/trunk/158224 2025-08-14T21:25:14.1845137Z * [new tag] ciflow/trunk/158529 -> ciflow/trunk/158529 2025-08-14T21:25:14.1845390Z * [new tag] ciflow/trunk/158647 -> ciflow/trunk/158647 2025-08-14T21:25:14.1846246Z * [new tag] ciflow/trunk/158810 -> ciflow/trunk/158810 2025-08-14T21:25:14.1846384Z * [new tag] ciflow/trunk/158812 -> ciflow/trunk/158812 2025-08-14T21:25:14.1846892Z * [new tag] ciflow/trunk/158863 -> ciflow/trunk/158863 2025-08-14T21:25:14.1847385Z * [new tag] ciflow/trunk/158864 -> ciflow/trunk/158864 2025-08-14T21:25:14.1847796Z * [new tag] ciflow/trunk/158883 -> ciflow/trunk/158883 2025-08-14T21:25:14.1848247Z * [new tag] ciflow/trunk/158914 -> ciflow/trunk/158914 2025-08-14T21:25:14.1848715Z * [new tag] ciflow/trunk/158965 -> ciflow/trunk/158965 2025-08-14T21:25:14.1849178Z * [new tag] ciflow/trunk/158987 -> ciflow/trunk/158987 2025-08-14T21:25:14.1850021Z * [new tag] ciflow/trunk/159033 -> ciflow/trunk/159033 2025-08-14T21:25:14.1850146Z * [new tag] ciflow/trunk/159140 -> ciflow/trunk/159140 2025-08-14T21:25:14.1853157Z * [new tag] ciflow/trunk/159158 -> ciflow/trunk/159158 2025-08-14T21:25:14.1853323Z * [new tag] ciflow/trunk/159553 -> ciflow/trunk/159553 2025-08-14T21:25:14.1853446Z * [new tag] ciflow/trunk/159562 -> ciflow/trunk/159562 2025-08-14T21:25:14.1853565Z * [new tag] ciflow/trunk/159682 -> ciflow/trunk/159682 2025-08-14T21:25:14.1859293Z * [new tag] ciflow/trunk/159691 -> ciflow/trunk/159691 2025-08-14T21:25:14.1859622Z * [new tag] ciflow/trunk/159842 -> ciflow/trunk/159842 2025-08-14T21:25:14.1859895Z * [new tag] ciflow/trunk/159889 -> ciflow/trunk/159889 2025-08-14T21:25:14.1860043Z * [new tag] ciflow/trunk/159923 -> ciflow/trunk/159923 2025-08-14T21:25:14.1860186Z * [new tag] ciflow/trunk/160004 -> ciflow/trunk/160004 2025-08-14T21:25:14.1860312Z * [new tag] ciflow/trunk/160113 -> ciflow/trunk/160113 2025-08-14T21:25:14.1860429Z * [new tag] ciflow/trunk/160161 -> ciflow/trunk/160161 2025-08-14T21:25:14.1860698Z * [new tag] ciflow/trunk/160168 -> ciflow/trunk/160168 2025-08-14T21:25:14.1860842Z * [new tag] ciflow/trunk/160181 -> ciflow/trunk/160181 2025-08-14T21:25:14.1860960Z * [new tag] ciflow/trunk/160183 -> ciflow/trunk/160183 2025-08-14T21:25:14.1861072Z * [new tag] ciflow/trunk/160190 -> ciflow/trunk/160190 2025-08-14T21:25:14.1861196Z * [new tag] ciflow/trunk/160198 -> ciflow/trunk/160198 2025-08-14T21:25:14.1861315Z * [new tag] ciflow/trunk/160205 -> ciflow/trunk/160205 2025-08-14T21:25:14.1861432Z * [new tag] ciflow/trunk/160219 -> ciflow/trunk/160219 2025-08-14T21:25:14.1866251Z * [new tag] ciflow/trunk/160224 -> ciflow/trunk/160224 2025-08-14T21:25:14.1866564Z * [new tag] ciflow/trunk/160250 -> ciflow/trunk/160250 2025-08-14T21:25:14.1866731Z * [new tag] ciflow/trunk/160253 -> ciflow/trunk/160253 2025-08-14T21:25:14.1866857Z * [new tag] ciflow/trunk/160335 -> ciflow/trunk/160335 2025-08-14T21:25:14.1867113Z * [new tag] ciflow/trunk/160338 -> ciflow/trunk/160338 2025-08-14T21:25:14.1867263Z * [new tag] ciflow/trunk/160383 -> ciflow/trunk/160383 2025-08-14T21:25:14.1867381Z * [new tag] ciflow/trunk/160401 -> ciflow/trunk/160401 2025-08-14T21:25:14.1867495Z * [new tag] ciflow/trunk/160403 -> ciflow/trunk/160403 2025-08-14T21:25:14.1867608Z * [new tag] ciflow/trunk/160430 -> ciflow/trunk/160430 2025-08-14T21:25:14.1867729Z * [new tag] ciflow/trunk/160431 -> ciflow/trunk/160431 2025-08-14T21:25:14.1867838Z * [new tag] ciflow/trunk/160439 -> ciflow/trunk/160439 2025-08-14T21:25:14.1867954Z * [new tag] ciflow/trunk/160449 -> ciflow/trunk/160449 2025-08-14T21:25:14.1868222Z * [new tag] ciflow/trunk/160454 -> ciflow/trunk/160454 2025-08-14T21:25:14.1868353Z * [new tag] ciflow/trunk/160468 -> ciflow/trunk/160468 2025-08-14T21:25:14.1868473Z * [new tag] ciflow/trunk/160481 -> ciflow/trunk/160481 2025-08-14T21:25:14.1868588Z * [new tag] ciflow/trunk/160485 -> ciflow/trunk/160485 2025-08-14T21:25:14.1868701Z * [new tag] ciflow/trunk/160519 -> ciflow/trunk/160519 2025-08-14T21:25:14.1868823Z * [new tag] ciflow/trunk/160527 -> ciflow/trunk/160527 2025-08-14T21:25:14.1868937Z * [new tag] ciflow/trunk/160560 -> ciflow/trunk/160560 2025-08-14T21:25:14.1869058Z * [new tag] ciflow/trunk/160578 -> ciflow/trunk/160578 2025-08-14T21:25:14.1869171Z * [new tag] ciflow/trunk/160589 -> ciflow/trunk/160589 2025-08-14T21:25:14.1869285Z * [new tag] ciflow/trunk/160592 -> ciflow/trunk/160592 2025-08-14T21:25:14.1869406Z * [new tag] ciflow/trunk/160649 -> ciflow/trunk/160649 2025-08-14T21:25:14.1869516Z * [new tag] ciflow/trunk/160656 -> ciflow/trunk/160656 2025-08-14T21:25:14.1869639Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-08-14T21:25:14.1869763Z * [new tag] ciflow/vllm/160116 -> ciflow/vllm/160116 2025-08-14T21:25:14.1869876Z * [new tag] ciflow/vllm/160583 -> ciflow/vllm/160583 2025-08-14T21:25:14.1869993Z * [new tag] ciflow/vllm/160619 -> ciflow/vllm/160619 2025-08-14T21:25:14.1870241Z * [new tag] ciflow/vllm/160625 -> ciflow/vllm/160625 2025-08-14T21:25:14.1870374Z * [new tag] ciflow/vllm/160627 -> ciflow/vllm/160627 2025-08-14T21:25:14.1870932Z * [new tag] ciflow/win-arm64/156049 -> ciflow/win-arm64/156049 2025-08-14T21:25:14.1871318Z * [new tag] ciflow/win-arm64/158104 -> ciflow/win-arm64/158104 2025-08-14T21:25:14.1871787Z * [new tag] ciflow/win-arm64/159553 -> ciflow/win-arm64/159553 2025-08-14T21:25:14.1872224Z * [new tag] ciflow/win-arm64/159562 -> ciflow/win-arm64/159562 2025-08-14T21:25:14.1872912Z * [new tag] ciflow/win-arm64/159777 -> ciflow/win-arm64/159777 2025-08-14T21:25:14.1873192Z * [new tag] ciflow/win-arm64/159780 -> ciflow/win-arm64/159780 2025-08-14T21:25:14.1873687Z * [new tag] ciflow/win-arm64/159842 -> ciflow/win-arm64/159842 2025-08-14T21:25:14.1874402Z * [new tag] ciflow/win-arm64/160250 -> ciflow/win-arm64/160250 2025-08-14T21:25:14.1874602Z * [new tag] ciflow/win-arm64/160253 -> ciflow/win-arm64/160253 2025-08-14T21:25:14.1875003Z * [new tag] ciflow/win-arm64/160454 -> ciflow/win-arm64/160454 2025-08-14T21:25:14.1875458Z * [new tag] ciflow/win-arm64/160560 -> ciflow/win-arm64/160560 2025-08-14T21:25:14.1876018Z * [new tag] ciflow/xpu/138996 -> ciflow/xpu/138996 2025-08-14T21:25:14.1876803Z * [new tag] ciflow/xpu/139971 -> ciflow/xpu/139971 2025-08-14T21:25:14.1877113Z * [new tag] ciflow/xpu/140972 -> ciflow/xpu/140972 2025-08-14T21:25:14.1877619Z * [new tag] ciflow/xpu/143553 -> ciflow/xpu/143553 2025-08-14T21:25:14.1882687Z * [new tag] ciflow/xpu/156272 -> ciflow/xpu/156272 2025-08-14T21:25:14.1882943Z * [new tag] ciflow/xpu/156812 -> ciflow/xpu/156812 2025-08-14T21:25:14.1883242Z * [new tag] ciflow/xpu/157699 -> ciflow/xpu/157699 2025-08-14T21:25:14.1883393Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-08-14T21:25:14.1883564Z * [new tag] ciflow/xpu/158336 -> ciflow/xpu/158336 2025-08-14T21:25:14.1883893Z * [new tag] ciflow/xpu/158733 -> ciflow/xpu/158733 2025-08-14T21:25:14.1884153Z * [new tag] ciflow/xpu/159033 -> ciflow/xpu/159033 2025-08-14T21:25:14.1884407Z * [new tag] ciflow/xpu/159118 -> ciflow/xpu/159118 2025-08-14T21:25:14.1884511Z * [new tag] ciflow/xpu/159140 -> ciflow/xpu/159140 2025-08-14T21:25:14.1884707Z * [new tag] ciflow/xpu/159241 -> ciflow/xpu/159241 2025-08-14T21:25:14.1885297Z * [new tag] ciflow/xpu/159473 -> ciflow/xpu/159473 2025-08-14T21:25:14.1885467Z * [new tag] ciflow/xpu/159474 -> ciflow/xpu/159474 2025-08-14T21:25:14.1885604Z * [new tag] ciflow/xpu/159553 -> ciflow/xpu/159553 2025-08-14T21:25:14.1885728Z * [new tag] ciflow/xpu/159944 -> ciflow/xpu/159944 2025-08-14T21:25:14.1886045Z * [new tag] ciflow/xpu/160062 -> ciflow/xpu/160062 2025-08-14T21:25:14.1886342Z * [new tag] ciflow/xpu/160067 -> ciflow/xpu/160067 2025-08-14T21:25:14.1887125Z * [new tag] ciflow/xpu/160158 -> ciflow/xpu/160158 2025-08-14T21:25:14.1887733Z * [new tag] ciflow/xpu/160173 -> ciflow/xpu/160173 2025-08-14T21:25:14.1888042Z * [new tag] ciflow/xpu/160183 -> ciflow/xpu/160183 2025-08-14T21:25:14.1888188Z * [new tag] ciflow/xpu/160301 -> ciflow/xpu/160301 2025-08-14T21:25:14.1888555Z * [new tag] ciflow/xpu/160403 -> ciflow/xpu/160403 2025-08-14T21:25:14.1889374Z * [new tag] ciflow/xpu/160606 -> ciflow/xpu/160606 2025-08-14T21:25:14.1889520Z * [new tag] cslpull75 -> cslpull75 2025-08-14T21:25:14.1892202Z * [new tag] cslpull76 -> cslpull76 2025-08-14T21:25:14.1892355Z * [new tag] cslpull77 -> cslpull77 2025-08-14T21:25:14.1892474Z * [new tag] cslpull78 -> cslpull78 2025-08-14T21:25:14.1892568Z * [new tag] cslpull79 -> cslpull79 2025-08-14T21:25:14.1892830Z * [new tag] cslpull80 -> cslpull80 2025-08-14T21:25:14.1892947Z * [new tag] cslpull81 -> cslpull81 2025-08-14T21:25:14.1894270Z * [new tag] cslpull82 -> cslpull82 2025-08-14T21:25:14.1894426Z * [new tag] cslpull83 -> cslpull83 2025-08-14T21:25:14.1894854Z * [new tag] cslpull84 -> cslpull84 2025-08-14T21:25:14.1896886Z * [new tag] cslpull85 -> cslpull85 2025-08-14T21:25:14.1897194Z * [new tag] cslpull86 -> cslpull86 2025-08-14T21:25:14.1897316Z * [new tag] cslpull87 -> cslpull87 2025-08-14T21:25:14.1897455Z * [new tag] cslpull88 -> cslpull88 2025-08-14T21:25:14.1897555Z * [new tag] cslpull89 -> cslpull89 2025-08-14T21:25:14.1897887Z * [new tag] cslpull90 -> cslpull90 2025-08-14T21:25:14.1901809Z * [new tag] cslpull91 -> cslpull91 2025-08-14T21:25:14.1902116Z * [new tag] cslpull92 -> cslpull92 2025-08-14T21:25:14.1902248Z * [new tag] flight_5 -> flight_5 2025-08-14T21:25:14.1902462Z * [new tag] flight_5.1 -> flight_5.1 2025-08-14T21:25:14.1902582Z * [new tag] flight_5.2 -> flight_5.2 2025-08-14T21:25:14.1902722Z * [new tag] flight_5.3 -> flight_5.3 2025-08-14T21:25:14.1902859Z * [new tag] forpull1 -> forpull1 2025-08-14T21:25:14.1903584Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-08-14T21:25:14.1903735Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-08-14T21:25:14.1903891Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-08-14T21:25:14.1905337Z * [new tag] nightly-binary -> nightly-binary 2025-08-14T21:25:14.1905656Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-08-14T21:25:14.1905828Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-08-14T21:25:14.1906080Z * [new tag] trunk/01584d2a7d029c9749eb73678cf1dc313cc35df6 -> trunk/01584d2a7d029c9749eb73678cf1dc313cc35df6 2025-08-14T21:25:14.1906631Z * [new tag] trunk/017259f9c65b6fad55fb9597d7077e2543eaae46 -> trunk/017259f9c65b6fad55fb9597d7077e2543eaae46 2025-08-14T21:25:14.1908587Z * [new tag] trunk/01bcf9a40dea937637d2cdd530bed2652510943d -> trunk/01bcf9a40dea937637d2cdd530bed2652510943d 2025-08-14T21:25:14.1909156Z * [new tag] trunk/01f66d08d93365015f4af005a252f439c4d4013a -> trunk/01f66d08d93365015f4af005a252f439c4d4013a 2025-08-14T21:25:14.1909544Z * [new tag] trunk/03b254e49f2d4c092e6ca712e5702cf2895aa47e -> trunk/03b254e49f2d4c092e6ca712e5702cf2895aa47e 2025-08-14T21:25:14.1909943Z * [new tag] trunk/05029ad1c30865d3f7e7fd13384db9d826e563eb -> trunk/05029ad1c30865d3f7e7fd13384db9d826e563eb 2025-08-14T21:25:14.1910294Z * [new tag] trunk/05c19d1acecc01b0d2512364183058a6885b9869 -> trunk/05c19d1acecc01b0d2512364183058a6885b9869 2025-08-14T21:25:14.1910606Z * [new tag] trunk/05c417715f791875fbf28cfc3fc86142de1a3206 -> trunk/05c417715f791875fbf28cfc3fc86142de1a3206 2025-08-14T21:25:14.1911151Z * [new tag] trunk/06824f3c7268bb807a422b663047cd0900ddd126 -> trunk/06824f3c7268bb807a422b663047cd0900ddd126 2025-08-14T21:25:14.1911729Z * [new tag] trunk/077cb389746a7d61cfc018aad2ba29a8aa195610 -> trunk/077cb389746a7d61cfc018aad2ba29a8aa195610 2025-08-14T21:25:14.1912531Z * [new tag] trunk/089c4a1ba007ed4abb3e5e0eafd97b7584566057 -> trunk/089c4a1ba007ed4abb3e5e0eafd97b7584566057 2025-08-14T21:25:14.1912861Z * [new tag] trunk/09381f5dacda7bbbfa361f5df76bde5cd309adc1 -> trunk/09381f5dacda7bbbfa361f5df76bde5cd309adc1 2025-08-14T21:25:14.1913353Z * [new tag] trunk/0bd3af4fb87445f4de3a1f9b823e399c8b3cefde -> trunk/0bd3af4fb87445f4de3a1f9b823e399c8b3cefde 2025-08-14T21:25:14.1916866Z * [new tag] trunk/0d3461bac0fb5177e35152d980b301ea3a0aa2c4 -> trunk/0d3461bac0fb5177e35152d980b301ea3a0aa2c4 2025-08-14T21:25:14.1917163Z * [new tag] trunk/0d40ff3b496e68193bc16d5391fa2e3623709f81 -> trunk/0d40ff3b496e68193bc16d5391fa2e3623709f81 2025-08-14T21:25:14.1917419Z * [new tag] trunk/0d71ca2c46753bb268bfdcf815c14415c122a289 -> trunk/0d71ca2c46753bb268bfdcf815c14415c122a289 2025-08-14T21:25:14.1917667Z * [new tag] trunk/0d88593dd826544c9e7bd4aa615ef86847a78d2b -> trunk/0d88593dd826544c9e7bd4aa615ef86847a78d2b 2025-08-14T21:25:14.1917885Z * [new tag] trunk/0e3e377bd5126cfcc69d70c4d77b352d3404cc11 -> trunk/0e3e377bd5126cfcc69d70c4d77b352d3404cc11 2025-08-14T21:25:14.1918113Z * [new tag] trunk/0f3b10b8eebe68e3c75d473d499b87dfe14a2eca -> trunk/0f3b10b8eebe68e3c75d473d499b87dfe14a2eca 2025-08-14T21:25:14.1918334Z * [new tag] trunk/101276f81b4d2a8c31bfd6796b986d4c1bfdf483 -> trunk/101276f81b4d2a8c31bfd6796b986d4c1bfdf483 2025-08-14T21:25:14.1918643Z * [new tag] trunk/1028c5e2d50e121865bf98307e7c035f549a24b2 -> trunk/1028c5e2d50e121865bf98307e7c035f549a24b2 2025-08-14T21:25:14.1922658Z * [new tag] trunk/10bc36fe840cb3510fab84d2ea22663b76702f1e -> trunk/10bc36fe840cb3510fab84d2ea22663b76702f1e 2025-08-14T21:25:14.1924664Z * [new tag] trunk/10e3514c962b58cbbee994257872a626ff76d51b -> trunk/10e3514c962b58cbbee994257872a626ff76d51b 2025-08-14T21:25:14.1925010Z * [new tag] trunk/1128f4c2a822cbe34a9d966306af15097179ffe1 -> trunk/1128f4c2a822cbe34a9d966306af15097179ffe1 2025-08-14T21:25:14.1931310Z * [new tag] trunk/114a6c40434bfb9cfa5abc30e9e34d81300d743e -> trunk/114a6c40434bfb9cfa5abc30e9e34d81300d743e 2025-08-14T21:25:14.1931728Z * [new tag] trunk/118bc97b14c24ac88a4b0c0750a9e7bf93154c76 -> trunk/118bc97b14c24ac88a4b0c0750a9e7bf93154c76 2025-08-14T21:25:14.1931988Z * [new tag] trunk/1196bb1c2e4d5a7edc09f2260e3034132f0c6c91 -> trunk/1196bb1c2e4d5a7edc09f2260e3034132f0c6c91 2025-08-14T21:25:14.1932334Z * [new tag] trunk/11a3565f1872bbad9c253a127e8d4ce7a1b40ec8 -> trunk/11a3565f1872bbad9c253a127e8d4ce7a1b40ec8 2025-08-14T21:25:14.1933061Z * [new tag] trunk/15e49f61643e4c0eef420f0981609709ef55b848 -> trunk/15e49f61643e4c0eef420f0981609709ef55b848 2025-08-14T21:25:14.1933357Z * [new tag] trunk/16d15445f8bd8740095b23de4af89d757af793ca -> trunk/16d15445f8bd8740095b23de4af89d757af793ca 2025-08-14T21:25:14.1933587Z * [new tag] trunk/178515d0ff6833c8e9221482b2a650ab31e00019 -> trunk/178515d0ff6833c8e9221482b2a650ab31e00019 2025-08-14T21:25:14.1933818Z * [new tag] trunk/182efe31dbe43376e7eef7338356aaf94d5bcabe -> trunk/182efe31dbe43376e7eef7338356aaf94d5bcabe 2025-08-14T21:25:14.1934050Z * [new tag] trunk/194fcfcfbdad0add1a1b695321e31a576058f4cf -> trunk/194fcfcfbdad0add1a1b695321e31a576058f4cf 2025-08-14T21:25:14.1934283Z * [new tag] trunk/195b5c2e27eb8f21cbc8ad1e90f42db5a8cfccca -> trunk/195b5c2e27eb8f21cbc8ad1e90f42db5a8cfccca 2025-08-14T21:25:14.1934502Z * [new tag] trunk/198b5fd2d47fa3d5110ceba6827a3b18e0064014 -> trunk/198b5fd2d47fa3d5110ceba6827a3b18e0064014 2025-08-14T21:25:14.1934907Z * [new tag] trunk/199e9abb6a366bbd27c39d1da7c3123b4eea9b5a -> trunk/199e9abb6a366bbd27c39d1da7c3123b4eea9b5a 2025-08-14T21:25:14.1935136Z * [new tag] trunk/19b4283884b2d9b3a0eb364da10b1540d14ab7a7 -> trunk/19b4283884b2d9b3a0eb364da10b1540d14ab7a7 2025-08-14T21:25:14.1935349Z * [new tag] trunk/1c2587119152cec3905647a47c65d3d26619c5a8 -> trunk/1c2587119152cec3905647a47c65d3d26619c5a8 2025-08-14T21:25:14.1935570Z * [new tag] trunk/1c26c53851c212a7c90a325549a72f0571613a8c -> trunk/1c26c53851c212a7c90a325549a72f0571613a8c 2025-08-14T21:25:14.1935802Z * [new tag] trunk/1c2cba17eab2b09d87142883da2bdbdbcf018613 -> trunk/1c2cba17eab2b09d87142883da2bdbdbcf018613 2025-08-14T21:25:14.1936033Z * [new tag] trunk/1d80d697a269234b47ec7ede192faf3bb9b159e3 -> trunk/1d80d697a269234b47ec7ede192faf3bb9b159e3 2025-08-14T21:25:14.1936264Z * [new tag] trunk/1ea688f9a2602fbcde32c0302b822526ca4219dc -> trunk/1ea688f9a2602fbcde32c0302b822526ca4219dc 2025-08-14T21:25:14.1936482Z * [new tag] trunk/1f4057c11ac941fb324386ca594d0a6882185aad -> trunk/1f4057c11ac941fb324386ca594d0a6882185aad 2025-08-14T21:25:14.1936703Z * [new tag] trunk/1fc683cf17c8c673044538d10266c00f92987be2 -> trunk/1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:25:14.1936928Z * [new tag] trunk/1febab2a89302464f6c7d69cfbef7a24c421ea65 -> trunk/1febab2a89302464f6c7d69cfbef7a24c421ea65 2025-08-14T21:25:14.1937154Z * [new tag] trunk/206c1eef6571f906c2792d899a09136b3fce9673 -> trunk/206c1eef6571f906c2792d899a09136b3fce9673 2025-08-14T21:25:14.1937381Z * [new tag] trunk/20bdabbb3c5d6b118a94b2e045c777662563d5bb -> trunk/20bdabbb3c5d6b118a94b2e045c777662563d5bb 2025-08-14T21:25:14.1937602Z * [new tag] trunk/21392c0e06ac2b2621950455975ca6332f0bf641 -> trunk/21392c0e06ac2b2621950455975ca6332f0bf641 2025-08-14T21:25:14.1937821Z * [new tag] trunk/2247aa6d1d43e256255f5c74a781c3190a4387b6 -> trunk/2247aa6d1d43e256255f5c74a781c3190a4387b6 2025-08-14T21:25:14.1938091Z * [new tag] trunk/2259dbed4e0d3f2a8174b5847fd0741aed42451d -> trunk/2259dbed4e0d3f2a8174b5847fd0741aed42451d 2025-08-14T21:25:14.1938324Z * [new tag] trunk/231c72240d80091f099c95e326d3600cba866eee -> trunk/231c72240d80091f099c95e326d3600cba866eee 2025-08-14T21:25:14.1938541Z * [new tag] trunk/24257f5bfaa37795f74d9f64c1b43584128d4b8c -> trunk/24257f5bfaa37795f74d9f64c1b43584128d4b8c 2025-08-14T21:25:14.1938773Z * [new tag] trunk/24f43d0da7ad9c6e95a09a2fee610387728cc1cd -> trunk/24f43d0da7ad9c6e95a09a2fee610387728cc1cd 2025-08-14T21:25:14.1939002Z * [new tag] trunk/2898d3f965e5cd9d02fc2ecdab7c580fd457fea9 -> trunk/2898d3f965e5cd9d02fc2ecdab7c580fd457fea9 2025-08-14T21:25:14.1939226Z * [new tag] trunk/28ccc9e7247798980fe00a11bcd64a8016b5f227 -> trunk/28ccc9e7247798980fe00a11bcd64a8016b5f227 2025-08-14T21:25:14.1939460Z * [new tag] trunk/29712314dd5cf500a8ea3d1c69483a3cb768ca72 -> trunk/29712314dd5cf500a8ea3d1c69483a3cb768ca72 2025-08-14T21:25:14.1939739Z * [new tag] trunk/29d20d49f0b7f4e362e1cefdcdc4b5659969312c -> trunk/29d20d49f0b7f4e362e1cefdcdc4b5659969312c 2025-08-14T21:25:14.1939972Z * [new tag] trunk/2c5e10a5fceb208b11c3d569ae02e348b5893b31 -> trunk/2c5e10a5fceb208b11c3d569ae02e348b5893b31 2025-08-14T21:25:14.1940211Z * [new tag] trunk/2d0cdee394bccadcd0abe19dd4623ed978a331ad -> trunk/2d0cdee394bccadcd0abe19dd4623ed978a331ad 2025-08-14T21:25:14.1940460Z * [new tag] trunk/2e4e5ab4be9e0aeffd9c49b5b2f9f820bd0895b1 -> trunk/2e4e5ab4be9e0aeffd9c49b5b2f9f820bd0895b1 2025-08-14T21:25:14.1940688Z * [new tag] trunk/2ea40fba841b3af8103f332ba62e54f350ba9a51 -> trunk/2ea40fba841b3af8103f332ba62e54f350ba9a51 2025-08-14T21:25:14.1940944Z * [new tag] trunk/2ee22e435131369a7e4f8cc4732579acc29a941b -> trunk/2ee22e435131369a7e4f8cc4732579acc29a941b 2025-08-14T21:25:14.1941170Z * [new tag] trunk/2f4c2226175512af787725c4d5ad7313c60d4db1 -> trunk/2f4c2226175512af787725c4d5ad7313c60d4db1 2025-08-14T21:25:14.1941579Z * [new tag] trunk/3008d985a8fc155eb89374afff50cb33a6bd10d5 -> trunk/3008d985a8fc155eb89374afff50cb33a6bd10d5 2025-08-14T21:25:14.1941950Z * [new tag] trunk/3028fa6ce9d9c96671722ab8213a1a30670d7cf2 -> trunk/3028fa6ce9d9c96671722ab8213a1a30670d7cf2 2025-08-14T21:25:14.1942299Z * [new tag] trunk/303c614f3df95ae2b659c5f6c1838b14e4776ce6 -> trunk/303c614f3df95ae2b659c5f6c1838b14e4776ce6 2025-08-14T21:25:14.1942615Z * [new tag] trunk/305fa2239365ad17ac9c534a68bba8a149c42d67 -> trunk/305fa2239365ad17ac9c534a68bba8a149c42d67 2025-08-14T21:25:14.1943169Z * [new tag] trunk/31c9ac4319c0cc2ed8c6be701c6ccf73f6cb4706 -> trunk/31c9ac4319c0cc2ed8c6be701c6ccf73f6cb4706 2025-08-14T21:25:14.1943713Z * [new tag] trunk/32099961d588fc19ead8afe805d6b5108de75669 -> trunk/32099961d588fc19ead8afe805d6b5108de75669 2025-08-14T21:25:14.1944824Z * [new tag] trunk/32e5e2f596d55bb9441d5d53f3c58bcb55828047 -> trunk/32e5e2f596d55bb9441d5d53f3c58bcb55828047 2025-08-14T21:25:14.1945110Z * [new tag] trunk/334b38ccc4427b1d14981c48a3a0b92180d58225 -> trunk/334b38ccc4427b1d14981c48a3a0b92180d58225 2025-08-14T21:25:14.1945817Z * [new tag] trunk/334ecbd4ffe11858cae7d23d1190ddb4777c2513 -> trunk/334ecbd4ffe11858cae7d23d1190ddb4777c2513 2025-08-14T21:25:14.1946337Z * [new tag] trunk/33d94018668951611b318b7515ae96f04e48eac0 -> trunk/33d94018668951611b318b7515ae96f04e48eac0 2025-08-14T21:25:14.1946792Z * [new tag] trunk/34358f335d95213d96b6cca6a83e7bf3af6a9fcb -> trunk/34358f335d95213d96b6cca6a83e7bf3af6a9fcb 2025-08-14T21:25:14.1947363Z * [new tag] trunk/34ec5ed275f8aa875c80daa97b3e82af0b06f673 -> trunk/34ec5ed275f8aa875c80daa97b3e82af0b06f673 2025-08-14T21:25:14.1948117Z * [new tag] trunk/355462e1278d818deb9ef4a184073d5b66074816 -> trunk/355462e1278d818deb9ef4a184073d5b66074816 2025-08-14T21:25:14.1954646Z * [new tag] trunk/3626ba711b34397d1fbf0a9b1979f85cbf68b919 -> trunk/3626ba711b34397d1fbf0a9b1979f85cbf68b919 2025-08-14T21:25:14.1954906Z * [new tag] trunk/36f46d082a4954921cb8493223f000f2aab79ed7 -> trunk/36f46d082a4954921cb8493223f000f2aab79ed7 2025-08-14T21:25:14.1955208Z * [new tag] trunk/39aa3d1471549b7829c207d634dfdc1d26e346a2 -> trunk/39aa3d1471549b7829c207d634dfdc1d26e346a2 2025-08-14T21:25:14.1955452Z * [new tag] trunk/3a562374401113187ce2566b87e3f1d87d7c53aa -> trunk/3a562374401113187ce2566b87e3f1d87d7c53aa 2025-08-14T21:25:14.1955691Z * [new tag] trunk/3ac86e728dfaa7383ff7f865e9e7d33486188dae -> trunk/3ac86e728dfaa7383ff7f865e9e7d33486188dae 2025-08-14T21:25:14.1955968Z * [new tag] trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08 -> trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08 2025-08-14T21:25:14.1956449Z * [new tag] trunk/3cec82a7e9aea040a34dd7a2587ae6d3bd65dba0 -> trunk/3cec82a7e9aea040a34dd7a2587ae6d3bd65dba0 2025-08-14T21:25:14.1956846Z * [new tag] trunk/3cf7b4024ef83e44e9ae223dbff7c7ab68240cb2 -> trunk/3cf7b4024ef83e44e9ae223dbff7c7ab68240cb2 2025-08-14T21:25:14.1963455Z * [new tag] trunk/3ef2e1ef769582a82c6ddf150e9d11bf4bf1c44f -> trunk/3ef2e1ef769582a82c6ddf150e9d11bf4bf1c44f 2025-08-14T21:25:14.1963899Z * [new tag] trunk/3f1636ebef9b45e8a3cb0eb20d327ee6acb74be0 -> trunk/3f1636ebef9b45e8a3cb0eb20d327ee6acb74be0 2025-08-14T21:25:14.1964302Z * [new tag] trunk/3faee0a6318afcbbbb48687009a459214910d820 -> trunk/3faee0a6318afcbbbb48687009a459214910d820 2025-08-14T21:25:14.1964826Z * [new tag] trunk/3fcd79e023da7156ac584992ebab29205d3b7881 -> trunk/3fcd79e023da7156ac584992ebab29205d3b7881 2025-08-14T21:25:14.1965151Z * [new tag] trunk/3fe19a7a0af3f4d692af30476c320be18c7e8ae6 -> trunk/3fe19a7a0af3f4d692af30476c320be18c7e8ae6 2025-08-14T21:25:14.1965407Z * [new tag] trunk/41673110cd7c5960824cc74a6fcaeda1a8bc7a23 -> trunk/41673110cd7c5960824cc74a6fcaeda1a8bc7a23 2025-08-14T21:25:14.1965675Z * [new tag] trunk/4183d4ff3dcc1d87400326a9a7998c3f9e966f60 -> trunk/4183d4ff3dcc1d87400326a9a7998c3f9e966f60 2025-08-14T21:25:14.1965926Z * [new tag] trunk/422bd6808bb98cbbac31d157d9c82ad11ba9732d -> trunk/422bd6808bb98cbbac31d157d9c82ad11ba9732d 2025-08-14T21:25:14.1966170Z * [new tag] trunk/42e51cd4b3973a053fcfa80878a3f346fd158e9f -> trunk/42e51cd4b3973a053fcfa80878a3f346fd158e9f 2025-08-14T21:25:14.1966406Z * [new tag] trunk/4416433c7c625127b7f975c92f8ec98ea4c67fd3 -> trunk/4416433c7c625127b7f975c92f8ec98ea4c67fd3 2025-08-14T21:25:14.1966652Z * [new tag] trunk/45ba7ecda876685b083cbbe932450560c566826b -> trunk/45ba7ecda876685b083cbbe932450560c566826b 2025-08-14T21:25:14.1966910Z * [new tag] trunk/47a1db823dfcdacdb99f317428fc3791a18c5812 -> trunk/47a1db823dfcdacdb99f317428fc3791a18c5812 2025-08-14T21:25:14.1967153Z * [new tag] trunk/4a773e1e867f28a8ff0b15203e5cd9548f74fcee -> trunk/4a773e1e867f28a8ff0b15203e5cd9548f74fcee 2025-08-14T21:25:14.1967400Z * [new tag] trunk/4a90dc0c1f68d1f98832b169f792ed1bb195a0f3 -> trunk/4a90dc0c1f68d1f98832b169f792ed1bb195a0f3 2025-08-14T21:25:14.1967654Z * [new tag] trunk/4cde0acc0e4e795e1a12cbdd9b93c8c04c1fa05d -> trunk/4cde0acc0e4e795e1a12cbdd9b93c8c04c1fa05d 2025-08-14T21:25:14.1967897Z * [new tag] trunk/4d419a74610c32b1372f8802dcc61893740a23cf -> trunk/4d419a74610c32b1372f8802dcc61893740a23cf 2025-08-14T21:25:14.1968151Z * [new tag] trunk/4d5b3f2d5af7c8e4f41da4ffca53fafe8bb86235 -> trunk/4d5b3f2d5af7c8e4f41da4ffca53fafe8bb86235 2025-08-14T21:25:14.1968411Z * [new tag] trunk/4e2ddb5db67617f9f5309c8bba0c17adc84cadbc -> trunk/4e2ddb5db67617f9f5309c8bba0c17adc84cadbc 2025-08-14T21:25:14.1968732Z * [new tag] trunk/50a8c118754a6c5a46968f5c8e215ccba6831d42 -> trunk/50a8c118754a6c5a46968f5c8e215ccba6831d42 2025-08-14T21:25:14.1968976Z * [new tag] trunk/50f23ff6f883db5021dd6bab4c146434f98dd15d -> trunk/50f23ff6f883db5021dd6bab4c146434f98dd15d 2025-08-14T21:25:14.1969231Z * [new tag] trunk/515cb70367e84fcbad23fcc5b39eb1d7706df2aa -> trunk/515cb70367e84fcbad23fcc5b39eb1d7706df2aa 2025-08-14T21:25:14.1969453Z * [new tag] trunk/53e39494958b7e2278cc8176f63636e812e8945f -> trunk/53e39494958b7e2278cc8176f63636e812e8945f 2025-08-14T21:25:14.1969706Z * [new tag] trunk/556e2a73f4f0643f7c2aeb5c7dddda43388a40ce -> trunk/556e2a73f4f0643f7c2aeb5c7dddda43388a40ce 2025-08-14T21:25:14.1969957Z * [new tag] trunk/5665dc9ab76b84d7c90d845ffb0f6349b3621919 -> trunk/5665dc9ab76b84d7c90d845ffb0f6349b3621919 2025-08-14T21:25:14.1970203Z * [new tag] trunk/566c6d52ef1411c8262d7b9cf85e2044fdfbe1a3 -> trunk/566c6d52ef1411c8262d7b9cf85e2044fdfbe1a3 2025-08-14T21:25:14.1970511Z * [new tag] trunk/56c828bef93eada0e18d2cc013207831ca80cc99 -> trunk/56c828bef93eada0e18d2cc013207831ca80cc99 2025-08-14T21:25:14.1970729Z * [new tag] trunk/5737372862253a0ac0292407a5844796f02380ad -> trunk/5737372862253a0ac0292407a5844796f02380ad 2025-08-14T21:25:14.1971253Z * [new tag] trunk/57f738b6357cc8fcdde479a0948e723809a1a44d -> trunk/57f738b6357cc8fcdde479a0948e723809a1a44d 2025-08-14T21:25:14.1971756Z * [new tag] trunk/5a40c5784482255b9baf14086cc4b9349fc6d512 -> trunk/5a40c5784482255b9baf14086cc4b9349fc6d512 2025-08-14T21:25:14.1972211Z * [new tag] trunk/5a9c4cfce42b9eb87da0de40c5633f083115c307 -> trunk/5a9c4cfce42b9eb87da0de40c5633f083115c307 2025-08-14T21:25:14.1972872Z * [new tag] trunk/5ace061254af71aa83d1baae81aa1864c9746add -> trunk/5ace061254af71aa83d1baae81aa1864c9746add 2025-08-14T21:25:14.1973485Z * [new tag] trunk/5dddcd5b07c6644efca8d613f4eca1dc95daa87f -> trunk/5dddcd5b07c6644efca8d613f4eca1dc95daa87f 2025-08-14T21:25:14.1974069Z * [new tag] trunk/5ed4f9177907fe403ec4c4499d0d0e9be6b68fcf -> trunk/5ed4f9177907fe403ec4c4499d0d0e9be6b68fcf 2025-08-14T21:25:14.1974741Z * [new tag] trunk/5f1010fbb3850d99c8fdf9a9de2f79260cdc586a -> trunk/5f1010fbb3850d99c8fdf9a9de2f79260cdc586a 2025-08-14T21:25:14.1975037Z * [new tag] trunk/5f5f508aa836a46dfe88857fb223049616b94e93 -> trunk/5f5f508aa836a46dfe88857fb223049616b94e93 2025-08-14T21:25:14.1975805Z * [new tag] trunk/62bac0798100e0e06a86b7a4cee1788413e3d0ca -> trunk/62bac0798100e0e06a86b7a4cee1788413e3d0ca 2025-08-14T21:25:14.1976424Z * [new tag] trunk/63654ba4c5178fd12220cfc9d1c878af2fdd07cc -> trunk/63654ba4c5178fd12220cfc9d1c878af2fdd07cc 2025-08-14T21:25:14.1976791Z * [new tag] trunk/639778b3ee3b80e0894367fdc4442b58ae1b3a62 -> trunk/639778b3ee3b80e0894367fdc4442b58ae1b3a62 2025-08-14T21:25:14.1977278Z * [new tag] trunk/641ee7478150f26969968f49d8b358e199679a8a -> trunk/641ee7478150f26969968f49d8b358e199679a8a 2025-08-14T21:25:14.1977870Z * [new tag] trunk/65053c03a3d209060cb239d20a229dac37cf9dd1 -> trunk/65053c03a3d209060cb239d20a229dac37cf9dd1 2025-08-14T21:25:14.1978362Z * [new tag] trunk/652a6f5954d039d61dc6e6575ccf89d385d74537 -> trunk/652a6f5954d039d61dc6e6575ccf89d385d74537 2025-08-14T21:25:14.1978918Z * [new tag] trunk/685f15dbea66e8ffa8564752f81ad2f6cb447a14 -> trunk/685f15dbea66e8ffa8564752f81ad2f6cb447a14 2025-08-14T21:25:14.1979409Z * [new tag] trunk/68a4b4b2e336cfd4451ce6546d900568e5ddf96c -> trunk/68a4b4b2e336cfd4451ce6546d900568e5ddf96c 2025-08-14T21:25:14.1980365Z * [new tag] trunk/69a0a9aa7f5e320a02e97fa789d2f72baff1554f -> trunk/69a0a9aa7f5e320a02e97fa789d2f72baff1554f 2025-08-14T21:25:14.1980909Z * [new tag] trunk/6be6d06295c870c77a6eb69f96b3170d983520d5 -> trunk/6be6d06295c870c77a6eb69f96b3170d983520d5 2025-08-14T21:25:14.1981166Z * [new tag] trunk/6c05ea6475beaf3acc05e1bda0f3f8fe3bdc1d49 -> trunk/6c05ea6475beaf3acc05e1bda0f3f8fe3bdc1d49 2025-08-14T21:25:14.1984081Z * [new tag] trunk/6da11d9aafc0d84dc7f66030c181608ff2614f66 -> trunk/6da11d9aafc0d84dc7f66030c181608ff2614f66 2025-08-14T21:25:14.1984350Z * [new tag] trunk/6e8865fbc161270e2ffc52817e6c667df417a3f7 -> trunk/6e8865fbc161270e2ffc52817e6c667df417a3f7 2025-08-14T21:25:14.1984591Z * [new tag] trunk/6ea8376f84232048d6be0f7b2edf82aec1b61d58 -> trunk/6ea8376f84232048d6be0f7b2edf82aec1b61d58 2025-08-14T21:25:14.1984806Z * [new tag] trunk/6ee175195ac7853734d64704171993cc6265eb38 -> trunk/6ee175195ac7853734d64704171993cc6265eb38 2025-08-14T21:25:14.1985038Z * [new tag] trunk/6f0f4e0c3eacd479864319127915f869f64e1935 -> trunk/6f0f4e0c3eacd479864319127915f869f64e1935 2025-08-14T21:25:14.1985289Z * [new tag] trunk/70ccdec44b89e355a2cb03ba14a634284f7750f8 -> trunk/70ccdec44b89e355a2cb03ba14a634284f7750f8 2025-08-14T21:25:14.1985713Z * [new tag] trunk/72009ec6bebca7714f99c18449183787f202af4d -> trunk/72009ec6bebca7714f99c18449183787f202af4d 2025-08-14T21:25:14.1986098Z * [new tag] trunk/731ee31f7b6ba19307daab323f6196172b71aaf8 -> trunk/731ee31f7b6ba19307daab323f6196172b71aaf8 2025-08-14T21:25:14.1986845Z * [new tag] trunk/76a0609b6bddb2bc40f1eb4ade12885023653d59 -> trunk/76a0609b6bddb2bc40f1eb4ade12885023653d59 2025-08-14T21:25:14.1987242Z * [new tag] trunk/781e9a7724c47496e3d38a81e6dd6194cf098c41 -> trunk/781e9a7724c47496e3d38a81e6dd6194cf098c41 2025-08-14T21:25:14.1987696Z * [new tag] trunk/78a2fe1d42edeaa2ef7020b0fa0ac82ee4a640e4 -> trunk/78a2fe1d42edeaa2ef7020b0fa0ac82ee4a640e4 2025-08-14T21:25:14.1988224Z * [new tag] trunk/7a974a88f2c529a614baeabe4debd00fc8a3b299 -> trunk/7a974a88f2c529a614baeabe4debd00fc8a3b299 2025-08-14T21:25:14.1990895Z * [new tag] trunk/7ae0629d64b404e0ef5d9c931433ad25e65d6114 -> trunk/7ae0629d64b404e0ef5d9c931433ad25e65d6114 2025-08-14T21:25:14.1991170Z * [new tag] trunk/7d2ec704e47f4b740cdecda5534b305e8e1875ef -> trunk/7d2ec704e47f4b740cdecda5534b305e8e1875ef 2025-08-14T21:25:14.1991400Z * [new tag] trunk/7d87e358ac8440f666fabbfd99058bb5342be6ac -> trunk/7d87e358ac8440f666fabbfd99058bb5342be6ac 2025-08-14T21:25:14.1991621Z * [new tag] trunk/7e27347fd353928c99620495c8c531a5eba7d56b -> trunk/7e27347fd353928c99620495c8c531a5eba7d56b 2025-08-14T21:25:14.1991848Z * [new tag] trunk/7e91394955721c77645fcdb75a5d47a255d65020 -> trunk/7e91394955721c77645fcdb75a5d47a255d65020 2025-08-14T21:25:14.1992092Z * [new tag] trunk/7f4cb4a3e018a621add2a37a3a2f67b982d51001 -> trunk/7f4cb4a3e018a621add2a37a3a2f67b982d51001 2025-08-14T21:25:14.1993763Z * [new tag] trunk/7fbc22855c17741ae016992803b2e147a13aa22d -> trunk/7fbc22855c17741ae016992803b2e147a13aa22d 2025-08-14T21:25:14.1996776Z * [new tag] trunk/8047421fbb607d70ede13b9cd5a60b7b8bdfe348 -> trunk/8047421fbb607d70ede13b9cd5a60b7b8bdfe348 2025-08-14T21:25:14.1997116Z * [new tag] trunk/8088cfa592504a2897b4c78f8a46fe658ab5c2c2 -> trunk/8088cfa592504a2897b4c78f8a46fe658ab5c2c2 2025-08-14T21:25:14.1997440Z * [new tag] trunk/80cca8307943ba64168208b54028f55b2c71daff -> trunk/80cca8307943ba64168208b54028f55b2c71daff 2025-08-14T21:25:14.2004007Z * [new tag] trunk/8147370733bbdcd034cad54e9212e51885a11892 -> trunk/8147370733bbdcd034cad54e9212e51885a11892 2025-08-14T21:25:14.2009103Z * [new tag] trunk/83875cdb5594ccb3c9206b8eb5745fe1d011cf26 -> trunk/83875cdb5594ccb3c9206b8eb5745fe1d011cf26 2025-08-14T21:25:14.2011067Z * [new tag] trunk/8399cf88ce8399d2be93355f29d4cb69f51c0654 -> trunk/8399cf88ce8399d2be93355f29d4cb69f51c0654 2025-08-14T21:25:14.2016322Z * [new tag] trunk/842cc77ab9aafd518593c2fce077d6abb42a5b7f -> trunk/842cc77ab9aafd518593c2fce077d6abb42a5b7f 2025-08-14T21:25:14.2020643Z * [new tag] trunk/85db508af533649d0b3447ff3f0d5fe083150c84 -> trunk/85db508af533649d0b3447ff3f0d5fe083150c84 2025-08-14T21:25:14.2020935Z * [new tag] trunk/86eb65f7f06016bcd5d7951dc9d74bc3993a827a -> trunk/86eb65f7f06016bcd5d7951dc9d74bc3993a827a 2025-08-14T21:25:14.2021165Z * [new tag] trunk/87e6c4079d8ec7d04aff00ed82096b39836a8367 -> trunk/87e6c4079d8ec7d04aff00ed82096b39836a8367 2025-08-14T21:25:14.2021404Z * [new tag] trunk/89654db1abccf7e5f261989a150db4d1619ea2aa -> trunk/89654db1abccf7e5f261989a150db4d1619ea2aa 2025-08-14T21:25:14.2021637Z * [new tag] trunk/8a37f0c90392a2c38b7c5955471fa49edcaf5cb1 -> trunk/8a37f0c90392a2c38b7c5955471fa49edcaf5cb1 2025-08-14T21:25:14.2021857Z * [new tag] trunk/8ab5868a2199fe485c2d66533b9244ccb97e487d -> trunk/8ab5868a2199fe485c2d66533b9244ccb97e487d 2025-08-14T21:25:14.2022077Z * [new tag] trunk/8ae4d2652f64b8444b3d5314b9232bd2119bcde6 -> trunk/8ae4d2652f64b8444b3d5314b9232bd2119bcde6 2025-08-14T21:25:14.2022305Z * [new tag] trunk/8c41cb800ae0411f02ea5da34bd5ccc3790633b0 -> trunk/8c41cb800ae0411f02ea5da34bd5ccc3790633b0 2025-08-14T21:25:14.2022531Z * [new tag] trunk/8cb91e20bc205b1416648d0ffd98d1ba1f3a6fc4 -> trunk/8cb91e20bc205b1416648d0ffd98d1ba1f3a6fc4 2025-08-14T21:25:14.2022753Z * [new tag] trunk/8cfaf51d4e29c9bd9f49ecc98d955ed53df1a13d -> trunk/8cfaf51d4e29c9bd9f49ecc98d955ed53df1a13d 2025-08-14T21:25:14.2022975Z * [new tag] trunk/8d1cf529229dce7cd5ea04abb0faac83b87ca6d1 -> trunk/8d1cf529229dce7cd5ea04abb0faac83b87ca6d1 2025-08-14T21:25:14.2023385Z * [new tag] trunk/8d3d1c844303cb1d46123a1caa76d4cf83973347 -> trunk/8d3d1c844303cb1d46123a1caa76d4cf83973347 2025-08-14T21:25:14.2023613Z * [new tag] trunk/8d6d3246316e1767a57d5e855acd6208da753b75 -> trunk/8d6d3246316e1767a57d5e855acd6208da753b75 2025-08-14T21:25:14.2023828Z * [new tag] trunk/8e6a3138581152ab827a0997f34c470271399f5e -> trunk/8e6a3138581152ab827a0997f34c470271399f5e 2025-08-14T21:25:14.2024045Z * [new tag] trunk/8eee08d2279b98af2522debb6512d37e837e89e3 -> trunk/8eee08d2279b98af2522debb6512d37e837e89e3 2025-08-14T21:25:14.2024255Z * [new tag] trunk/90b78ee50f73b5c963996076a3d54b74b1b965be -> trunk/90b78ee50f73b5c963996076a3d54b74b1b965be 2025-08-14T21:25:14.2024472Z * [new tag] trunk/94b91a876327820a4bb6f5d39d156f13f2553ab6 -> trunk/94b91a876327820a4bb6f5d39d156f13f2553ab6 2025-08-14T21:25:14.2024682Z * [new tag] trunk/95210cc409dd578988c7116b47725c304dea54c7 -> trunk/95210cc409dd578988c7116b47725c304dea54c7 2025-08-14T21:25:14.2024909Z * [new tag] trunk/96bd33b2de79598566df395f32e27c4d33673f05 -> trunk/96bd33b2de79598566df395f32e27c4d33673f05 2025-08-14T21:25:14.2025124Z * [new tag] trunk/9708fcf92db88b80b9010c68662d634434da3106 -> trunk/9708fcf92db88b80b9010c68662d634434da3106 2025-08-14T21:25:14.2025345Z * [new tag] trunk/97c8c98f8dcb9c5c188b691d156e0043dba6c7f8 -> trunk/97c8c98f8dcb9c5c188b691d156e0043dba6c7f8 2025-08-14T21:25:14.2025575Z * [new tag] trunk/9903ca4f70bdc1653016256f5b4fd74fdfc609f8 -> trunk/9903ca4f70bdc1653016256f5b4fd74fdfc609f8 2025-08-14T21:25:14.2025794Z * [new tag] trunk/99bc2f94c1955657e950ebdad5f77e518785ccbd -> trunk/99bc2f94c1955657e950ebdad5f77e518785ccbd 2025-08-14T21:25:14.2026024Z * [new tag] trunk/9a06e6d0310da9d8a59ae05e8ec9c0201b55cacd -> trunk/9a06e6d0310da9d8a59ae05e8ec9c0201b55cacd 2025-08-14T21:25:14.2026248Z * [new tag] trunk/9a0f7a3bb01b235ea04581ee540970a098071b72 -> trunk/9a0f7a3bb01b235ea04581ee540970a098071b72 2025-08-14T21:25:14.2026516Z * [new tag] trunk/9b803cdbe298009f08340c1aaccb25aafbca95d8 -> trunk/9b803cdbe298009f08340c1aaccb25aafbca95d8 2025-08-14T21:25:14.2026751Z * [new tag] trunk/9ccd0f5e31ea54fcf42101dfbaacc103494e34df -> trunk/9ccd0f5e31ea54fcf42101dfbaacc103494e34df 2025-08-14T21:25:14.2026976Z * [new tag] trunk/9d37c960a4fc44d5ac334ca8bf775f85b95d76fc -> trunk/9d37c960a4fc44d5ac334ca8bf775f85b95d76fc 2025-08-14T21:25:14.2027199Z * [new tag] trunk/9e07673deb212c87b1c6fea23799a97474c476ed -> trunk/9e07673deb212c87b1c6fea23799a97474c476ed 2025-08-14T21:25:14.2027421Z * [new tag] trunk/9eedd2a20b64302d0d116ea2802b50948d2ebb09 -> trunk/9eedd2a20b64302d0d116ea2802b50948d2ebb09 2025-08-14T21:25:14.2027658Z * [new tag] trunk/9fa8ce26cf638504469852cbc3e7d04579fc8674 -> trunk/9fa8ce26cf638504469852cbc3e7d04579fc8674 2025-08-14T21:25:14.2027882Z * [new tag] trunk/a06ec54d40013c97fbffc174ea8f524ea5a95715 -> trunk/a06ec54d40013c97fbffc174ea8f524ea5a95715 2025-08-14T21:25:14.2028099Z * [new tag] trunk/a288b15ea9f87ddd665f249d492e0fb0861f5a69 -> trunk/a288b15ea9f87ddd665f249d492e0fb0861f5a69 2025-08-14T21:25:14.2028326Z * [new tag] trunk/a2fd106d670bb4990cebfd00f25ecbae4145e76c -> trunk/a2fd106d670bb4990cebfd00f25ecbae4145e76c 2025-08-14T21:25:14.2028539Z * [new tag] trunk/a354fa91e26b376d96385a2206c5ff5b42aa4600 -> trunk/a354fa91e26b376d96385a2206c5ff5b42aa4600 2025-08-14T21:25:14.2028770Z * [new tag] trunk/a4f69a5da08eace1c1e6469dec6a18aa842da73b -> trunk/a4f69a5da08eace1c1e6469dec6a18aa842da73b 2025-08-14T21:25:14.2028989Z * [new tag] trunk/a53d14d5f846ac44f6c205abb1c5bc4d2f3126ae -> trunk/a53d14d5f846ac44f6c205abb1c5bc4d2f3126ae 2025-08-14T21:25:14.2029235Z * [new tag] trunk/a5652407e4f3d772fc44486ac2abf756decf0861 -> trunk/a5652407e4f3d772fc44486ac2abf756decf0861 2025-08-14T21:25:14.2029473Z * [new tag] trunk/a7abf57aabec0ce686092e2d66e53ba185dbc56b -> trunk/a7abf57aabec0ce686092e2d66e53ba185dbc56b 2025-08-14T21:25:14.2029691Z * [new tag] trunk/a84b60c0c4016785fd93b7b8a0c04f2d0770d332 -> trunk/a84b60c0c4016785fd93b7b8a0c04f2d0770d332 2025-08-14T21:25:14.2029938Z * [new tag] trunk/aa75e917bdb0f95bb6dee81853c2d3c4ab3e1883 -> trunk/aa75e917bdb0f95bb6dee81853c2d3c4ab3e1883 2025-08-14T21:25:14.2030161Z * [new tag] trunk/adcca7d9a1c053495e99012de801b2ea237faad0 -> trunk/adcca7d9a1c053495e99012de801b2ea237faad0 2025-08-14T21:25:14.2030385Z * [new tag] trunk/af10f1f86cc4effc93142a447693d8be55966615 -> trunk/af10f1f86cc4effc93142a447693d8be55966615 2025-08-14T21:25:14.2030619Z * [new tag] trunk/af3cabc55d5699f4da528e1ca39d83338f84ae8c -> trunk/af3cabc55d5699f4da528e1ca39d83338f84ae8c 2025-08-14T21:25:14.2030844Z * [new tag] trunk/b0df7715e8c590c0001d1f9cdb97057be80c9107 -> trunk/b0df7715e8c590c0001d1f9cdb97057be80c9107 2025-08-14T21:25:14.2031074Z * [new tag] trunk/b149c7204c218e7c4d6594a89dd74f72bd480ec5 -> trunk/b149c7204c218e7c4d6594a89dd74f72bd480ec5 2025-08-14T21:25:14.2031292Z * [new tag] trunk/b1a602762e6a6674b406a3137e7e7a678885a97b -> trunk/b1a602762e6a6674b406a3137e7e7a678885a97b 2025-08-14T21:25:14.2031530Z * [new tag] trunk/b1f43548cad8fc0e30bda250f6e196310fa7a4bc -> trunk/b1f43548cad8fc0e30bda250f6e196310fa7a4bc 2025-08-14T21:25:14.2031747Z * [new tag] trunk/b219ca2a00a305753c4f1ea4c9c5d23243d54753 -> trunk/b219ca2a00a305753c4f1ea4c9c5d23243d54753 2025-08-14T21:25:14.2031984Z * [new tag] trunk/b4596895b9d85a686c2cb978938b0a7797b3690a -> trunk/b4596895b9d85a686c2cb978938b0a7797b3690a 2025-08-14T21:25:14.2032249Z * [new tag] trunk/b5fd7223b1bf44720dc9183bda7dfcf7aeccff02 -> trunk/b5fd7223b1bf44720dc9183bda7dfcf7aeccff02 2025-08-14T21:25:14.2032549Z * [new tag] trunk/b602ea9cab7d43a7ee7b4051227090f23fbd3dbf -> trunk/b602ea9cab7d43a7ee7b4051227090f23fbd3dbf 2025-08-14T21:25:14.2032805Z * [new tag] trunk/b6b74aed604bd2e96389ff99aaaf39abc64fdc64 -> trunk/b6b74aed604bd2e96389ff99aaaf39abc64fdc64 2025-08-14T21:25:14.2033048Z * [new tag] trunk/b7db86600a2614adc71c92ca42d359a7ac534d78 -> trunk/b7db86600a2614adc71c92ca42d359a7ac534d78 2025-08-14T21:25:14.2033282Z * [new tag] trunk/b9003ed3d87699e81e436719625a21996a6654e5 -> trunk/b9003ed3d87699e81e436719625a21996a6654e5 2025-08-14T21:25:14.2033544Z * [new tag] trunk/b90feeac86bda00afc2789321bcd706015ff44e3 -> trunk/b90feeac86bda00afc2789321bcd706015ff44e3 2025-08-14T21:25:14.2033800Z * [new tag] trunk/b9d7de3a094598c3dc0dd52e57bce30eb684c9d8 -> trunk/b9d7de3a094598c3dc0dd52e57bce30eb684c9d8 2025-08-14T21:25:14.2034056Z * [new tag] trunk/ba47821f524eee50a214ed39fa2e7765d54aabf4 -> trunk/ba47821f524eee50a214ed39fa2e7765d54aabf4 2025-08-14T21:25:14.2034303Z * [new tag] trunk/ba4ccf5d67e3d237f435eacc2bce3c6025f08491 -> trunk/ba4ccf5d67e3d237f435eacc2bce3c6025f08491 2025-08-14T21:25:14.2034547Z * [new tag] trunk/bcf23ecc476df2bd7479f142567213e2623308ee -> trunk/bcf23ecc476df2bd7479f142567213e2623308ee 2025-08-14T21:25:14.2034806Z * [new tag] trunk/be53f609aaf6f01e2863f490975ea9eaac3ee9ff -> trunk/be53f609aaf6f01e2863f490975ea9eaac3ee9ff 2025-08-14T21:25:14.2035057Z * [new tag] trunk/beb4d7816dedc67a5de1f82e5a45b5910f407941 -> trunk/beb4d7816dedc67a5de1f82e5a45b5910f407941 2025-08-14T21:25:14.2035294Z * [new tag] trunk/bfc873d02ec413344717493e4175a902921359fd -> trunk/bfc873d02ec413344717493e4175a902921359fd 2025-08-14T21:25:14.2035580Z * [new tag] trunk/c184cb3852f0ff2d16a489d61abc3739c309e6ca -> trunk/c184cb3852f0ff2d16a489d61abc3739c309e6ca 2025-08-14T21:25:14.2036561Z * [new tag] trunk/c24ca7f4bf79f62fd623d76346ca27e53f731431 -> trunk/c24ca7f4bf79f62fd623d76346ca27e53f731431 2025-08-14T21:25:14.2036871Z * [new tag] trunk/c3dc8dc4122977893004c49d10e4676cd0a97da4 -> trunk/c3dc8dc4122977893004c49d10e4676cd0a97da4 2025-08-14T21:25:14.2037566Z * [new tag] trunk/c5ec5458a547f7a774468ea0eb2258d3de596492 -> trunk/c5ec5458a547f7a774468ea0eb2258d3de596492 2025-08-14T21:25:14.2040799Z * [new tag] trunk/c5efc5c8a66eca84865015058b3221013ebfe685 -> trunk/c5efc5c8a66eca84865015058b3221013ebfe685 2025-08-14T21:25:14.2041198Z * [new tag] trunk/c6563341208003f64c131854a9cf029555f786d2 -> trunk/c6563341208003f64c131854a9cf029555f786d2 2025-08-14T21:25:14.2041553Z * [new tag] trunk/c6d78d4dbda53837d298d23a5fbc09af90a42d9e -> trunk/c6d78d4dbda53837d298d23a5fbc09af90a42d9e 2025-08-14T21:25:14.2041919Z * [new tag] trunk/c8205cb35435f39d2c26f6c94b45e4adeb6dcb23 -> trunk/c8205cb35435f39d2c26f6c94b45e4adeb6dcb23 2025-08-14T21:25:14.2042261Z * [new tag] trunk/c859ba7114b1fcb49527e090745fa17091d1f8d5 -> trunk/c859ba7114b1fcb49527e090745fa17091d1f8d5 2025-08-14T21:25:14.2042565Z * [new tag] trunk/c86040a8e68f754b90a84099187d3624954c7f36 -> trunk/c86040a8e68f754b90a84099187d3624954c7f36 2025-08-14T21:25:14.2043307Z * [new tag] trunk/c9671dc865aa0fc1cb86df754e355b44d8e02bb4 -> trunk/c9671dc865aa0fc1cb86df754e355b44d8e02bb4 2025-08-14T21:25:14.2043592Z * [new tag] trunk/ca7315c17162ea21b1ca5ba23f4bf6168766c7b9 -> trunk/ca7315c17162ea21b1ca5ba23f4bf6168766c7b9 2025-08-14T21:25:14.2043854Z * [new tag] trunk/cae2b5e3d223829bdc553fc8601df4b1c1554cff -> trunk/cae2b5e3d223829bdc553fc8601df4b1c1554cff 2025-08-14T21:25:14.2044103Z * [new tag] trunk/cbffde774557752cf20447d42d99ec6102673c31 -> trunk/cbffde774557752cf20447d42d99ec6102673c31 2025-08-14T21:25:14.2044553Z * [new tag] trunk/cd8d8c18f5bafdc1c73d5ac0129e7b4d76ab45bc -> trunk/cd8d8c18f5bafdc1c73d5ac0129e7b4d76ab45bc 2025-08-14T21:25:14.2045093Z * [new tag] trunk/cf0a0dcb0afa5e84b95461cc542f862b51ca96bf -> trunk/cf0a0dcb0afa5e84b95461cc542f862b51ca96bf 2025-08-14T21:25:14.2045481Z * [new tag] trunk/cf4964be68fa9f4ffc334f01cce42d7424b1cc81 -> trunk/cf4964be68fa9f4ffc334f01cce42d7424b1cc81 2025-08-14T21:25:14.2046049Z * [new tag] trunk/d0e2240f680ea2a553f7ee8188f52482e130bfd0 -> trunk/d0e2240f680ea2a553f7ee8188f52482e130bfd0 2025-08-14T21:25:14.2046614Z * [new tag] trunk/d1950d4bb5cba8fb6b23e4d283eea5b9801737e2 -> trunk/d1950d4bb5cba8fb6b23e4d283eea5b9801737e2 2025-08-14T21:25:14.2047133Z * [new tag] trunk/d20c4c20e61adecf00335c4d8c22eb1ace472cd3 -> trunk/d20c4c20e61adecf00335c4d8c22eb1ace472cd3 2025-08-14T21:25:14.2048492Z * [new tag] trunk/d25c4f954d599ea512e2f70cd6df101c21479d4c -> trunk/d25c4f954d599ea512e2f70cd6df101c21479d4c 2025-08-14T21:25:14.2048915Z * [new tag] trunk/d3d359dbafa89173a371e2637f22b47398e94a24 -> trunk/d3d359dbafa89173a371e2637f22b47398e94a24 2025-08-14T21:25:14.2049262Z * [new tag] trunk/d46768db04499d07a5b0db984112a6d1b7d3b0c1 -> trunk/d46768db04499d07a5b0db984112a6d1b7d3b0c1 2025-08-14T21:25:14.2049554Z * [new tag] trunk/d4c1a08c89f37d249a0146ff511c82ecc5c53b8f -> trunk/d4c1a08c89f37d249a0146ff511c82ecc5c53b8f 2025-08-14T21:25:14.2049935Z * [new tag] trunk/d556586448f3caab85673c7da0978fe31c7748f7 -> trunk/d556586448f3caab85673c7da0978fe31c7748f7 2025-08-14T21:25:14.2051400Z * [new tag] trunk/d670304001429a1a833255a918ed788d7ec4989a -> trunk/d670304001429a1a833255a918ed788d7ec4989a 2025-08-14T21:25:14.2051809Z * [new tag] trunk/d6786741a77aba200c78002646cc069b7a1799b0 -> trunk/d6786741a77aba200c78002646cc069b7a1799b0 2025-08-14T21:25:14.2052305Z * [new tag] trunk/d68c323692dedcbb74e670801e3502944fd790ff -> trunk/d68c323692dedcbb74e670801e3502944fd790ff 2025-08-14T21:25:14.2052948Z * [new tag] trunk/d8cb3db5339b45e4b745b2b883ef3ecde9843e2c -> trunk/d8cb3db5339b45e4b745b2b883ef3ecde9843e2c 2025-08-14T21:25:14.2053269Z * [new tag] trunk/da1f608ca33f3062535d0a4866d95db19e72fcbd -> trunk/da1f608ca33f3062535d0a4866d95db19e72fcbd 2025-08-14T21:25:14.2053563Z * [new tag] trunk/db0b7f1cc9bb3fe71aaf8b964a644147ae8e1c35 -> trunk/db0b7f1cc9bb3fe71aaf8b964a644147ae8e1c35 2025-08-14T21:25:14.2054211Z * [new tag] trunk/db32b60662b2f2bdcad980127d5dc4b66b02a7e4 -> trunk/db32b60662b2f2bdcad980127d5dc4b66b02a7e4 2025-08-14T21:25:14.2054775Z * [new tag] trunk/db763b17175553ba09637362eb9773a91997a7ad -> trunk/db763b17175553ba09637362eb9773a91997a7ad 2025-08-14T21:25:14.2057960Z * [new tag] trunk/db78943a1ca13a32a3d6045eb15e2b719ee13a2f -> trunk/db78943a1ca13a32a3d6045eb15e2b719ee13a2f 2025-08-14T21:25:14.2058250Z * [new tag] trunk/dc0d18e023d9b7e314ebba0f234b6cb1579dbcfd -> trunk/dc0d18e023d9b7e314ebba0f234b6cb1579dbcfd 2025-08-14T21:25:14.2058486Z * [new tag] trunk/dd21c8a578038ab2841a7ba809a06921093ac9d8 -> trunk/dd21c8a578038ab2841a7ba809a06921093ac9d8 2025-08-14T21:25:14.2058709Z * [new tag] trunk/deea71a90e05eb320c04bebfead5317746637f0d -> trunk/deea71a90e05eb320c04bebfead5317746637f0d 2025-08-14T21:25:14.2058932Z * [new tag] trunk/df55ec7d4b35f6d21691e9dd41c82f27de762948 -> trunk/df55ec7d4b35f6d21691e9dd41c82f27de762948 2025-08-14T21:25:14.2059147Z * [new tag] trunk/e1cf0d496ea85d1807c8c740f296e77bf7bdc1df -> trunk/e1cf0d496ea85d1807c8c740f296e77bf7bdc1df 2025-08-14T21:25:14.2059361Z * [new tag] trunk/e248719ac03c103767ab72034f6b9fd56855bf98 -> trunk/e248719ac03c103767ab72034f6b9fd56855bf98 2025-08-14T21:25:14.2059588Z * [new tag] trunk/e49762026070f66be41bfa6537fbcf9bfc24e558 -> trunk/e49762026070f66be41bfa6537fbcf9bfc24e558 2025-08-14T21:25:14.2060260Z * [new tag] trunk/e4de93f6a3e342bab34d3757cf90ec0ccc87e168 -> trunk/e4de93f6a3e342bab34d3757cf90ec0ccc87e168 2025-08-14T21:25:14.2060516Z * [new tag] trunk/e619c6bb90b9dedaccd3cbeed86a288993a4e33f -> trunk/e619c6bb90b9dedaccd3cbeed86a288993a4e33f 2025-08-14T21:25:14.2060983Z * [new tag] trunk/e63c2b21c186a7d2ab8a8953b8aa1535f2e96e58 -> trunk/e63c2b21c186a7d2ab8a8953b8aa1535f2e96e58 2025-08-14T21:25:14.2062161Z * [new tag] trunk/e7152ff8a6a929a0db7f3f4a72a5b6d471769cd3 -> trunk/e7152ff8a6a929a0db7f3f4a72a5b6d471769cd3 2025-08-14T21:25:14.2062544Z * [new tag] trunk/e96c7c4bb0f6aeae2ab3b6f040f7d67edbec199a -> trunk/e96c7c4bb0f6aeae2ab3b6f040f7d67edbec199a 2025-08-14T21:25:14.2062915Z * [new tag] trunk/e9eb2096a59a79e7a94c3e28a0715e040369f34c -> trunk/e9eb2096a59a79e7a94c3e28a0715e040369f34c 2025-08-14T21:25:14.2063489Z * [new tag] trunk/eac2d9d695a32dd456050f45cac35134ec3809f4 -> trunk/eac2d9d695a32dd456050f45cac35134ec3809f4 2025-08-14T21:25:14.2065771Z * [new tag] trunk/ecde76c764752540edf9ef62a97936c86d984b17 -> trunk/ecde76c764752540edf9ef62a97936c86d984b17 2025-08-14T21:25:14.2066051Z * [new tag] trunk/ecea81117b2fdc52907c97b3c32d779e07b5d55b -> trunk/ecea81117b2fdc52907c97b3c32d779e07b5d55b 2025-08-14T21:25:14.2066298Z * [new tag] trunk/edaa151d0d5a4e75fbec9843f49cc78770eb61fb -> trunk/edaa151d0d5a4e75fbec9843f49cc78770eb61fb 2025-08-14T21:25:14.2066523Z * [new tag] trunk/ee1b0412b919dfb358d5a697b3be49621497fbc2 -> trunk/ee1b0412b919dfb358d5a697b3be49621497fbc2 2025-08-14T21:25:14.2066736Z * [new tag] trunk/ee1fb43450c2e985657f95a91b68328d6f20f24e -> trunk/ee1fb43450c2e985657f95a91b68328d6f20f24e 2025-08-14T21:25:14.2067099Z * [new tag] trunk/ee89cc7a0acd69de25f98fe4ef828546db7b444c -> trunk/ee89cc7a0acd69de25f98fe4ef828546db7b444c 2025-08-14T21:25:14.2067658Z * [new tag] trunk/ee9f8ba11d664b871a9e0c7933fdc8571635b78c -> trunk/ee9f8ba11d664b871a9e0c7933fdc8571635b78c 2025-08-14T21:25:14.2068167Z * [new tag] trunk/eed9dbf70f43ee529fec78ac00ed9a4fd74c6e76 -> trunk/eed9dbf70f43ee529fec78ac00ed9a4fd74c6e76 2025-08-14T21:25:14.2068691Z * [new tag] trunk/f077c2402e4eb5b0ed562b4ee5b7a0503f26ef94 -> trunk/f077c2402e4eb5b0ed562b4ee5b7a0503f26ef94 2025-08-14T21:25:14.2069388Z * [new tag] trunk/f0980fc0bbd656d6c02d23ad97e945353b314f35 -> trunk/f0980fc0bbd656d6c02d23ad97e945353b314f35 2025-08-14T21:25:14.2070132Z * [new tag] trunk/f15ada5c6fad97a7dcbfa4673f067b6942dda640 -> trunk/f15ada5c6fad97a7dcbfa4673f067b6942dda640 2025-08-14T21:25:14.2070584Z * [new tag] trunk/f27232a2134150cb5e55d26a74d8c36c6a961ca5 -> trunk/f27232a2134150cb5e55d26a74d8c36c6a961ca5 2025-08-14T21:25:14.2071067Z * [new tag] trunk/f33ce40bc062a281e1a1f57e8c1926d0a7d155cc -> trunk/f33ce40bc062a281e1a1f57e8c1926d0a7d155cc 2025-08-14T21:25:14.2071629Z * [new tag] trunk/f341077ce4710172da20cfad916ee37159bfe9fe -> trunk/f341077ce4710172da20cfad916ee37159bfe9fe 2025-08-14T21:25:14.2072210Z * [new tag] trunk/f3a4d742ece08de4cb0e59dcc62e0093a7d0b0c7 -> trunk/f3a4d742ece08de4cb0e59dcc62e0093a7d0b0c7 2025-08-14T21:25:14.2072808Z * [new tag] trunk/f3f159ff8c4bad2edec99c68a941c628e983d04c -> trunk/f3f159ff8c4bad2edec99c68a941c628e983d04c 2025-08-14T21:25:14.2073395Z * [new tag] trunk/f60454cce8b93e5bbf67f2f3c88c8ac01ed65457 -> trunk/f60454cce8b93e5bbf67f2f3c88c8ac01ed65457 2025-08-14T21:25:14.2073989Z * [new tag] trunk/f7b2f3314cf7aede67d5fa5c75e4243208484344 -> trunk/f7b2f3314cf7aede67d5fa5c75e4243208484344 2025-08-14T21:25:14.2075360Z * [new tag] trunk/f8f0414a5983ff481a2188e0c18594150430c8c5 -> trunk/f8f0414a5983ff481a2188e0c18594150430c8c5 2025-08-14T21:25:14.2075665Z * [new tag] trunk/f95b58c2844b3444cd8446fed8570729dc4216eb -> trunk/f95b58c2844b3444cd8446fed8570729dc4216eb 2025-08-14T21:25:14.2076311Z * [new tag] trunk/f990490a23815ea6ee27e487c70ba2cf513ba43d -> trunk/f990490a23815ea6ee27e487c70ba2cf513ba43d 2025-08-14T21:25:14.2076709Z * [new tag] trunk/fb887c3bb588cfe782615e67f6c26db636b8539b -> trunk/fb887c3bb588cfe782615e67f6c26db636b8539b 2025-08-14T21:25:14.2078349Z * [new tag] trunk/fc25c68f20f772290927a7031b998b92615259cf -> trunk/fc25c68f20f772290927a7031b998b92615259cf 2025-08-14T21:25:14.2078727Z * [new tag] trunk/fc80f6859e0ccf66513a40f04b9e735e759d4ddb -> trunk/fc80f6859e0ccf66513a40f04b9e735e759d4ddb 2025-08-14T21:25:14.2079221Z * [new tag] trunk/fdfd69bb05488d76123db9cc1cdd90ac4137bbfb -> trunk/fdfd69bb05488d76123db9cc1cdd90ac4137bbfb 2025-08-14T21:25:14.2079828Z * [new tag] trunk/fe3f5fe4ea2ff6f56406dc5d954636ebb08d0a08 -> trunk/fe3f5fe4ea2ff6f56406dc5d954636ebb08d0a08 2025-08-14T21:25:14.2080384Z * [new tag] trunk/fea7e9dd37c02c334b130f6624af6163fde6b2ab -> trunk/fea7e9dd37c02c334b130f6624af6163fde6b2ab 2025-08-14T21:25:14.2080894Z * [new tag] trunk/ff0d56d03592aa03f3ced8359241d21df1783393 -> trunk/ff0d56d03592aa03f3ced8359241d21df1783393 2025-08-14T21:25:14.2081469Z * [new tag] v0.1.1 -> v0.1.1 2025-08-14T21:25:14.2081894Z * [new tag] v0.1.10 -> v0.1.10 2025-08-14T21:25:14.2082752Z * [new tag] v0.1.11 -> v0.1.11 2025-08-14T21:25:14.2082865Z * [new tag] v0.1.12 -> v0.1.12 2025-08-14T21:25:14.2085783Z * [new tag] v0.1.2 -> v0.1.2 2025-08-14T21:25:14.2085915Z * [new tag] v0.1.3 -> v0.1.3 2025-08-14T21:25:14.2086160Z * [new tag] v0.1.4 -> v0.1.4 2025-08-14T21:25:14.2086264Z * [new tag] v0.1.5 -> v0.1.5 2025-08-14T21:25:14.2086361Z * [new tag] v0.1.6 -> v0.1.6 2025-08-14T21:25:14.2086454Z * [new tag] v0.1.7 -> v0.1.7 2025-08-14T21:25:14.2086581Z * [new tag] v0.1.8 -> v0.1.8 2025-08-14T21:25:14.2086977Z * [new tag] v0.1.9 -> v0.1.9 2025-08-14T21:25:14.2087577Z * [new tag] v0.2.0 -> v0.2.0 2025-08-14T21:25:14.2088185Z * [new tag] v0.3.0 -> v0.3.0 2025-08-14T21:25:14.2088613Z * [new tag] v0.3.1 -> v0.3.1 2025-08-14T21:25:14.2091341Z * [new tag] v0.4.0 -> v0.4.0 2025-08-14T21:25:14.2091466Z * [new tag] v0.4.1 -> v0.4.1 2025-08-14T21:25:14.2091585Z * [new tag] v1.0.0 -> v1.0.0 2025-08-14T21:25:14.2091696Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-08-14T21:25:14.2091805Z * [new tag] v1.0.1 -> v1.0.1 2025-08-14T21:25:14.2092346Z * [new tag] v1.0rc0 -> v1.0rc0 2025-08-14T21:25:14.2092545Z * [new tag] v1.0rc1 -> v1.0rc1 2025-08-14T21:25:14.2093429Z * [new tag] v1.1.0 -> v1.1.0 2025-08-14T21:25:14.2093544Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-08-14T21:25:14.2097884Z * [new tag] v1.10.0 -> v1.10.0 2025-08-14T21:25:14.2098040Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-08-14T21:25:14.2098142Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-08-14T21:25:14.2098252Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-08-14T21:25:14.2098503Z * [new tag] v1.10.1 -> v1.10.1 2025-08-14T21:25:14.2098600Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-08-14T21:25:14.2098704Z * [new tag] v1.10.2 -> v1.10.2 2025-08-14T21:25:14.2098800Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-08-14T21:25:14.2098893Z * [new tag] v1.11.0 -> v1.11.0 2025-08-14T21:25:14.2098997Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-08-14T21:25:14.2099859Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-08-14T21:25:14.2100157Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-08-14T21:25:14.2101151Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-08-14T21:25:14.2101373Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-08-14T21:25:14.2101784Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-08-14T21:25:14.2102185Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-08-14T21:25:14.2104850Z * [new tag] v1.12.0 -> v1.12.0 2025-08-14T21:25:14.2105003Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-08-14T21:25:14.2105106Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-08-14T21:25:14.2105210Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-08-14T21:25:14.2105306Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-08-14T21:25:14.2105881Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-08-14T21:25:14.2106350Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-08-14T21:25:14.2106736Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-08-14T21:25:14.2107293Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-08-14T21:25:14.2107497Z * [new tag] v1.12.1 -> v1.12.1 2025-08-14T21:25:14.2108464Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-08-14T21:25:14.2108583Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-08-14T21:25:14.2111398Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-08-14T21:25:14.2111540Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-08-14T21:25:14.2111640Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-08-14T21:25:14.2111743Z * [new tag] v1.13.0 -> v1.13.0 2025-08-14T21:25:14.2111849Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-08-14T21:25:14.2112262Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-08-14T21:25:14.2113120Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-08-14T21:25:14.2114195Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-08-14T21:25:14.2114421Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-08-14T21:25:14.2114722Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-08-14T21:25:14.2115712Z * [new tag] v1.13.1 -> v1.13.1 2025-08-14T21:25:14.2116352Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-08-14T21:25:14.2116480Z * [new tag] v1.2.0 -> v1.2.0 2025-08-14T21:25:14.2117757Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-08-14T21:25:14.2117878Z * [new tag] v1.3.0 -> v1.3.0 2025-08-14T21:25:14.2118212Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-08-14T21:25:14.2119107Z * [new tag] v1.3.1 -> v1.3.1 2025-08-14T21:25:14.2119417Z * [new tag] v1.4.0 -> v1.4.0 2025-08-14T21:25:14.2119981Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-08-14T21:25:14.2120324Z * [new tag] v1.4.1 -> v1.4.1 2025-08-14T21:25:14.2121368Z * [new tag] v1.5.0 -> v1.5.0 2025-08-14T21:25:14.2121568Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-08-14T21:25:14.2122627Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-08-14T21:25:14.2122855Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-08-14T21:25:14.2124378Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-08-14T21:25:14.2124618Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-08-14T21:25:14.2125004Z * [new tag] v1.5.1 -> v1.5.1 2025-08-14T21:25:14.2125125Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-08-14T21:25:14.2125562Z * [new tag] v1.6.0 -> v1.6.0 2025-08-14T21:25:14.2125903Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-08-14T21:25:14.2126723Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-08-14T21:25:14.2126951Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-08-14T21:25:14.2128016Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-08-14T21:25:14.2128250Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-08-14T21:25:14.2128732Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-08-14T21:25:14.2128988Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-08-14T21:25:14.2129999Z * [new tag] v1.7.0 -> v1.7.0 2025-08-14T21:25:14.2132512Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-08-14T21:25:14.2132622Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-08-14T21:25:14.2132726Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-08-14T21:25:14.2132816Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-08-14T21:25:14.2132906Z * [new tag] v1.7.1 -> v1.7.1 2025-08-14T21:25:14.2133004Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-08-14T21:25:14.2133432Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-08-14T21:25:14.2133539Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-08-14T21:25:14.2141702Z * [new tag] v1.8.0 -> v1.8.0 2025-08-14T21:25:14.2141845Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-08-14T21:25:14.2141944Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-08-14T21:25:14.2142077Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-08-14T21:25:14.2142197Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-08-14T21:25:14.2142298Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-08-14T21:25:14.2142394Z * [new tag] v1.8.1 -> v1.8.1 2025-08-14T21:25:14.2142490Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-08-14T21:25:14.2142590Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-08-14T21:25:14.2142682Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-08-14T21:25:14.2142777Z * [new tag] v1.8.2 -> v1.8.2 2025-08-14T21:25:14.2142879Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-08-14T21:25:14.2142969Z * [new tag] v1.9.0 -> v1.9.0 2025-08-14T21:25:14.2143321Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-08-14T21:25:14.2143564Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-08-14T21:25:14.2143657Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-08-14T21:25:14.2143757Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-08-14T21:25:14.2143845Z * [new tag] v1.9.1 -> v1.9.1 2025-08-14T21:25:14.2143943Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-08-14T21:25:14.2144033Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-08-14T21:25:14.2144128Z * [new tag] v2.0.0 -> v2.0.0 2025-08-14T21:25:14.2144252Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-08-14T21:25:14.2152119Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-08-14T21:25:14.2152288Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-08-14T21:25:14.2152394Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-08-14T21:25:14.2152491Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-08-14T21:25:14.2152595Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-08-14T21:25:14.2152698Z * [new tag] v2.0.1 -> v2.0.1 2025-08-14T21:25:14.2152804Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-08-14T21:25:14.2152897Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-08-14T21:25:14.2152989Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-08-14T21:25:14.2153088Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-08-14T21:25:14.2153180Z * [new tag] v2.1.0 -> v2.1.0 2025-08-14T21:25:14.2153272Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-08-14T21:25:14.2153503Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-08-14T21:25:14.2153608Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-08-14T21:25:14.2153710Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-08-14T21:25:14.2153804Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-08-14T21:25:14.2153905Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-08-14T21:25:14.2154004Z * [new tag] v2.1.1 -> v2.1.1 2025-08-14T21:25:14.2154096Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-08-14T21:25:14.2154196Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-08-14T21:25:14.2154288Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-08-14T21:25:14.2154438Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-08-14T21:25:14.2155438Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-08-14T21:25:14.2155651Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-08-14T21:25:14.2156107Z * [new tag] v2.1.2 -> v2.1.2 2025-08-14T21:25:14.2157061Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-08-14T21:25:14.2157374Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-08-14T21:25:14.2157837Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-08-14T21:25:14.2160960Z * [new tag] v2.2.0 -> v2.2.0 2025-08-14T21:25:14.2161112Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-08-14T21:25:14.2161213Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-08-14T21:25:14.2161307Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-08-14T21:25:14.2161426Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-08-14T21:25:14.2161675Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-08-14T21:25:14.2161951Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-08-14T21:25:14.2162060Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-08-14T21:25:14.2162456Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-08-14T21:25:14.2164006Z * [new tag] v2.2.1 -> v2.2.1 2025-08-14T21:25:14.2164300Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-08-14T21:25:14.2164443Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-08-14T21:25:14.2164626Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-08-14T21:25:14.2165017Z * [new tag] v2.2.2 -> v2.2.2 2025-08-14T21:25:14.2166941Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-08-14T21:25:14.2167213Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-08-14T21:25:14.2167621Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-08-14T21:25:14.2167735Z * [new tag] v2.3.0 -> v2.3.0 2025-08-14T21:25:14.2167842Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-08-14T21:25:14.2168210Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-08-14T21:25:14.2170858Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-08-14T21:25:14.2171168Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-08-14T21:25:14.2171309Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-08-14T21:25:14.2171497Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-08-14T21:25:14.2171633Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-08-14T21:25:14.2171890Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-08-14T21:25:14.2172769Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-08-14T21:25:14.2172909Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-08-14T21:25:14.2173452Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-08-14T21:25:14.2173758Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-08-14T21:25:14.2174227Z * [new tag] v2.3.1 -> v2.3.1 2025-08-14T21:25:14.2178990Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-08-14T21:25:14.2184141Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-08-14T21:25:14.2188546Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-08-14T21:25:14.2190485Z * [new tag] v2.4.0 -> v2.4.0 2025-08-14T21:25:14.2190632Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-08-14T21:25:14.2190742Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-08-14T21:25:14.2190849Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-08-14T21:25:14.2190945Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-08-14T21:25:14.2191047Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-08-14T21:25:14.2191143Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-08-14T21:25:14.2191250Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-08-14T21:25:14.2191353Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-08-14T21:25:14.2191449Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-08-14T21:25:14.2191554Z * [new tag] v2.4.1 -> v2.4.1 2025-08-14T21:25:14.2191657Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-08-14T21:25:14.2191890Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-08-14T21:25:14.2191994Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-08-14T21:25:14.2192090Z * [new tag] v2.5.0 -> v2.5.0 2025-08-14T21:25:14.2192187Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-08-14T21:25:14.2192304Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-08-14T21:25:14.2192404Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-08-14T21:25:14.2192511Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-08-14T21:25:14.2192607Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-08-14T21:25:14.2192702Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-08-14T21:25:14.2192806Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-08-14T21:25:14.2192909Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-08-14T21:25:14.2193018Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-08-14T21:25:14.2193115Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-08-14T21:25:14.2193216Z * [new tag] v2.5.1 -> v2.5.1 2025-08-14T21:25:14.2193322Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-08-14T21:25:14.2193416Z * [new tag] v2.6.0 -> v2.6.0 2025-08-14T21:25:14.2193514Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-08-14T21:25:14.2193642Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-08-14T21:25:14.2194975Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-08-14T21:25:14.2195434Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-08-14T21:25:14.2195953Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-08-14T21:25:14.2196611Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-08-14T21:25:14.2201912Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-08-14T21:25:14.2202193Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-08-14T21:25:14.2202332Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-08-14T21:25:14.2202441Z * [new tag] v2.7.0 -> v2.7.0 2025-08-14T21:25:14.2202686Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-08-14T21:25:14.2202795Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-08-14T21:25:14.2202891Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-08-14T21:25:14.2202994Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-08-14T21:25:14.2203104Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-08-14T21:25:14.2203345Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-08-14T21:25:14.2203974Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-08-14T21:25:14.2204149Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-08-14T21:25:14.2205564Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-08-14T21:25:14.2205868Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-08-14T21:25:14.2206001Z * [new tag] v2.7.1 -> v2.7.1 2025-08-14T21:25:14.2206340Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-08-14T21:25:14.2208529Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-08-14T21:25:14.2208837Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-08-14T21:25:14.2208971Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-08-14T21:25:14.2209494Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-08-14T21:25:14.2209610Z * [new tag] v2.8.0 -> v2.8.0 2025-08-14T21:25:14.2209973Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-08-14T21:25:14.2210908Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-08-14T21:25:14.2211251Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-08-14T21:25:14.2214008Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-08-14T21:25:14.2214137Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-08-14T21:25:14.2214239Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-08-14T21:25:14.2214331Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-08-14T21:25:14.2214430Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-08-14T21:25:14.2214878Z * [new tag] whc_flight_1 -> whc_flight_1 2025-08-14T21:25:14.2215191Z * [new tag] whc_flight_2 -> whc_flight_2 2025-08-14T21:25:14.2219564Z * [new tag] whc_flight_4 -> whc_flight_4 2025-08-14T21:25:14.2682754Z [command]/usr/bin/git rev-parse --verify --quiet 1fc683cf17c8c673044538d10266c00f92987be2^{object} 2025-08-14T21:25:14.2714961Z 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:25:14.2715662Z ##[endgroup] 2025-08-14T21:25:14.2715873Z ##[group]Determining the checkout info 2025-08-14T21:25:14.2716396Z ##[endgroup] 2025-08-14T21:25:14.2724973Z [command]/usr/bin/git sparse-checkout disable 2025-08-14T21:25:14.2772125Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-08-14T21:25:14.2797357Z ##[group]Checking out the ref 2025-08-14T21:25:14.2797783Z [command]/usr/bin/git checkout --progress --force 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:25:15.3100755Z Note: switching to '1fc683cf17c8c673044538d10266c00f92987be2'. 2025-08-14T21:25:15.3101080Z 2025-08-14T21:25:15.3101251Z You are in 'detached HEAD' state. You can look around, make experimental 2025-08-14T21:25:15.3101610Z changes and commit them, and you can discard any commits you make in this 2025-08-14T21:25:15.3101988Z state without impacting any branches by switching back to a branch. 2025-08-14T21:25:15.3102232Z 2025-08-14T21:25:15.3102391Z If you want to create a new branch to retain commits you create, you may 2025-08-14T21:25:15.3102751Z do so (now or later) by using -c with the switch command. Example: 2025-08-14T21:25:15.3102949Z 2025-08-14T21:25:15.3103051Z git switch -c 2025-08-14T21:25:15.3103209Z 2025-08-14T21:25:15.3103293Z Or undo this operation with: 2025-08-14T21:25:15.3103429Z 2025-08-14T21:25:15.3103499Z git switch - 2025-08-14T21:25:15.3103607Z 2025-08-14T21:25:15.3103766Z Turn off this advice by setting config variable advice.detachedHead to false 2025-08-14T21:25:15.3103990Z 2025-08-14T21:25:15.3104253Z HEAD is now at 1fc683cf17c [Inductor] Allow indexing a flexible layout for extract_input_node_reduction_ranges (#160645) 2025-08-14T21:25:15.3152156Z ##[endgroup] 2025-08-14T21:25:15.3152605Z ##[group]Setting up auth for fetching submodules 2025-08-14T21:25:15.3157122Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-08-14T21:25:15.3210813Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-08-14T21:25:15.3251138Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-08-14T21:25:15.3269741Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-08-14T21:25:15.3293696Z ##[endgroup] 2025-08-14T21:25:15.3294131Z ##[group]Fetching submodules 2025-08-14T21:25:15.3297472Z [command]/usr/bin/git submodule sync --recursive 2025-08-14T21:25:15.3622473Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-08-14T21:25:15.3929133Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-08-14T21:25:15.3930170Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-08-14T21:25:15.3930821Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-08-14T21:25:15.3931425Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-08-14T21:25:15.4193889Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-08-14T21:25:15.4194765Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-08-14T21:25:15.4195673Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-08-14T21:25:15.4196664Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-08-14T21:25:15.4199297Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-08-14T21:25:15.4201344Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-08-14T21:25:15.4203428Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-08-14T21:25:15.4208046Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-08-14T21:25:15.4227115Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-08-14T21:25:15.4229782Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-08-14T21:25:15.4230317Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-08-14T21:25:15.4230894Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-08-14T21:25:15.4231576Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-08-14T21:25:15.4232141Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-08-14T21:25:15.4232995Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:25:15.4235777Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-08-14T21:25:15.4243024Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-08-14T21:25:15.4246692Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-08-14T21:25:15.4254640Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-08-14T21:25:15.4255173Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-08-14T21:25:15.4259273Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-08-14T21:25:15.4259831Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-08-14T21:25:15.4263512Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-08-14T21:25:15.4268388Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-08-14T21:25:15.4269237Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-08-14T21:25:15.4270669Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-08-14T21:25:15.4274220Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-08-14T21:25:15.4276680Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-08-14T21:25:15.4288036Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-08-14T21:25:15.4292323Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-08-14T21:25:15.4296955Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-08-14T21:25:15.4298108Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-08-14T21:25:15.4301641Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-08-14T21:25:15.4340511Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-08-14T21:25:15.6667478Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-08-14T21:25:15.6667978Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-08-14T21:25:15.6668410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-08-14T21:25:15.6669118Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-08-14T21:25:15.6669590Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-08-14T21:25:15.6687363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-08-14T21:25:15.8479741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-08-14T21:25:15.8480994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-08-14T21:25:15.8591979Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-08-14T21:25:16.8977787Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-08-14T21:25:16.8978678Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-08-14T21:25:16.8979564Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-08-14T21:25:16.8980692Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-08-14T21:25:16.8981690Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-08-14T21:25:16.8982638Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-08-14T21:25:16.8983608Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-08-14T21:25:16.8984529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-08-14T21:25:16.8985631Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-08-14T21:25:16.8988290Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-08-14T21:25:16.8989335Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-08-14T21:25:16.8990319Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-08-14T21:25:16.8991614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-08-14T21:25:16.8992558Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-08-14T21:25:16.9978589Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-08-14T21:25:17.1669068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-08-14T21:25:17.2125188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-08-14T21:25:28.9067550Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-08-14T21:25:28.9069004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-08-14T21:25:28.9069511Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-08-14T21:25:28.9069969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-08-14T21:25:28.9070425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-08-14T21:25:28.9070892Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-08-14T21:25:28.9071348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-08-14T21:25:28.9071822Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-08-14T21:25:28.9072357Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-08-14T21:25:28.9072790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-08-14T21:25:28.9198995Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-08-14T21:25:28.9315806Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-08-14T21:25:28.9403688Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-08-14T21:25:28.9608633Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-08-14T21:25:29.0296031Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07' 2025-08-14T21:25:29.0762342Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-08-14T21:25:29.6157883Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-08-14T21:25:29.7427612Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-08-14T21:25:29.7447425Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:25:29.7468293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-08-14T21:25:33.0620234Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-08-14T21:25:33.0825972Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-08-14T21:25:33.3263188Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-08-14T21:25:33.3710079Z Submodule path 'third_party/cpp-httplib': checked out '3af7f2c16147f3fbc6e4d717032daf505dc1652c' 2025-08-14T21:25:33.4580495Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d' 2025-08-14T21:25:33.4968876Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6' 2025-08-14T21:25:34.0189672Z Submodule path 'third_party/cutlass': checked out 'e51efbfe18fe4f4cbb66ab814c55bf4aa0185491' 2025-08-14T21:25:34.1342460Z Submodule path 'third_party/fbgemm': checked out '21c7d30c526c0f1ad873ecc632dca6cfa8a69067' 2025-08-14T21:25:34.1360797Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-08-14T21:25:34.1366059Z Submodule 'external/composable_kernel' (https://github.com/jwfromm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:25:34.1370836Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:25:34.1376485Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-08-14T21:25:34.1378127Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-08-14T21:25:34.1378914Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:25:34.1379613Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-08-14T21:25:34.1395770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-08-14T21:25:35.2894334Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-08-14T21:25:35.2894936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-08-14T21:25:35.2895480Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-08-14T21:25:35.3207719Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-08-14T21:25:35.4213753Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-08-14T21:25:36.4067522Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-08-14T21:25:40.5542763Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-08-14T21:25:40.7654624Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out 'b1281b8b08d973a7064f864f47eeb30f3e2596e9' 2025-08-14T21:25:40.8541233Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-08-14T21:25:41.3706078Z Submodule path 'third_party/fbgemm/external/cutlass': checked out 'b40777404c174b9694a870bff5c13ce6b7f656ad' 2025-08-14T21:25:41.4116676Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-08-14T21:25:41.4239493Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out 'a4337c69fe0e2552a7b7b0669178926beeed828c' 2025-08-14T21:25:41.5159229Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-08-14T21:25:41.5765778Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-08-14T21:25:41.5781712Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:25:41.5786363Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:25:41.5806344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-08-14T21:25:44.7288442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-08-14T21:25:44.9150924Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-08-14T21:25:45.3839309Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-08-14T21:25:45.4944529Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-08-14T21:25:45.5246781Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-08-14T21:25:45.5599777Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-08-14T21:25:45.5825076Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2' 2025-08-14T21:25:45.6217739Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-08-14T21:25:45.6337457Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-08-14T21:25:45.6349365Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-08-14T21:25:45.6383236Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-08-14T21:25:56.9510029Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-08-14T21:25:56.9692831Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-08-14T21:25:57.0557438Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658' 2025-08-14T21:25:57.0574530Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:25:57.0576907Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:25:57.0577967Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:25:57.0607057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-08-14T21:25:57.6932886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-08-14T21:25:58.3077708Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-08-14T21:25:58.3801183Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2025-08-14T21:25:58.3820422Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:25:58.3821700Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:25:58.3822483Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:25:58.3823261Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:25:58.3824809Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:25:58.3828343Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:25:58.3829209Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:25:58.3829991Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:25:58.3858793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-08-14T21:25:59.6703639Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-08-14T21:25:59.6704367Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-08-14T21:25:59.6705040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-08-14T21:25:59.6705698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-08-14T21:25:59.6706403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-08-14T21:25:59.6707053Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-08-14T21:25:59.7707624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-08-14T21:26:04.9635151Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-08-14T21:26:04.9791853Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-08-14T21:26:05.0121341Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-08-14T21:26:05.0254290Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-08-14T21:26:05.0266771Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:26:05.0288258Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-08-14T21:26:05.6230294Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-08-14T21:26:05.6401632Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-08-14T21:26:05.6763840Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2025-08-14T21:26:05.7637451Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-08-14T21:26:05.7786905Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-08-14T21:26:05.8144394Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2025-08-14T21:26:05.8671616Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2025-08-14T21:26:05.9045806Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7' 2025-08-14T21:26:05.9383121Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-08-14T21:26:06.0354406Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-08-14T21:26:06.3292796Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-08-14T21:26:06.3324194Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-08-14T21:26:06.3351670Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-08-14T21:26:08.1457239Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-08-14T21:26:08.2016719Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-08-14T21:26:08.2031562Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:26:08.2032551Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:26:08.2033416Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:26:08.2034127Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:26:08.2034950Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:26:08.2035840Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:26:08.2036946Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:26:08.2042744Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:26:08.2070471Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-08-14T21:26:08.6193114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-08-14T21:26:08.6193884Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-08-14T21:26:08.6194556Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-08-14T21:26:08.6195202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-08-14T21:26:08.7194774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-08-14T21:26:09.2669823Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-08-14T21:26:16.5577361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-08-14T21:26:16.7546611Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-08-14T21:26:16.7903027Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-08-14T21:26:16.8066476Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-08-14T21:26:16.9017451Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-08-14T21:26:16.9153681Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-08-14T21:26:16.9288212Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-08-14T21:26:16.9434127Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-08-14T21:26:16.9453293Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:26:16.9458155Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:26:16.9476955Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-08-14T21:26:18.8694665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-08-14T21:26:19.0874317Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-08-14T21:26:19.1293063Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-08-14T21:26:19.4654539Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-08-14T21:26:19.4767194Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-08-14T21:26:19.7015353Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-08-14T21:26:19.7032109Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:26:19.7033047Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-08-14T21:26:19.7064417Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-08-14T21:26:20.2435732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-08-14T21:26:20.6678065Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-08-14T21:26:20.7317373Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-08-14T21:26:20.7409957Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-08-14T21:26:20.7526895Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-08-14T21:26:20.7852209Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-08-14T21:26:20.8104188Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-08-14T21:26:20.8497302Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-08-14T21:26:20.8730256Z Submodule path 'third_party/tensorpipe': checked out 'dacda0567d9f23d4bc503e1c4f84aa65f33ac38a' 2025-08-14T21:26:20.8748322Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:26:20.8750166Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:26:20.8750872Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:26:20.8755273Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:26:20.8782682Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-08-14T21:26:21.8121511Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-08-14T21:26:21.8122163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-08-14T21:26:21.9121431Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-08-14T21:26:22.1058746Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-08-14T21:26:22.1193032Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-08-14T21:26:22.1854254Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-08-14T21:26:22.2105204Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-08-14T21:26:22.2121642Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:26:22.2147446Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-08-14T21:26:22.4191741Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-08-14T21:26:22.4226582Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-08-14T21:26:22.4549442Z Entering 'android/libs/fbjni' 2025-08-14T21:26:22.4591514Z Entering 'third_party/FP16' 2025-08-14T21:26:22.4632220Z Entering 'third_party/FXdiv' 2025-08-14T21:26:22.4672553Z Entering 'third_party/NNPACK' 2025-08-14T21:26:22.4715647Z Entering 'third_party/NVTX' 2025-08-14T21:26:22.4763349Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:26:22.4802905Z Entering 'third_party/XNNPACK' 2025-08-14T21:26:22.4863180Z Entering 'third_party/aiter' 2025-08-14T21:26:22.4900212Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:26:22.4952021Z Entering 'third_party/benchmark' 2025-08-14T21:26:22.4992397Z Entering 'third_party/composable_kernel' 2025-08-14T21:26:22.5047274Z Entering 'third_party/cpp-httplib' 2025-08-14T21:26:22.5086134Z Entering 'third_party/cpuinfo' 2025-08-14T21:26:22.5127593Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:26:22.5171512Z Entering 'third_party/cutlass' 2025-08-14T21:26:22.5216694Z Entering 'third_party/fbgemm' 2025-08-14T21:26:22.5257621Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:26:22.5297271Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:26:22.5343548Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:26:22.5382755Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:26:22.5429820Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:26:22.5469932Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:26:22.5513586Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:26:22.5558935Z Entering 'third_party/flash-attention' 2025-08-14T21:26:22.5598815Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:26:22.5643714Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:26:22.5693533Z Entering 'third_party/flatbuffers' 2025-08-14T21:26:22.5745386Z Entering 'third_party/fmt' 2025-08-14T21:26:22.5787448Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:26:22.5827775Z Entering 'third_party/gloo' 2025-08-14T21:26:22.5873810Z Entering 'third_party/googletest' 2025-08-14T21:26:22.5910590Z Entering 'third_party/ideep' 2025-08-14T21:26:22.5950766Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:26:22.5996232Z Entering 'third_party/ittapi' 2025-08-14T21:26:22.6035859Z Entering 'third_party/kineto' 2025-08-14T21:26:22.6072928Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:26:22.6110193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:26:22.6153827Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:26:22.6193420Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:26:22.6235187Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:26:22.6278455Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:26:22.6328210Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:26:22.6367521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:26:22.6403962Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:26:22.6447353Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:26:22.6484708Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:26:22.6529071Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:26:22.6574601Z Entering 'third_party/kleidiai' 2025-08-14T21:26:22.6616174Z Entering 'third_party/mimalloc' 2025-08-14T21:26:22.6661216Z Entering 'third_party/nlohmann' 2025-08-14T21:26:22.6701308Z Entering 'third_party/onnx' 2025-08-14T21:26:22.6754738Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:26:22.6796773Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:26:22.6842625Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:26:22.6876360Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:26:22.6915410Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:26:22.6960166Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:26:22.6994618Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:26:22.7032202Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:26:22.7071938Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:26:22.7107742Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:26:22.7151231Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:26:22.7196830Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:26:22.7256757Z Entering 'third_party/pocketfft' 2025-08-14T21:26:22.7293948Z Entering 'third_party/protobuf' 2025-08-14T21:26:22.7333871Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:26:22.7376097Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:26:22.7421888Z Entering 'third_party/psimd' 2025-08-14T21:26:22.7464025Z Entering 'third_party/pthreadpool' 2025-08-14T21:26:22.7506006Z Entering 'third_party/pybind11' 2025-08-14T21:26:22.7551506Z Entering 'third_party/python-peachpy' 2025-08-14T21:26:22.7590192Z Entering 'third_party/sleef' 2025-08-14T21:26:22.7631159Z Entering 'third_party/tensorpipe' 2025-08-14T21:26:22.7669125Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:26:22.7709561Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:26:22.7751000Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:26:22.7788557Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:26:22.7830036Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:26:22.7881360Z ##[endgroup] 2025-08-14T21:26:22.7881802Z ##[group]Persisting credentials for submodules 2025-08-14T21:26:22.7890227Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-08-14T21:26:22.8200432Z Entering 'android/libs/fbjni' 2025-08-14T21:26:22.8252281Z Entering 'third_party/FP16' 2025-08-14T21:26:22.8306772Z Entering 'third_party/FXdiv' 2025-08-14T21:26:22.8359805Z Entering 'third_party/NNPACK' 2025-08-14T21:26:22.8417358Z Entering 'third_party/NVTX' 2025-08-14T21:26:22.8473684Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:26:22.8528956Z Entering 'third_party/XNNPACK' 2025-08-14T21:26:22.8597815Z Entering 'third_party/aiter' 2025-08-14T21:26:22.8654097Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:26:22.8709816Z Entering 'third_party/benchmark' 2025-08-14T21:26:22.8778983Z Entering 'third_party/composable_kernel' 2025-08-14T21:26:22.8831551Z Entering 'third_party/cpp-httplib' 2025-08-14T21:26:22.8887925Z Entering 'third_party/cpuinfo' 2025-08-14T21:26:22.8949981Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:26:22.9003665Z Entering 'third_party/cutlass' 2025-08-14T21:26:22.9066502Z Entering 'third_party/fbgemm' 2025-08-14T21:26:22.9120234Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:26:22.9175812Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:26:22.9234300Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:26:22.9289219Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:26:22.9365002Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:26:22.9413780Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:26:22.9474382Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:26:22.9534129Z Entering 'third_party/flash-attention' 2025-08-14T21:26:22.9590379Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:26:22.9647433Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:26:22.9712250Z Entering 'third_party/flatbuffers' 2025-08-14T21:26:22.9769723Z Entering 'third_party/fmt' 2025-08-14T21:26:22.9830253Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:26:22.9875376Z Entering 'third_party/gloo' 2025-08-14T21:26:22.9929898Z Entering 'third_party/googletest' 2025-08-14T21:26:22.9985034Z Entering 'third_party/ideep' 2025-08-14T21:26:23.0043205Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:26:23.0101469Z Entering 'third_party/ittapi' 2025-08-14T21:26:23.0162103Z Entering 'third_party/kineto' 2025-08-14T21:26:23.0212499Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:26:23.0272505Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:26:23.0322098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:26:23.0380587Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:26:23.0434233Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:26:23.0482792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:26:23.0543823Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:26:23.0594741Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:26:23.0650219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:26:23.0706416Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:26:23.0755149Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:26:23.0807235Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:26:23.0870316Z Entering 'third_party/kleidiai' 2025-08-14T21:26:23.0924933Z Entering 'third_party/mimalloc' 2025-08-14T21:26:23.0974547Z Entering 'third_party/nlohmann' 2025-08-14T21:26:23.1026278Z Entering 'third_party/onnx' 2025-08-14T21:26:23.1096366Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:26:23.1156357Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:26:23.1208070Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:26:23.1268861Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:26:23.1322584Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:26:23.1377026Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:26:23.1430825Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:26:23.1487509Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:26:23.1546816Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:26:23.1593867Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:26:23.1651743Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:26:23.1705603Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:26:23.1780698Z Entering 'third_party/pocketfft' 2025-08-14T21:26:23.1828685Z Entering 'third_party/protobuf' 2025-08-14T21:26:23.1882424Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:26:23.1940136Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:26:23.1994443Z Entering 'third_party/psimd' 2025-08-14T21:26:23.2053720Z Entering 'third_party/pthreadpool' 2025-08-14T21:26:23.2105537Z Entering 'third_party/pybind11' 2025-08-14T21:26:23.2170651Z Entering 'third_party/python-peachpy' 2025-08-14T21:26:23.2221721Z Entering 'third_party/sleef' 2025-08-14T21:26:23.2275106Z Entering 'third_party/tensorpipe' 2025-08-14T21:26:23.2322209Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:26:23.2370940Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:26:23.2434919Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:26:23.2484631Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:26:23.2541695Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:26:23.2614615Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-08-14T21:26:23.2921362Z Entering 'android/libs/fbjni' 2025-08-14T21:26:23.2969573Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-08-14T21:26:23.2986259Z Entering 'third_party/FP16' 2025-08-14T21:26:23.3035222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-08-14T21:26:23.3050015Z Entering 'third_party/FXdiv' 2025-08-14T21:26:23.3102380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-08-14T21:26:23.3116631Z Entering 'third_party/NNPACK' 2025-08-14T21:26:23.3164613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-08-14T21:26:23.3180537Z Entering 'third_party/NVTX' 2025-08-14T21:26:23.3230925Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-08-14T21:26:23.3246572Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:26:23.3291683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-08-14T21:26:23.3306795Z Entering 'third_party/XNNPACK' 2025-08-14T21:26:23.3357973Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-08-14T21:26:23.3384815Z Entering 'third_party/aiter' 2025-08-14T21:26:23.3429337Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-08-14T21:26:23.3450016Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:26:23.3493666Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-08-14T21:26:23.3519277Z Entering 'third_party/benchmark' 2025-08-14T21:26:23.3571554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-08-14T21:26:23.3589809Z Entering 'third_party/composable_kernel' 2025-08-14T21:26:23.3643439Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-08-14T21:26:23.3665188Z Entering 'third_party/cpp-httplib' 2025-08-14T21:26:23.3716932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-08-14T21:26:23.3738368Z Entering 'third_party/cpuinfo' 2025-08-14T21:26:23.3782816Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-08-14T21:26:23.3802372Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:26:23.3855530Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-08-14T21:26:23.3874775Z Entering 'third_party/cutlass' 2025-08-14T21:26:23.3925224Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-08-14T21:26:23.3949646Z Entering 'third_party/fbgemm' 2025-08-14T21:26:23.3999286Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-08-14T21:26:23.4018687Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:26:23.4067487Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-08-14T21:26:23.4084121Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:26:23.4137404Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-08-14T21:26:23.4160740Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:26:23.4206034Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-08-14T21:26:23.4223359Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:26:23.4270698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-08-14T21:26:23.4292510Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:26:23.4340841Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-08-14T21:26:23.4360688Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:26:23.4405436Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-08-14T21:26:23.4421913Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:26:23.4479409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-08-14T21:26:23.4498873Z Entering 'third_party/flash-attention' 2025-08-14T21:26:23.4552316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-08-14T21:26:23.4571761Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:26:23.4618049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-08-14T21:26:23.4636765Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:26:23.4685597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-08-14T21:26:23.4714334Z Entering 'third_party/flatbuffers' 2025-08-14T21:26:23.4767957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-08-14T21:26:23.4784632Z Entering 'third_party/fmt' 2025-08-14T21:26:23.4831686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-08-14T21:26:23.4854001Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:26:23.4897177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-08-14T21:26:23.4915317Z Entering 'third_party/gloo' 2025-08-14T21:26:23.4964599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-08-14T21:26:23.4977764Z Entering 'third_party/googletest' 2025-08-14T21:26:23.5028831Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:26:23.5055913Z Entering 'third_party/ideep' 2025-08-14T21:26:23.5098435Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-08-14T21:26:23.5114839Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:26:23.5162580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-08-14T21:26:23.5190138Z Entering 'third_party/ittapi' 2025-08-14T21:26:23.5238662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-08-14T21:26:23.5262327Z Entering 'third_party/kineto' 2025-08-14T21:26:23.5311108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-08-14T21:26:23.5332782Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:26:23.5378455Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-08-14T21:26:23.5394157Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:26:23.5444106Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-08-14T21:26:23.5465364Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:26:23.5510799Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-08-14T21:26:23.5527567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:26:23.5578878Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-08-14T21:26:23.5595301Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:26:23.5649169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-08-14T21:26:23.5662386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:26:23.5712171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-08-14T21:26:23.5733524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:26:23.5789551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-08-14T21:26:23.5799939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:26:23.5847154Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:26:23.5871723Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:26:23.5913355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-08-14T21:26:23.5930960Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:26:23.5984237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-08-14T21:26:23.5999075Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:26:23.6050846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-08-14T21:26:23.6064877Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:26:23.6113816Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-08-14T21:26:23.6136097Z Entering 'third_party/kleidiai' 2025-08-14T21:26:23.6181511Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-08-14T21:26:23.6197623Z Entering 'third_party/mimalloc' 2025-08-14T21:26:23.6247149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-08-14T21:26:23.6265364Z Entering 'third_party/nlohmann' 2025-08-14T21:26:23.6313005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-08-14T21:26:23.6331515Z Entering 'third_party/onnx' 2025-08-14T21:26:23.6377331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-08-14T21:26:23.6410762Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:26:23.6460908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-08-14T21:26:23.6477207Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:26:23.6532052Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-08-14T21:26:23.6557203Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:26:23.6603869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-08-14T21:26:23.6623975Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:26:23.6674722Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:26:23.6689881Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:26:23.6743614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-08-14T21:26:23.6758626Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:26:23.6807016Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-08-14T21:26:23.6824611Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:26:23.6873677Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-08-14T21:26:23.6889408Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:26:23.6937492Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-08-14T21:26:23.6952552Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:26:23.7000865Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-08-14T21:26:23.7015430Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:26:23.7065089Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-08-14T21:26:23.7082892Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:26:23.7137113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-08-14T21:26:23.7161297Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:26:23.7208948Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-08-14T21:26:23.7244616Z Entering 'third_party/pocketfft' 2025-08-14T21:26:23.7291070Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-08-14T21:26:23.7305599Z Entering 'third_party/protobuf' 2025-08-14T21:26:23.7356809Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-08-14T21:26:23.7378428Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:26:23.7424534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-08-14T21:26:23.7445291Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:26:23.7497037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:26:23.7513537Z Entering 'third_party/psimd' 2025-08-14T21:26:23.7568538Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-08-14T21:26:23.7584369Z Entering 'third_party/pthreadpool' 2025-08-14T21:26:23.7633764Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-08-14T21:26:23.7652887Z Entering 'third_party/pybind11' 2025-08-14T21:26:23.7702344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-08-14T21:26:23.7715792Z Entering 'third_party/python-peachpy' 2025-08-14T21:26:23.7760789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-08-14T21:26:23.7780659Z Entering 'third_party/sleef' 2025-08-14T21:26:23.7831637Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-08-14T21:26:23.7850083Z Entering 'third_party/tensorpipe' 2025-08-14T21:26:23.7896826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-08-14T21:26:23.7916707Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:26:23.7964957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:26:23.7982799Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:26:23.8033550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-08-14T21:26:23.8048892Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:26:23.8097344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-08-14T21:26:23.8111841Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:26:23.8160941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-08-14T21:26:23.8178502Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:26:23.8228897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-08-14T21:26:23.9322437Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-08-14T21:26:23.9657278Z Entering 'android/libs/fbjni' 2025-08-14T21:26:23.9696012Z Entering 'third_party/FP16' 2025-08-14T21:26:23.9745716Z Entering 'third_party/FXdiv' 2025-08-14T21:26:23.9781037Z Entering 'third_party/NNPACK' 2025-08-14T21:26:23.9827849Z Entering 'third_party/NVTX' 2025-08-14T21:26:23.9869499Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:26:23.9908527Z Entering 'third_party/XNNPACK' 2025-08-14T21:26:23.9965658Z Entering 'third_party/aiter' 2025-08-14T21:26:24.0003767Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:26:24.0059402Z Entering 'third_party/benchmark' 2025-08-14T21:26:24.0096307Z Entering 'third_party/composable_kernel' 2025-08-14T21:26:24.0149823Z Entering 'third_party/cpp-httplib' 2025-08-14T21:26:24.0194280Z Entering 'third_party/cpuinfo' 2025-08-14T21:26:24.0241001Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:26:24.0281278Z Entering 'third_party/cutlass' 2025-08-14T21:26:24.0327901Z Entering 'third_party/fbgemm' 2025-08-14T21:26:24.0369738Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:26:24.0405131Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:26:24.0451096Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:26:24.0494987Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:26:24.0538253Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:26:24.0583491Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:26:24.0623343Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:26:24.0668444Z Entering 'third_party/flash-attention' 2025-08-14T21:26:24.0705333Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:26:24.0752793Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:26:24.0800233Z Entering 'third_party/flatbuffers' 2025-08-14T21:26:24.0843754Z Entering 'third_party/fmt' 2025-08-14T21:26:24.0881417Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:26:24.0923328Z Entering 'third_party/gloo' 2025-08-14T21:26:24.0965194Z Entering 'third_party/googletest' 2025-08-14T21:26:24.1002657Z Entering 'third_party/ideep' 2025-08-14T21:26:24.1046487Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:26:24.1092827Z Entering 'third_party/ittapi' 2025-08-14T21:26:24.1139029Z Entering 'third_party/kineto' 2025-08-14T21:26:24.1177737Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:26:24.1216939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:26:24.1257217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:26:24.1296749Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:26:24.1334374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:26:24.1377815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:26:24.1420228Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:26:24.1463536Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:26:24.1501201Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:26:24.1543953Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:26:24.1586374Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:26:24.1625105Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:26:24.1668279Z Entering 'third_party/kleidiai' 2025-08-14T21:26:24.1704439Z Entering 'third_party/mimalloc' 2025-08-14T21:26:24.1746949Z Entering 'third_party/nlohmann' 2025-08-14T21:26:24.1786302Z Entering 'third_party/onnx' 2025-08-14T21:26:24.1845896Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:26:24.1888441Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:26:24.1932809Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:26:24.1969545Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:26:24.2011072Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:26:24.2053428Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:26:24.2092204Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:26:24.2130329Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:26:24.2175026Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:26:24.2209588Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:26:24.2250763Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:26:24.2290242Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:26:24.2351227Z Entering 'third_party/pocketfft' 2025-08-14T21:26:24.2386053Z Entering 'third_party/protobuf' 2025-08-14T21:26:24.2425354Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:26:24.2469130Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:26:24.2511315Z Entering 'third_party/psimd' 2025-08-14T21:26:24.2553440Z Entering 'third_party/pthreadpool' 2025-08-14T21:26:24.2590905Z Entering 'third_party/pybind11' 2025-08-14T21:26:24.2632999Z Entering 'third_party/python-peachpy' 2025-08-14T21:26:24.2673890Z Entering 'third_party/sleef' 2025-08-14T21:26:24.2714664Z Entering 'third_party/tensorpipe' 2025-08-14T21:26:24.2753436Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:26:24.2793861Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:26:24.2834869Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:26:24.2874559Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:26:24.2914706Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:26:24.2977377Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-08-14T21:26:24.3280915Z Entering 'android/libs/fbjni' 2025-08-14T21:26:24.3322627Z Entering 'third_party/FP16' 2025-08-14T21:26:24.3363391Z Entering 'third_party/FXdiv' 2025-08-14T21:26:24.3399921Z Entering 'third_party/NNPACK' 2025-08-14T21:26:24.3440605Z Entering 'third_party/NVTX' 2025-08-14T21:26:24.3484163Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:26:24.3522588Z Entering 'third_party/XNNPACK' 2025-08-14T21:26:24.3576529Z Entering 'third_party/aiter' 2025-08-14T21:26:24.3622638Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:26:24.3668686Z Entering 'third_party/benchmark' 2025-08-14T21:26:24.3720180Z Entering 'third_party/composable_kernel' 2025-08-14T21:26:24.3768631Z Entering 'third_party/cpp-httplib' 2025-08-14T21:26:24.3806947Z Entering 'third_party/cpuinfo' 2025-08-14T21:26:24.3848835Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:26:24.3889700Z Entering 'third_party/cutlass' 2025-08-14T21:26:24.3944510Z Entering 'third_party/fbgemm' 2025-08-14T21:26:24.3987104Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:26:24.4030457Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:26:24.4075560Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:26:24.4113218Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:26:24.4160481Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:26:24.4196615Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:26:24.4234132Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:26:24.4280755Z Entering 'third_party/flash-attention' 2025-08-14T21:26:24.4326714Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:26:24.4375197Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:26:24.4424331Z Entering 'third_party/flatbuffers' 2025-08-14T21:26:24.4464725Z Entering 'third_party/fmt' 2025-08-14T21:26:24.4504807Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:26:24.4549879Z Entering 'third_party/gloo' 2025-08-14T21:26:24.4589113Z Entering 'third_party/googletest' 2025-08-14T21:26:24.4631454Z Entering 'third_party/ideep' 2025-08-14T21:26:24.4670112Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:26:24.4722429Z Entering 'third_party/ittapi' 2025-08-14T21:26:24.4763659Z Entering 'third_party/kineto' 2025-08-14T21:26:24.4801118Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:26:24.4869928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:26:24.4908138Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:26:24.4952100Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:26:24.4992706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:26:24.5033977Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:26:24.5071322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:26:24.5111666Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:26:24.5152245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:26:24.5190715Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:26:24.5234974Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:26:24.5274769Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:26:24.5312574Z Entering 'third_party/kleidiai' 2025-08-14T21:26:24.5354992Z Entering 'third_party/mimalloc' 2025-08-14T21:26:24.5396637Z Entering 'third_party/nlohmann' 2025-08-14T21:26:24.5444141Z Entering 'third_party/onnx' 2025-08-14T21:26:24.5499179Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:26:24.5546581Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:26:24.5588237Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:26:24.5631495Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:26:24.5668637Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:26:24.5708792Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:26:24.5753986Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:26:24.5790944Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:26:24.5831650Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:26:24.5867246Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:26:24.5913220Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:26:24.5955418Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:26:24.6015900Z Entering 'third_party/pocketfft' 2025-08-14T21:26:24.6055262Z Entering 'third_party/protobuf' 2025-08-14T21:26:24.6097067Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:26:24.6136570Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:26:24.6189556Z Entering 'third_party/psimd' 2025-08-14T21:26:24.6228076Z Entering 'third_party/pthreadpool' 2025-08-14T21:26:24.6271157Z Entering 'third_party/pybind11' 2025-08-14T21:26:24.6311916Z Entering 'third_party/python-peachpy' 2025-08-14T21:26:24.6355562Z Entering 'third_party/sleef' 2025-08-14T21:26:24.6393338Z Entering 'third_party/tensorpipe' 2025-08-14T21:26:24.6431427Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:26:24.6474583Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:26:24.6512980Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:26:24.6554839Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:26:24.6594508Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:26:24.6657078Z ##[endgroup] 2025-08-14T21:26:24.6687648Z [command]/usr/bin/git log -1 --format=%H 2025-08-14T21:26:24.6709041Z 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:26:24.6883540Z Prepare all required actions 2025-08-14T21:26:24.6884079Z Getting action download info 2025-08-14T21:26:24.8624350Z ##[group]Run ./.github/actions/setup-linux 2025-08-14T21:26:24.8624595Z env: 2025-08-14T21:26:24.8624767Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:24.8624948Z ##[endgroup] 2025-08-14T21:26:24.8663043Z ##[group]Run set -euo pipefail 2025-08-14T21:26:24.8663319Z set -euo pipefail 2025-08-14T21:26:24.8663533Z function get_ec2_metadata() { 2025-08-14T21:26:24.8663793Z  # Pulled from instance metadata endpoint for EC2 2025-08-14T21:26:24.8664222Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-08-14T21:26:24.8664595Z  category=$1 2025-08-14T21:26:24.8664847Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-08-14T21:26:24.8665146Z  runner_name_str=i-0aaf71856f9399359 2025-08-14T21:26:24.8665436Z  if [[ -f /.inarc ]]; then 2025-08-14T21:26:24.8665678Z  echo "ARC Runner, no info on ec2 metadata" 2025-08-14T21:26:24.8665964Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-08-14T21:26:24.8666280Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-08-14T21:26:24.8666555Z  else 2025-08-14T21:26:24.8667121Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-08-14T21:26:24.8667686Z  fi 2025-08-14T21:26:24.8667855Z } 2025-08-14T21:26:24.8668047Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-08-14T21:26:24.8668345Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-08-14T21:26:24.8668669Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-08-14T21:26:24.8668953Z echo "system info $(uname -a)" 2025-08-14T21:26:24.8678159Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:24.8678432Z env: 2025-08-14T21:26:24.8678610Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:24.8678797Z ##[endgroup] 2025-08-14T21:26:24.8829340Z ami-id: ami-05ffe3c48a9991133 2025-08-14T21:26:24.8927462Z instance-id: i-0aaf71856f9399359 2025-08-14T21:26:24.9019524Z instance-type: m7i-flex.8xlarge 2025-08-14T21:26:24.9032754Z system info Linux ip-10-0-36-175.ec2.internal 6.1.141-155.222.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 17 10:29:47 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-08-14T21:26:24.9051153Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:26:24.9051719Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:26:24.9056592Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:24.9056850Z env: 2025-08-14T21:26:24.9057123Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:24.9057299Z ##[endgroup] 2025-08-14T21:26:24.9100970Z ##[group]Run if systemctl is-active --quiet docker; then 2025-08-14T21:26:24.9101283Z if systemctl is-active --quiet docker; then 2025-08-14T21:26:24.9101551Z  echo "Docker daemon is running..."; 2025-08-14T21:26:24.9101750Z else 2025-08-14T21:26:24.9101980Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-08-14T21:26:24.9102238Z fi 2025-08-14T21:26:24.9106285Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:24.9106529Z env: 2025-08-14T21:26:24.9106694Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:24.9106884Z ##[endgroup] 2025-08-14T21:26:24.9212707Z Docker daemon is running... 2025-08-14T21:26:24.9241003Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-14T21:26:24.9241234Z with: 2025-08-14T21:26:24.9241392Z shell: bash 2025-08-14T21:26:24.9241717Z timeout_minutes: 5 2025-08-14T21:26:24.9241908Z max_attempts: 3 2025-08-14T21:26:24.9242092Z retry_wait_seconds: 30 2025-08-14T21:26:24.9243509Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-08-14T21:26:24.9244866Z polling_interval_seconds: 1 2025-08-14T21:26:24.9245066Z warning_on_retry: true 2025-08-14T21:26:24.9245246Z continue_on_error: false 2025-08-14T21:26:24.9245430Z env: 2025-08-14T21:26:24.9245591Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:24.9245779Z AWS_RETRY_MODE: standard 2025-08-14T21:26:24.9245956Z AWS_MAX_ATTEMPTS: 5 2025-08-14T21:26:24.9246133Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:26:24.9246309Z ##[endgroup] 2025-08-14T21:26:25.9112191Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-14T21:26:25.9112635Z Configure a credential helper to remove this warning. See 2025-08-14T21:26:25.9113494Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-14T21:26:25.9113742Z 2025-08-14T21:26:25.9113816Z Login Succeeded 2025-08-14T21:26:25.9968614Z Command completed after 1 attempt(s). 2025-08-14T21:26:26.0022845Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:26:26.0023203Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:26:26.0023485Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:26:26.0029550Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:26.0029824Z env: 2025-08-14T21:26:26.0029993Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:26.0030177Z ##[endgroup] 2025-08-14T21:26:26.0107359Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T21:26:26.0107717Z # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T21:26:26.0107991Z # shellcheck disable=SC2046 2025-08-14T21:26:26.0108214Z docker stop $(docker ps -q) || true 2025-08-14T21:26:26.0108430Z # Prune all of the docker images 2025-08-14T21:26:26.0108796Z docker system prune -af 2025-08-14T21:26:26.0113124Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:26.0113368Z env: 2025-08-14T21:26:26.0113532Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:26.0113719Z ##[endgroup] 2025-08-14T21:26:26.0595465Z "docker stop" requires at least 1 argument. 2025-08-14T21:26:26.0595810Z See 'docker stop --help'. 2025-08-14T21:26:26.0595973Z 2025-08-14T21:26:26.0596222Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-08-14T21:26:26.0596810Z 2025-08-14T21:26:26.0596899Z Stop one or more running containers 2025-08-14T21:26:26.0791187Z Total reclaimed space: 0B 2025-08-14T21:26:26.0818993Z ##[group]Run set +e 2025-08-14T21:26:26.0819209Z set +e 2025-08-14T21:26:26.0819376Z set -x 2025-08-14T21:26:26.0819530Z  2025-08-14T21:26:26.0819698Z PT_DOMAIN=download.pytorch.org 2025-08-14T21:26:26.0820067Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2025-08-14T21:26:26.0820512Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2025-08-14T21:26:26.0820827Z # one is returned at random 2025-08-14T21:26:26.0821079Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2025-08-14T21:26:26.0821318Z  2025-08-14T21:26:26.0821597Z if [ -z "${RESOLVED_IP}" ]; then 2025-08-14T21:26:26.0821882Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2025-08-14T21:26:26.0822228Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2025-08-14T21:26:26.0822479Z  2025-08-14T21:26:26.0822647Z  if [ -z "${RESOLVED_IP}" ]; then 2025-08-14T21:26:26.0822901Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2025-08-14T21:26:26.0823132Z  exit 1 2025-08-14T21:26:26.0823292Z  fi 2025-08-14T21:26:26.0823437Z fi 2025-08-14T21:26:26.0823583Z  2025-08-14T21:26:26.0823756Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2025-08-14T21:26:26.0823984Z  # Clean up any old records first 2025-08-14T21:26:26.0824228Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2025-08-14T21:26:26.0824443Z fi 2025-08-14T21:26:26.0824605Z  2025-08-14T21:26:26.0824804Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2025-08-14T21:26:26.0825052Z cat /etc/hosts 2025-08-14T21:26:26.0830002Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:26.0830251Z env: 2025-08-14T21:26:26.0830414Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:26.0830603Z ##[endgroup] 2025-08-14T21:26:26.0853694Z + PT_DOMAIN=download.pytorch.org 2025-08-14T21:26:26.0862551Z ++ dig -4 +short download.pytorch.org 2025-08-14T21:26:26.0862835Z ++ tail -n1 2025-08-14T21:26:26.1301102Z + RESOLVED_IP=18.160.10.28 2025-08-14T21:26:26.1301408Z + '[' -z 18.160.10.28 ']' 2025-08-14T21:26:26.1301637Z + grep -r download.pytorch.org /etc/hosts 2025-08-14T21:26:26.1321399Z + echo '18.160.10.28 download.pytorch.org' 2025-08-14T21:26:26.1321666Z + sudo tee -a /etc/hosts 2025-08-14T21:26:26.4168258Z 18.160.10.28 download.pytorch.org 2025-08-14T21:26:26.4184937Z + cat /etc/hosts 2025-08-14T21:26:26.4193752Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2025-08-14T21:26:26.4200638Z ::1 localhost6 localhost6.localdomain6 2025-08-14T21:26:26.4201008Z 18.160.10.28 download.pytorch.org 2025-08-14T21:26:26.4312635Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-08-14T21:26:26.4312981Z with: 2025-08-14T21:26:26.4313567Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:26.4314199Z use-custom-docker-registry: true 2025-08-14T21:26:26.4314441Z docker-build-dir: .ci/docker 2025-08-14T21:26:26.4314668Z docker-build-script: ./build.sh 2025-08-14T21:26:26.4314883Z working-directory: . 2025-08-14T21:26:26.4315148Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:26.4315432Z force-push: false 2025-08-14T21:26:26.4315601Z env: 2025-08-14T21:26:26.4315768Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:26.4315961Z ##[endgroup] 2025-08-14T21:26:26.4331612Z ##[group]Run set -ex 2025-08-14T21:26:26.4331863Z set -ex 2025-08-14T21:26:26.4332144Z  2025-08-14T21:26:26.4332474Z # If the docker build directory or the build script doesn't exist, the action will 2025-08-14T21:26:26.4332906Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-08-14T21:26:26.4333272Z # job could then download the pre-built image as usual 2025-08-14T21:26:26.4333718Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-08-14T21:26:26.4334132Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4334354Z else 2025-08-14T21:26:26.4334552Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4334857Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4335126Z  2025-08-14T21:26:26.4335490Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-08-14T21:26:26.4335890Z  exit 0 2025-08-14T21:26:26.4336055Z fi 2025-08-14T21:26:26.4336209Z  2025-08-14T21:26:26.4336445Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-08-14T21:26:26.4336834Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-08-14T21:26:26.4337180Z  # use it as it is, but first let's extract the tag 2025-08-14T21:26:26.4337496Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-08-14T21:26:26.4337832Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4338154Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4338421Z else 2025-08-14T21:26:26.4338610Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-08-14T21:26:26.4338872Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-08-14T21:26:26.4339147Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-08-14T21:26:26.4339366Z  fi 2025-08-14T21:26:26.4339673Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-08-14T21:26:26.4340064Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4340472Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4340915Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4341199Z fi 2025-08-14T21:26:26.4349116Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:26.4349364Z env: 2025-08-14T21:26:26.4349533Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:26.4349734Z REPO_NAME: pytorch 2025-08-14T21:26:26.4350399Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:26.4350980Z DOCKER_BUILD_DIR: .ci/docker 2025-08-14T21:26:26.4351183Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-08-14T21:26:26.4351438Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:26.4351691Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-08-14T21:26:26.4351900Z CUSTOM_TAG_PREFIX: 2025-08-14T21:26:26.4352079Z ##[endgroup] 2025-08-14T21:26:26.4379464Z + [[ -d .ci/docker ]] 2025-08-14T21:26:26.4379855Z + [[ -f .ci/docker/./build.sh ]] 2025-08-14T21:26:26.4380084Z + [[ true == \t\r\u\e ]] 2025-08-14T21:26:26.4380262Z + echo skip=false 2025-08-14T21:26:26.4381081Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-08-14T21:26:26.4383650Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:26.4386029Z ++ awk -F '[:,]' '{print $2}' 2025-08-14T21:26:26.4424797Z + DOCKER_TAG=pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:26.4425693Z + echo docker-tag=pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:26.4426645Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:26.4445907Z ##[group]Run set +e 2025-08-14T21:26:26.4446133Z set +e 2025-08-14T21:26:26.4446298Z set -x 2025-08-14T21:26:26.4446446Z  2025-08-14T21:26:26.4446593Z login() { 2025-08-14T21:26:26.4446905Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-08-14T21:26:26.4447228Z } 2025-08-14T21:26:26.4447376Z  2025-08-14T21:26:26.4447522Z retry () { 2025-08-14T21:26:26.4447697Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-08-14T21:26:26.4447902Z } 2025-08-14T21:26:26.4448043Z  2025-08-14T21:26:26.4448196Z retry login "${DOCKER_REGISTRY}" 2025-08-14T21:26:26.4448392Z  2025-08-14T21:26:26.4448542Z START_TIME=$(date +%s) 2025-08-14T21:26:26.4448754Z # Wait up to 120 minutes 2025-08-14T21:26:26.4449006Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-08-14T21:26:26.4449330Z  # Check if image already exists, if it does then skip building it 2025-08-14T21:26:26.4449632Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-08-14T21:26:26.4449851Z  exit 0 2025-08-14T21:26:26.4450011Z  fi 2025-08-14T21:26:26.4450162Z  2025-08-14T21:26:26.4450406Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-08-14T21:26:26.4450793Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-08-14T21:26:26.4451178Z  # latter, it will wait for the Docker images to become available before continuing 2025-08-14T21:26:26.4451499Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-08-14T21:26:26.4451748Z  # It's a Docker build job, let's build the image 2025-08-14T21:26:26.4451963Z  break 2025-08-14T21:26:26.4452121Z  else 2025-08-14T21:26:26.4452345Z  # It's a regular build job, wait for the image to become available 2025-08-14T21:26:26.4452593Z  sleep 300 2025-08-14T21:26:26.4452762Z  fi 2025-08-14T21:26:26.4452914Z done 2025-08-14T21:26:26.4453054Z  2025-08-14T21:26:26.4453285Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-08-14T21:26:26.4453727Z # be empty. The default action would be to continue rebuild the image 2025-08-14T21:26:26.4454037Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-08-14T21:26:26.4454323Z  # if we're on the base branch then use the parent commit 2025-08-14T21:26:26.4454577Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-08-14T21:26:26.4454776Z else 2025-08-14T21:26:26.4454982Z  # otherwise we're on a PR, so use the most recent base commit 2025-08-14T21:26:26.4455278Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-08-14T21:26:26.4455540Z fi 2025-08-14T21:26:26.4455679Z  2025-08-14T21:26:26.4455840Z if [[ -z "${MERGE_BASE}" ]]; then 2025-08-14T21:26:26.4456069Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4456277Z  2025-08-14T21:26:26.4456566Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-08-14T21:26:26.4456979Z  exit 0 2025-08-14T21:26:26.4457138Z fi 2025-08-14T21:26:26.4457280Z  2025-08-14T21:26:26.4457497Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-08-14T21:26:26.4457915Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-08-14T21:26:26.4458271Z  exit 1 2025-08-14T21:26:26.4458424Z fi 2025-08-14T21:26:26.4458573Z  2025-08-14T21:26:26.4458823Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-08-14T21:26:26.4459228Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-08-14T21:26:26.4459576Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-08-14T21:26:26.4459996Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-08-14T21:26:26.4460465Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-08-14T21:26:26.4460745Z fi 2025-08-14T21:26:26.4460897Z  2025-08-14T21:26:26.4461083Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-08-14T21:26:26.4465693Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:26.4465951Z env: 2025-08-14T21:26:26.4466120Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:26.4466315Z DOCKER_BUILD_DIR: .ci/docker 2025-08-14T21:26:26.4466558Z BASE_REVISION: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:26:26.4467178Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:26.4467952Z DOCKER_TAG: pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:26.4468424Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:26.4468681Z DOCKER_PUSH: 2025-08-14T21:26:26.4468849Z ##[endgroup] 2025-08-14T21:26:26.4493009Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:26.4493344Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:26.4495344Z + aws ecr get-login-password --region us-east-1 2025-08-14T21:26:26.4495961Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:26.8763636Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-14T21:26:26.8764033Z Login Succeeded 2025-08-14T21:26:26.8766260Z Configure a credential helper to remove this warning. See 2025-08-14T21:26:26.8766841Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-14T21:26:26.8767185Z 2025-08-14T21:26:26.8790316Z ++ date +%s 2025-08-14T21:26:26.8799808Z + START_TIME=1755206786 2025-08-14T21:26:26.8805405Z ++ date +%s 2025-08-14T21:26:26.8819126Z + [[ 1755199586 -lt 1755206786 ]] 2025-08-14T21:26:26.8820627Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:27.1019068Z { 2025-08-14T21:26:27.1019607Z "schemaVersion": 2, 2025-08-14T21:26:27.1020617Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-08-14T21:26:27.1020975Z "config": { 2025-08-14T21:26:27.1021229Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-08-14T21:26:27.1021492Z "size": 30151, 2025-08-14T21:26:27.1021774Z "digest": "sha256:0899ae453036ee7a91795ea95b1db61000579eeb74b140edab5976919ee64bbe" 2025-08-14T21:26:27.1022066Z }, 2025-08-14T21:26:27.1022201Z "layers": [ 2025-08-14T21:26:27.1022348Z { 2025-08-14T21:26:27.1022573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1023238Z "size": 30448173, 2025-08-14T21:26:27.1023543Z "digest": "sha256:660ffc76f83b006444a5731b215acc2e35138d8be5cac8ed1ffd40f947117495" 2025-08-14T21:26:27.1023842Z }, 2025-08-14T21:26:27.1023971Z { 2025-08-14T21:26:27.1024190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1024475Z "size": 1554, 2025-08-14T21:26:27.1024842Z "digest": "sha256:c7b4a852a45516e27a9256df90878663d770f96d271d6155d43be78cc5225eef" 2025-08-14T21:26:27.1025113Z }, 2025-08-14T21:26:27.1025246Z { 2025-08-14T21:26:27.1025473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1025752Z "size": 313280151, 2025-08-14T21:26:27.1026035Z "digest": "sha256:e5a28988c8932eb5797557621582a064ce48651dbb5eaed379e9978535daccb9" 2025-08-14T21:26:27.1026315Z }, 2025-08-14T21:26:27.1026439Z { 2025-08-14T21:26:27.1026651Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1026909Z "size": 793, 2025-08-14T21:26:27.1027174Z "digest": "sha256:76a69b57b6837bef07dbc1b481cf28a62dfd7c7063219d9f6e0d0d63067653c7" 2025-08-14T21:26:27.1027461Z }, 2025-08-14T21:26:27.1027593Z { 2025-08-14T21:26:27.1027791Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1028047Z "size": 106, 2025-08-14T21:26:27.1028314Z "digest": "sha256:5c785dcb4cdbf1f2ceffe4d1d8e85d73225a56d0236e7ed6e36a95c836996052" 2025-08-14T21:26:27.1028600Z }, 2025-08-14T21:26:27.1028722Z { 2025-08-14T21:26:27.1028946Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1029222Z "size": 704, 2025-08-14T21:26:27.1029490Z "digest": "sha256:836ab08052e8eb2bae68e69ae086fd23a5f04a8491c320718ab47f84f03aebb1" 2025-08-14T21:26:27.1029788Z }, 2025-08-14T21:26:27.1029923Z { 2025-08-14T21:26:27.1030133Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1030405Z "size": 1217, 2025-08-14T21:26:27.1030686Z "digest": "sha256:53b11c77468cbefca210560f7d8be8e58f9eeb415e096ab0c3fb0277f0b41caf" 2025-08-14T21:26:27.1030982Z }, 2025-08-14T21:26:27.1031119Z { 2025-08-14T21:26:27.1031338Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1031614Z "size": 485, 2025-08-14T21:26:27.1031897Z "digest": "sha256:e97311a6a967664cbe10c5027a1ec60c514caa9a1160167d8363088fd1f9fe09" 2025-08-14T21:26:27.1032192Z }, 2025-08-14T21:26:27.1032328Z { 2025-08-14T21:26:27.1032540Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1032820Z "size": 110343699, 2025-08-14T21:26:27.1033103Z "digest": "sha256:2c414689d31dc46a22fe02d4f43699f528cc1c02fb505824768383fa0bbf1c74" 2025-08-14T21:26:27.1033388Z }, 2025-08-14T21:26:27.1033526Z { 2025-08-14T21:26:27.1033742Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1034004Z "size": 4817, 2025-08-14T21:26:27.1034295Z "digest": "sha256:6d89b5f065d59e4abcaa9b5ff3bf0afded2394d493d2df0f7babf7154f7548e0" 2025-08-14T21:26:27.1034742Z }, 2025-08-14T21:26:27.1034884Z { 2025-08-14T21:26:27.1035109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1035386Z "size": 1709, 2025-08-14T21:26:27.1035674Z "digest": "sha256:5a5cc76ada432cccf7d18e0eb79379afb95deaaa7afec482406267924d291ae4" 2025-08-14T21:26:27.1036196Z }, 2025-08-14T21:26:27.1036354Z { 2025-08-14T21:26:27.1036583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1036864Z "size": 724, 2025-08-14T21:26:27.1037157Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:26:27.1037461Z }, 2025-08-14T21:26:27.1037591Z { 2025-08-14T21:26:27.1037820Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1038178Z "size": 542, 2025-08-14T21:26:27.1038456Z "digest": "sha256:2e16579078600b91216fd14aca1e0ce0f9d1801b230689dd309980e8d2783935" 2025-08-14T21:26:27.1038815Z }, 2025-08-14T21:26:27.1038953Z { 2025-08-14T21:26:27.1039177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1039483Z "size": 3397512507, 2025-08-14T21:26:27.1039774Z "digest": "sha256:7b92d7a4b8c766d7b7873aa33088e171fb44a8e968645e4b31dfe6de2968aead" 2025-08-14T21:26:27.1040082Z }, 2025-08-14T21:26:27.1040214Z { 2025-08-14T21:26:27.1040432Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1040695Z "size": 32, 2025-08-14T21:26:27.1040964Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1041270Z }, 2025-08-14T21:26:27.1041416Z { 2025-08-14T21:26:27.1041616Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1041854Z "size": 380, 2025-08-14T21:26:27.1042098Z "digest": "sha256:d6226eb61f823984003d5ac28f4d66fec9b27baf5d54a9513286483f5912cd88" 2025-08-14T21:26:27.1042369Z }, 2025-08-14T21:26:27.1042490Z { 2025-08-14T21:26:27.1042691Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1042936Z "size": 234681, 2025-08-14T21:26:27.1043182Z "digest": "sha256:83c70f4266a6ee5f8f44a88d4cb951382f6c960323b8250046bddc080e62268b" 2025-08-14T21:26:27.1043454Z }, 2025-08-14T21:26:27.1043580Z { 2025-08-14T21:26:27.1043770Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1044014Z "size": 231, 2025-08-14T21:26:27.1044260Z "digest": "sha256:60c725d21861c24c417efe3a5474414ba04f0f49c78c6d6451478ab9e45469ec" 2025-08-14T21:26:27.1044528Z }, 2025-08-14T21:26:27.1044647Z { 2025-08-14T21:26:27.1044847Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1045094Z "size": 4464546, 2025-08-14T21:26:27.1045348Z "digest": "sha256:a504e76e66a49926b4ea837b7a7ff3c842a27b2caaa4d80cf5057a1e55293666" 2025-08-14T21:26:27.1045624Z }, 2025-08-14T21:26:27.1045752Z { 2025-08-14T21:26:27.1045948Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1046195Z "size": 1864, 2025-08-14T21:26:27.1046460Z "digest": "sha256:fc1c200a4f77face2af0146f9b03ad04f31fe06fec216473ffd2ebd538cde056" 2025-08-14T21:26:27.1046737Z }, 2025-08-14T21:26:27.1046867Z { 2025-08-14T21:26:27.1047072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1047307Z "size": 475, 2025-08-14T21:26:27.1047554Z "digest": "sha256:43273c22704f81f162741d2039015f745273eee1d1fdec47be35c9b2a90dcc5b" 2025-08-14T21:26:27.1047821Z }, 2025-08-14T21:26:27.1047947Z { 2025-08-14T21:26:27.1048139Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1048382Z "size": 178, 2025-08-14T21:26:27.1048636Z "digest": "sha256:89df389d042adbd7621a94d36b6e3db60ff6c559efb95c6fcc11b8afd42f0599" 2025-08-14T21:26:27.1048906Z }, 2025-08-14T21:26:27.1049032Z { 2025-08-14T21:26:27.1049232Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1049521Z "size": 586, 2025-08-14T21:26:27.1049771Z "digest": "sha256:684349f50d9456597026ee5c1bd890c51d1e498614f367adf03329c5227add79" 2025-08-14T21:26:27.1050036Z }, 2025-08-14T21:26:27.1050156Z { 2025-08-14T21:26:27.1050357Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1050602Z "size": 218, 2025-08-14T21:26:27.1050849Z "digest": "sha256:21d0eae87fb3ac753b3f0e91ae638360d23922d4cd119410a5a1b97bbe0ca435" 2025-08-14T21:26:27.1051127Z }, 2025-08-14T21:26:27.1051254Z { 2025-08-14T21:26:27.1051452Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1051687Z "size": 802, 2025-08-14T21:26:27.1051935Z "digest": "sha256:c9c2b424b8e08d943dc259a3796d66eede3a1e93a6460df5db132c0036d3d6af" 2025-08-14T21:26:27.1052211Z }, 2025-08-14T21:26:27.1052329Z { 2025-08-14T21:26:27.1052527Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1052812Z "size": 32, 2025-08-14T21:26:27.1053059Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1053336Z }, 2025-08-14T21:26:27.1053463Z { 2025-08-14T21:26:27.1053654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1053898Z "size": 104, 2025-08-14T21:26:27.1054151Z "digest": "sha256:98dda28f339592e3ca6d589d551e69b8314f2b7fc2a1544eacc1b3c2d3378521" 2025-08-14T21:26:27.1054428Z }, 2025-08-14T21:26:27.1054548Z { 2025-08-14T21:26:27.1054749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1054993Z "size": 1496, 2025-08-14T21:26:27.1055241Z "digest": "sha256:acf5babd87f23aa905883eb434073e9a00ff41679134f2f4827dd86949f5a9d9" 2025-08-14T21:26:27.1055518Z }, 2025-08-14T21:26:27.1055644Z { 2025-08-14T21:26:27.1055839Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1056089Z "size": 453555614, 2025-08-14T21:26:27.1056360Z "digest": "sha256:7c5050d8408d3c4f9f5e8f2cb215245473bfc2f1510fe5ee01c2a6c505068b5a" 2025-08-14T21:26:27.1056628Z }, 2025-08-14T21:26:27.1056755Z { 2025-08-14T21:26:27.1056955Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1057195Z "size": 163, 2025-08-14T21:26:27.1057450Z "digest": "sha256:7ddd14e2b548b9ae6e216a081bb20116434aacbbe571c99b40e60fb2fde22a2a" 2025-08-14T21:26:27.1057730Z }, 2025-08-14T21:26:27.1057857Z { 2025-08-14T21:26:27.1058053Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1058296Z "size": 347, 2025-08-14T21:26:27.1058547Z "digest": "sha256:4ba8e7a736c8199931fd7ff9931a5f17b7b931d0383a3e158f1b12b191a1d250" 2025-08-14T21:26:27.1058815Z }, 2025-08-14T21:26:27.1058942Z { 2025-08-14T21:26:27.1059144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1059380Z "size": 32, 2025-08-14T21:26:27.1059643Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1059920Z }, 2025-08-14T21:26:27.1060036Z { 2025-08-14T21:26:27.1060230Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1060471Z "size": 106, 2025-08-14T21:26:27.1060709Z "digest": "sha256:907c320fee2f90da0cf5028c90a0ef49a137518baf79b483dcf7f22d5a0a497d" 2025-08-14T21:26:27.1060982Z }, 2025-08-14T21:26:27.1061106Z { 2025-08-14T21:26:27.1061302Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1061534Z "size": 425, 2025-08-14T21:26:27.1061779Z "digest": "sha256:18c4ed1ec491095788e352ae018afd84de0f251fbcfb8f74d5d893e1e9ab196d" 2025-08-14T21:26:27.1062050Z }, 2025-08-14T21:26:27.1062167Z { 2025-08-14T21:26:27.1062367Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1062612Z "size": 19308711, 2025-08-14T21:26:27.1062871Z "digest": "sha256:d7618c2df6cdb4bbf3d9870ba2d089094ac46c429b573d9adb94411fac54cfca" 2025-08-14T21:26:27.1063152Z }, 2025-08-14T21:26:27.1063319Z { 2025-08-14T21:26:27.1063516Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1063758Z "size": 108, 2025-08-14T21:26:27.1064017Z "digest": "sha256:b7bdd9a6f789ba483a46c92e5d373638850f33e88b1baa4bbe67e1c6a09cb7d0" 2025-08-14T21:26:27.1064295Z }, 2025-08-14T21:26:27.1064413Z { 2025-08-14T21:26:27.1064614Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1064858Z "size": 691, 2025-08-14T21:26:27.1065103Z "digest": "sha256:6738ba83282e002d92bff3d2b4951e3c1a67f5ec2c1bad2fd780c2f5d444748f" 2025-08-14T21:26:27.1065378Z }, 2025-08-14T21:26:27.1065510Z { 2025-08-14T21:26:27.1065707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1065951Z "size": 724, 2025-08-14T21:26:27.1066203Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:26:27.1066479Z }, 2025-08-14T21:26:27.1066683Z { 2025-08-14T21:26:27.1066905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1067145Z "size": 116, 2025-08-14T21:26:27.1067399Z "digest": "sha256:dfb0f24886393e1d394f1f433dc9346026679dafd7a60c3a93de17d94078c1ca" 2025-08-14T21:26:27.1067678Z }, 2025-08-14T21:26:27.1067841Z { 2025-08-14T21:26:27.1068037Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1068283Z "size": 136, 2025-08-14T21:26:27.1068532Z "digest": "sha256:dc833b0762f2e144670a660f6b7ce62cec71a5fdd24df4e67b5c6173d5834451" 2025-08-14T21:26:27.1068798Z }, 2025-08-14T21:26:27.1068922Z { 2025-08-14T21:26:27.1069123Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1069360Z "size": 139, 2025-08-14T21:26:27.1069609Z "digest": "sha256:8827df8ca2da347e0032d1bff3b0312437f711c5d0b5f2164f8a60c3368a9827" 2025-08-14T21:26:27.1069884Z }, 2025-08-14T21:26:27.1070003Z { 2025-08-14T21:26:27.1070211Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1070457Z "size": 17672683360, 2025-08-14T21:26:27.1070721Z "digest": "sha256:fac8f3bd0f85eaffb43df539683dc3d861c370e583623253559fd7a1f5b00229" 2025-08-14T21:26:27.1071001Z }, 2025-08-14T21:26:27.1071127Z { 2025-08-14T21:26:27.1071329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1071567Z "size": 214, 2025-08-14T21:26:27.1071823Z "digest": "sha256:d7cf7f140df32761610e1d58686db7f7c66a85affa4bb4b9d3c245e232443a8f" 2025-08-14T21:26:27.1072103Z }, 2025-08-14T21:26:27.1072223Z { 2025-08-14T21:26:27.1072430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1072683Z "size": 272992162, 2025-08-14T21:26:27.1072951Z "digest": "sha256:733eedc8da8d8e7bd5a85a58d3d7818f14ed9a4fdf2dbd587038bb7725fbb9f7" 2025-08-14T21:26:27.1073235Z }, 2025-08-14T21:26:27.1073364Z { 2025-08-14T21:26:27.1073560Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1073818Z "size": 6435582332, 2025-08-14T21:26:27.1074086Z "digest": "sha256:5b092eb06909a2ea8906849acac588a10864da349670d65c0bfea342187edba2" 2025-08-14T21:26:27.1074363Z }, 2025-08-14T21:26:27.1074485Z { 2025-08-14T21:26:27.1074690Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1074964Z "size": 129, 2025-08-14T21:26:27.1075219Z "digest": "sha256:bc596103109216e154006085503386753b0b114b5900bf44758cdff324df5504" 2025-08-14T21:26:27.1075505Z }, 2025-08-14T21:26:27.1075640Z { 2025-08-14T21:26:27.1075854Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1076250Z "size": 776, 2025-08-14T21:26:27.1076534Z "digest": "sha256:0531cc34c12ab9127f1858c4cf365bb3a02bc31e8d6df5eabba2e1b6ef026ccf" 2025-08-14T21:26:27.1076836Z }, 2025-08-14T21:26:27.1076980Z { 2025-08-14T21:26:27.1077205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1077560Z "size": 724, 2025-08-14T21:26:27.1077856Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:26:27.1078149Z }, 2025-08-14T21:26:27.1078288Z { 2025-08-14T21:26:27.1078500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1078771Z "size": 141, 2025-08-14T21:26:27.1079041Z "digest": "sha256:38c303d3b62eb463762816db04062a480014a6f3c9754386f3e83ba331ab4d1d" 2025-08-14T21:26:27.1079326Z }, 2025-08-14T21:26:27.1079465Z { 2025-08-14T21:26:27.1079685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1079942Z "size": 32, 2025-08-14T21:26:27.1080227Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1080530Z }, 2025-08-14T21:26:27.1080658Z { 2025-08-14T21:26:27.1080886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1081163Z "size": 160, 2025-08-14T21:26:27.1081470Z "digest": "sha256:e06d15594a2a76995baebbce7032946ff9f94e281246fbc3f8ab19d8bcc38b81" 2025-08-14T21:26:27.1081768Z }, 2025-08-14T21:26:27.1081905Z { 2025-08-14T21:26:27.1082123Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1082428Z "size": 1010, 2025-08-14T21:26:27.1082724Z "digest": "sha256:0e55deb5cb38fd36b600183f7d86eaca0dabc04d2ff4d49ec2266ee3329edc4a" 2025-08-14T21:26:27.1083027Z }, 2025-08-14T21:26:27.1083157Z { 2025-08-14T21:26:27.1083375Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1083689Z "size": 724, 2025-08-14T21:26:27.1083947Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:26:27.1084241Z }, 2025-08-14T21:26:27.1084375Z { 2025-08-14T21:26:27.1084582Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1084854Z "size": 134, 2025-08-14T21:26:27.1085129Z "digest": "sha256:4a53d66dce071bb7416414aa1adbc3e4a59003300c0d42038612fabdeb5a1b01" 2025-08-14T21:26:27.1085435Z }, 2025-08-14T21:26:27.1085562Z { 2025-08-14T21:26:27.1085774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1086044Z "size": 32, 2025-08-14T21:26:27.1086307Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1086666Z }, 2025-08-14T21:26:27.1086794Z { 2025-08-14T21:26:27.1086991Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1087243Z "size": 159, 2025-08-14T21:26:27.1087499Z "digest": "sha256:1519daa051b8b80e04125f2f2215dc412dcdbb9502711925e97aeccbda069eaf" 2025-08-14T21:26:27.1087772Z }, 2025-08-14T21:26:27.1087902Z { 2025-08-14T21:26:27.1088108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1088354Z "size": 1371, 2025-08-14T21:26:27.1088621Z "digest": "sha256:381ed91d2119f078fbba19102a65befc4cb242f8cf47a11fb6f76ea424690692" 2025-08-14T21:26:27.1088913Z }, 2025-08-14T21:26:27.1089047Z { 2025-08-14T21:26:27.1089243Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1089495Z "size": 32, 2025-08-14T21:26:27.1089752Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1090028Z }, 2025-08-14T21:26:27.1090159Z { 2025-08-14T21:26:27.1090364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1090608Z "size": 137, 2025-08-14T21:26:27.1090867Z "digest": "sha256:c6b0a01a96dd479640297d4b012031ffc1bd9fc0daf61d86058f9b675c0a0705" 2025-08-14T21:26:27.1091147Z }, 2025-08-14T21:26:27.1091268Z { 2025-08-14T21:26:27.1091473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1091725Z "size": 380, 2025-08-14T21:26:27.1091976Z "digest": "sha256:62df6413daeefebde04dcc401134734952e4ea37fc85ff23c89cb9b4fbd45155" 2025-08-14T21:26:27.1092264Z }, 2025-08-14T21:26:27.1092394Z { 2025-08-14T21:26:27.1092648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1092894Z "size": 32, 2025-08-14T21:26:27.1093162Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1093452Z }, 2025-08-14T21:26:27.1093580Z { 2025-08-14T21:26:27.1093795Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1094063Z "size": 104, 2025-08-14T21:26:27.1094339Z "digest": "sha256:7a18bc2a6881b76a6f591c98dafb47e44d903f7a905f7eba0fc3aedb5c90fff7" 2025-08-14T21:26:27.1094653Z }, 2025-08-14T21:26:27.1094795Z { 2025-08-14T21:26:27.1095011Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1095287Z "size": 407, 2025-08-14T21:26:27.1095549Z "digest": "sha256:93359cd58a8cece344fd4291b27647e57761c9399bb54bb0c18149c12af5f66a" 2025-08-14T21:26:27.1095836Z }, 2025-08-14T21:26:27.1095995Z { 2025-08-14T21:26:27.1096201Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1096454Z "size": 32, 2025-08-14T21:26:27.1096701Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1096986Z }, 2025-08-14T21:26:27.1097117Z { 2025-08-14T21:26:27.1097316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1097569Z "size": 109, 2025-08-14T21:26:27.1097830Z "digest": "sha256:c35ba0a1f353d6894c914a4bfbea9a2c9b8ac1b526af64d34cbe9a12bd83c78e" 2025-08-14T21:26:27.1098110Z }, 2025-08-14T21:26:27.1098242Z { 2025-08-14T21:26:27.1098447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1098691Z "size": 1896, 2025-08-14T21:26:27.1098953Z "digest": "sha256:dcf1e01c98d6a6f72674d79a4e8e4047b54796576cd06ad682c225a92820a8f5" 2025-08-14T21:26:27.1099236Z }, 2025-08-14T21:26:27.1099366Z { 2025-08-14T21:26:27.1099568Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1099826Z "size": 242635753, 2025-08-14T21:26:27.1100102Z "digest": "sha256:bad0564f61fdf377e3ae31f6fec0ec28b6922da0b9db28408b55b8e97ff1ea51" 2025-08-14T21:26:27.1100383Z }, 2025-08-14T21:26:27.1100513Z { 2025-08-14T21:26:27.1100723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1100965Z "size": 106, 2025-08-14T21:26:27.1101223Z "digest": "sha256:539ded9057364aade7abe23ab908d2caf53966a186734aa58ae84a56bee659eb" 2025-08-14T21:26:27.1101506Z }, 2025-08-14T21:26:27.1101626Z { 2025-08-14T21:26:27.1101830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1102080Z "size": 163, 2025-08-14T21:26:27.1102321Z "digest": "sha256:28d482062637d32514edfc447913e98745d7c13d2f277531e64ffcf090ae6d92" 2025-08-14T21:26:27.1102594Z }, 2025-08-14T21:26:27.1102723Z { 2025-08-14T21:26:27.1102931Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1103177Z "size": 7943, 2025-08-14T21:26:27.1103437Z "digest": "sha256:3245316ff51b50b27da4ef7279733c92f76cc652b3fce3877c0e3d510430e8b3" 2025-08-14T21:26:27.1103719Z }, 2025-08-14T21:26:27.1103843Z { 2025-08-14T21:26:27.1104049Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1104301Z "size": 8073, 2025-08-14T21:26:27.1104553Z "digest": "sha256:b53167d1a6df0e4b67d637d073150dff1fb87a823864c0c98d77c15e56babc24" 2025-08-14T21:26:27.1104835Z }, 2025-08-14T21:26:27.1104965Z { 2025-08-14T21:26:27.1105164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1105414Z "size": 303, 2025-08-14T21:26:27.1105668Z "digest": "sha256:7f5277f691672469f431fd90a8c2bb702c6c68333f6be2cff868f00e416c5a1a" 2025-08-14T21:26:27.1105945Z }, 2025-08-14T21:26:27.1106069Z { 2025-08-14T21:26:27.1106275Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1106528Z "size": 32, 2025-08-14T21:26:27.1106813Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1107103Z }, 2025-08-14T21:26:27.1107233Z { 2025-08-14T21:26:27.1107431Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1107679Z "size": 108, 2025-08-14T21:26:27.1107936Z "digest": "sha256:23dff10cdaa5b1e9c7250f0c58a6279f104b35408281e951bfe9983f97e3d9ed" 2025-08-14T21:26:27.1108211Z }, 2025-08-14T21:26:27.1108338Z { 2025-08-14T21:26:27.1108543Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1109040Z "size": 54145699, 2025-08-14T21:26:27.1109338Z "digest": "sha256:9fb73296da6ac15f37f36663bd10afc98abb8a01fb40bff4848de7247d28e018" 2025-08-14T21:26:27.1109642Z }, 2025-08-14T21:26:27.1109784Z { 2025-08-14T21:26:27.1110000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:26:27.1110266Z "size": 32, 2025-08-14T21:26:27.1110624Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:26:27.1110920Z } 2025-08-14T21:26:27.1111060Z ] 2025-08-14T21:26:27.1111205Z } 2025-08-14T21:26:27.1111359Z + exit 0 2025-08-14T21:26:27.1131572Z ##[group]Run set -eux 2025-08-14T21:26:27.1131796Z set -eux 2025-08-14T21:26:27.1132359Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2025-08-14T21:26:27.1139081Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:27.1139340Z env: 2025-08-14T21:26:27.1139511Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:27.1139704Z ##[endgroup] 2025-08-14T21:26:27.1166141Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-08-14T21:26:27.1166533Z + jq --raw-output .SecretString 2025-08-14T21:26:27.1173392Z + jq -r .docker_hub_readonly_token 2025-08-14T21:26:27.1173848Z + docker login --username pytorchbot --password-stdin 2025-08-14T21:26:27.5947086Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-14T21:26:27.5947540Z Configure a credential helper to remove this warning. See 2025-08-14T21:26:27.5947940Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-14T21:26:27.5948206Z 2025-08-14T21:26:27.5950848Z Login Succeeded 2025-08-14T21:26:27.6023869Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-08-14T21:26:27.6024135Z tag=${ECR_DOCKER_IMAGE##*:} 2025-08-14T21:26:27.6024394Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-08-14T21:26:27.6029029Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:27.6029281Z env: 2025-08-14T21:26:27.6029440Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:27.6030012Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:27.6030595Z ##[endgroup] 2025-08-14T21:26:27.6059244Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:27.6092438Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-08-14T21:26:27.6092768Z with: 2025-08-14T21:26:27.6093379Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:27.6094070Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:27.6094365Z env: 2025-08-14T21:26:27.6094547Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:27.6094753Z ##[endgroup] 2025-08-14T21:26:27.6105719Z ##[group]Run set -x 2025-08-14T21:26:27.6105956Z set -x 2025-08-14T21:26:27.6106148Z set +e 2025-08-14T21:26:27.6106334Z  2025-08-14T21:26:27.6106506Z login() { 2025-08-14T21:26:27.6106892Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-08-14T21:26:27.6107274Z } 2025-08-14T21:26:27.6107444Z  2025-08-14T21:26:27.6107676Z retry () { 2025-08-14T21:26:27.6107900Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-08-14T21:26:27.6108151Z } 2025-08-14T21:26:27.6108325Z  2025-08-14T21:26:27.6108518Z retry login "${DOCKER_REGISTRY}" 2025-08-14T21:26:27.6109185Z  2025-08-14T21:26:27.6109570Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-08-14T21:26:27.6110068Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-08-14T21:26:27.6110365Z  2025-08-14T21:26:27.6110549Z set -e 2025-08-14T21:26:27.6110840Z # ignore output since only exit code is used for conditional 2025-08-14T21:26:27.6111335Z # only pull docker image if it's not available locally 2025-08-14T21:26:27.6111740Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-08-14T21:26:27.6112135Z  retry docker pull "${DOCKER_IMAGE}" 2025-08-14T21:26:27.6112397Z fi 2025-08-14T21:26:27.6117543Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:26:27.6117841Z env: 2025-08-14T21:26:27.6118033Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:26:27.6118684Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:27.6119401Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:27.6119697Z ##[endgroup] 2025-08-14T21:26:27.6142476Z + set +e 2025-08-14T21:26:27.6142979Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:27.6143649Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:27.6144006Z + aws ecr get-login-password --region us-east-1 2025-08-14T21:26:27.6144365Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:26:28.0335415Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-14T21:26:28.0335896Z Configure a credential helper to remove this warning. See 2025-08-14T21:26:28.0336293Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-14T21:26:28.0336566Z 2025-08-14T21:26:28.0336789Z Login Succeeded 2025-08-14T21:26:28.0364810Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-08-14T21:26:28.0365655Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:28.2650376Z + IMAGE_SIZE=27663.483686447144 2025-08-14T21:26:28.2650955Z + echo 'Compressed size of image in MB: 27663.483686447144' 2025-08-14T21:26:28.2651315Z Compressed size of image in MB: 27663.483686447144 2025-08-14T21:26:28.2652117Z + set -e 2025-08-14T21:26:28.2653342Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:28.2797798Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:28.2798820Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:26:28.5201088Z pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe: Pulling from pytorch/ci-image 2025-08-14T21:26:28.5201732Z 660ffc76f83b: Pulling fs layer 2025-08-14T21:26:28.5201954Z c7b4a852a455: Pulling fs layer 2025-08-14T21:26:28.5202237Z e5a28988c893: Pulling fs layer 2025-08-14T21:26:28.5202435Z 76a69b57b683: Pulling fs layer 2025-08-14T21:26:28.5202625Z 5c785dcb4cdb: Pulling fs layer 2025-08-14T21:26:28.5202820Z 836ab08052e8: Pulling fs layer 2025-08-14T21:26:28.5203023Z 53b11c77468c: Pulling fs layer 2025-08-14T21:26:28.5203208Z e97311a6a967: Pulling fs layer 2025-08-14T21:26:28.5203397Z 2c414689d31d: Pulling fs layer 2025-08-14T21:26:28.5203610Z 6d89b5f065d5: Pulling fs layer 2025-08-14T21:26:28.5203810Z 5a5cc76ada43: Pulling fs layer 2025-08-14T21:26:28.5204052Z fc6b37d40530: Pulling fs layer 2025-08-14T21:26:28.5204243Z 2e1657907860: Pulling fs layer 2025-08-14T21:26:28.5204432Z 7b92d7a4b8c7: Pulling fs layer 2025-08-14T21:26:28.5204616Z 4f4fb700ef54: Pulling fs layer 2025-08-14T21:26:28.5204809Z d6226eb61f82: Pulling fs layer 2025-08-14T21:26:28.5205004Z 83c70f4266a6: Pulling fs layer 2025-08-14T21:26:28.5205184Z 60c725d21861: Pulling fs layer 2025-08-14T21:26:28.5205372Z a504e76e66a4: Pulling fs layer 2025-08-14T21:26:28.5205808Z fc1c200a4f77: Pulling fs layer 2025-08-14T21:26:28.5205996Z 43273c22704f: Pulling fs layer 2025-08-14T21:26:28.5206192Z 89df389d042a: Pulling fs layer 2025-08-14T21:26:28.5206385Z 684349f50d94: Pulling fs layer 2025-08-14T21:26:28.5206622Z 21d0eae87fb3: Pulling fs layer 2025-08-14T21:26:28.5206806Z c9c2b424b8e0: Pulling fs layer 2025-08-14T21:26:28.5206998Z 98dda28f3395: Pulling fs layer 2025-08-14T21:26:28.5207188Z acf5babd87f2: Pulling fs layer 2025-08-14T21:26:28.5207371Z 7c5050d8408d: Pulling fs layer 2025-08-14T21:26:28.5207562Z 7ddd14e2b548: Pulling fs layer 2025-08-14T21:26:28.5207756Z 4ba8e7a736c8: Pulling fs layer 2025-08-14T21:26:28.5207938Z 907c320fee2f: Pulling fs layer 2025-08-14T21:26:28.5208131Z 18c4ed1ec491: Pulling fs layer 2025-08-14T21:26:28.5208319Z d7618c2df6cd: Pulling fs layer 2025-08-14T21:26:28.5208502Z b7bdd9a6f789: Pulling fs layer 2025-08-14T21:26:28.5208888Z 6738ba83282e: Pulling fs layer 2025-08-14T21:26:28.5209093Z dfb0f2488639: Pulling fs layer 2025-08-14T21:26:28.5209282Z dc833b0762f2: Pulling fs layer 2025-08-14T21:26:28.5209481Z 8827df8ca2da: Pulling fs layer 2025-08-14T21:26:28.5209681Z fac8f3bd0f85: Pulling fs layer 2025-08-14T21:26:28.5209880Z d7cf7f140df3: Pulling fs layer 2025-08-14T21:26:28.5210070Z 733eedc8da8d: Pulling fs layer 2025-08-14T21:26:28.5210409Z 5b092eb06909: Pulling fs layer 2025-08-14T21:26:28.5210610Z bc5961031092: Pulling fs layer 2025-08-14T21:26:28.5210787Z 0531cc34c12a: Pulling fs layer 2025-08-14T21:26:28.5211043Z 38c303d3b62e: Pulling fs layer 2025-08-14T21:26:28.5211240Z e06d15594a2a: Pulling fs layer 2025-08-14T21:26:28.5211419Z 0e55deb5cb38: Pulling fs layer 2025-08-14T21:26:28.5211609Z 4a53d66dce07: Pulling fs layer 2025-08-14T21:26:28.5211796Z 1519daa051b8: Pulling fs layer 2025-08-14T21:26:28.5211974Z 381ed91d2119: Pulling fs layer 2025-08-14T21:26:28.5212160Z c6b0a01a96dd: Pulling fs layer 2025-08-14T21:26:28.5212353Z 62df6413daee: Pulling fs layer 2025-08-14T21:26:28.5212534Z 7a18bc2a6881: Pulling fs layer 2025-08-14T21:26:28.5212781Z 93359cd58a8c: Pulling fs layer 2025-08-14T21:26:28.5212981Z c35ba0a1f353: Pulling fs layer 2025-08-14T21:26:28.5213173Z dcf1e01c98d6: Pulling fs layer 2025-08-14T21:26:28.5213358Z bad0564f61fd: Pulling fs layer 2025-08-14T21:26:28.5213682Z 539ded905736: Pulling fs layer 2025-08-14T21:26:28.5213877Z 28d482062637: Pulling fs layer 2025-08-14T21:26:28.5214054Z 3245316ff51b: Pulling fs layer 2025-08-14T21:26:28.5214244Z b53167d1a6df: Pulling fs layer 2025-08-14T21:26:28.5214433Z 7f5277f69167: Pulling fs layer 2025-08-14T21:26:28.5214615Z 23dff10cdaa5: Pulling fs layer 2025-08-14T21:26:28.5214811Z 9fb73296da6a: Pulling fs layer 2025-08-14T21:26:28.5220252Z c9c2b424b8e0: Waiting 2025-08-14T21:26:28.5220738Z 733eedc8da8d: Waiting 2025-08-14T21:26:28.5220930Z 5b092eb06909: Waiting 2025-08-14T21:26:28.5221104Z bc5961031092: Waiting 2025-08-14T21:26:28.5221278Z 1519daa051b8: Waiting 2025-08-14T21:26:28.5221440Z 907c320fee2f: Waiting 2025-08-14T21:26:28.5221613Z 0531cc34c12a: Waiting 2025-08-14T21:26:28.5221808Z dcf1e01c98d6: Waiting 2025-08-14T21:26:28.5221965Z 381ed91d2119: Waiting 2025-08-14T21:26:28.5222130Z bad0564f61fd: Waiting 2025-08-14T21:26:28.5222299Z 38c303d3b62e: Waiting 2025-08-14T21:26:28.5222457Z 539ded905736: Waiting 2025-08-14T21:26:28.5222626Z 98dda28f3395: Waiting 2025-08-14T21:26:28.5222811Z 836ab08052e8: Waiting 2025-08-14T21:26:28.5222969Z 62df6413daee: Waiting 2025-08-14T21:26:28.5223138Z 3245316ff51b: Waiting 2025-08-14T21:26:28.5223306Z b53167d1a6df: Waiting 2025-08-14T21:26:28.5223473Z c6b0a01a96dd: Waiting 2025-08-14T21:26:28.5223631Z 2c414689d31d: Waiting 2025-08-14T21:26:28.5223795Z 7f5277f69167: Waiting 2025-08-14T21:26:28.5223962Z e06d15594a2a: Waiting 2025-08-14T21:26:28.5224122Z 2e1657907860: Waiting 2025-08-14T21:26:28.5224286Z 0e55deb5cb38: Waiting 2025-08-14T21:26:28.5224994Z 7b92d7a4b8c7: Waiting 2025-08-14T21:26:28.5225171Z 4a53d66dce07: Waiting 2025-08-14T21:26:28.5225337Z b7bdd9a6f789: Waiting 2025-08-14T21:26:28.5225500Z 6738ba83282e: Waiting 2025-08-14T21:26:28.5225844Z 6d89b5f065d5: Waiting 2025-08-14T21:26:28.5226010Z 7c5050d8408d: Waiting 2025-08-14T21:26:28.5226177Z a504e76e66a4: Waiting 2025-08-14T21:26:28.5226393Z 5a5cc76ada43: Waiting 2025-08-14T21:26:28.5226565Z 60c725d21861: Waiting 2025-08-14T21:26:28.5226739Z fc6b37d40530: Waiting 2025-08-14T21:26:28.5226899Z d6226eb61f82: Waiting 2025-08-14T21:26:28.5227064Z 4f4fb700ef54: Waiting 2025-08-14T21:26:28.5227235Z 93359cd58a8c: Waiting 2025-08-14T21:26:28.5227390Z 43273c22704f: Waiting 2025-08-14T21:26:28.5227554Z 76a69b57b683: Waiting 2025-08-14T21:26:28.5227720Z 28d482062637: Waiting 2025-08-14T21:26:28.5227874Z 7ddd14e2b548: Waiting 2025-08-14T21:26:28.5228043Z 89df389d042a: Waiting 2025-08-14T21:26:28.5228211Z 23dff10cdaa5: Waiting 2025-08-14T21:26:28.5228370Z 9fb73296da6a: Waiting 2025-08-14T21:26:28.5228533Z 684349f50d94: Waiting 2025-08-14T21:26:28.5228692Z 53b11c77468c: Waiting 2025-08-14T21:26:28.5228858Z c35ba0a1f353: Waiting 2025-08-14T21:26:28.5229016Z dfb0f2488639: Waiting 2025-08-14T21:26:28.5229180Z 18c4ed1ec491: Waiting 2025-08-14T21:26:28.5229354Z 8827df8ca2da: Waiting 2025-08-14T21:26:28.5229515Z fac8f3bd0f85: Waiting 2025-08-14T21:26:28.5229683Z fc1c200a4f77: Waiting 2025-08-14T21:26:28.5229849Z e97311a6a967: Waiting 2025-08-14T21:26:28.5230009Z d7cf7f140df3: Waiting 2025-08-14T21:26:28.5230183Z dc833b0762f2: Waiting 2025-08-14T21:26:28.5230343Z 83c70f4266a6: Waiting 2025-08-14T21:26:28.5230498Z d7618c2df6cd: Waiting 2025-08-14T21:26:28.5230662Z 21d0eae87fb3: Waiting 2025-08-14T21:26:28.5230827Z acf5babd87f2: Waiting 2025-08-14T21:26:28.5230985Z 7a18bc2a6881: Waiting 2025-08-14T21:26:28.5231150Z 5c785dcb4cdb: Waiting 2025-08-14T21:26:28.6057574Z c7b4a852a455: Verifying Checksum 2025-08-14T21:26:28.6058587Z c7b4a852a455: Download complete 2025-08-14T21:26:28.6711056Z 76a69b57b683: Download complete 2025-08-14T21:26:28.7993763Z 5c785dcb4cdb: Verifying Checksum 2025-08-14T21:26:28.7994088Z 5c785dcb4cdb: Download complete 2025-08-14T21:26:28.8677395Z 660ffc76f83b: Verifying Checksum 2025-08-14T21:26:28.8677758Z 660ffc76f83b: Download complete 2025-08-14T21:26:28.9137150Z 836ab08052e8: Verifying Checksum 2025-08-14T21:26:28.9137541Z 836ab08052e8: Download complete 2025-08-14T21:26:28.9382051Z 53b11c77468c: Verifying Checksum 2025-08-14T21:26:28.9382847Z 53b11c77468c: Download complete 2025-08-14T21:26:28.9874738Z e97311a6a967: Verifying Checksum 2025-08-14T21:26:28.9875063Z e97311a6a967: Download complete 2025-08-14T21:26:29.0701480Z 6d89b5f065d5: Verifying Checksum 2025-08-14T21:26:29.0701866Z 6d89b5f065d5: Download complete 2025-08-14T21:26:29.1539437Z 5a5cc76ada43: Verifying Checksum 2025-08-14T21:26:29.1542159Z 5a5cc76ada43: Download complete 2025-08-14T21:26:29.2383782Z fc6b37d40530: Verifying Checksum 2025-08-14T21:26:29.2384073Z fc6b37d40530: Download complete 2025-08-14T21:26:29.3185751Z 2e1657907860: Download complete 2025-08-14T21:26:30.0160639Z 660ffc76f83b: Pull complete 2025-08-14T21:26:30.0285703Z c7b4a852a455: Pull complete 2025-08-14T21:26:30.1056009Z 2c414689d31d: Verifying Checksum 2025-08-14T21:26:30.1060848Z 2c414689d31d: Download complete 2025-08-14T21:26:30.1127904Z 4f4fb700ef54: Verifying Checksum 2025-08-14T21:26:30.1128173Z 4f4fb700ef54: Download complete 2025-08-14T21:26:30.1968606Z d6226eb61f82: Verifying Checksum 2025-08-14T21:26:30.1968952Z d6226eb61f82: Download complete 2025-08-14T21:26:30.2804631Z 83c70f4266a6: Download complete 2025-08-14T21:26:30.3775685Z 60c725d21861: Download complete 2025-08-14T21:26:30.5113058Z a504e76e66a4: Verifying Checksum 2025-08-14T21:26:30.5113370Z a504e76e66a4: Download complete 2025-08-14T21:26:30.5966605Z fc1c200a4f77: Verifying Checksum 2025-08-14T21:26:30.5966906Z fc1c200a4f77: Download complete 2025-08-14T21:26:30.7565879Z 89df389d042a: Verifying Checksum 2025-08-14T21:26:30.7572939Z 89df389d042a: Download complete 2025-08-14T21:26:30.8327252Z 684349f50d94: Verifying Checksum 2025-08-14T21:26:30.8330933Z 684349f50d94: Download complete 2025-08-14T21:26:30.9039673Z 21d0eae87fb3: Verifying Checksum 2025-08-14T21:26:30.9039966Z 21d0eae87fb3: Download complete 2025-08-14T21:26:30.9988618Z c9c2b424b8e0: Verifying Checksum 2025-08-14T21:26:30.9988966Z c9c2b424b8e0: Download complete 2025-08-14T21:26:31.0745159Z 98dda28f3395: Verifying Checksum 2025-08-14T21:26:31.0745505Z 98dda28f3395: Download complete 2025-08-14T21:26:31.1491773Z acf5babd87f2: Verifying Checksum 2025-08-14T21:26:31.1492133Z acf5babd87f2: Download complete 2025-08-14T21:26:31.7031410Z e5a28988c893: Verifying Checksum 2025-08-14T21:26:31.7031996Z e5a28988c893: Download complete 2025-08-14T21:26:31.7805552Z 7ddd14e2b548: Verifying Checksum 2025-08-14T21:26:31.7805879Z 7ddd14e2b548: Download complete 2025-08-14T21:26:31.8553063Z 4ba8e7a736c8: Download complete 2025-08-14T21:26:32.0153135Z 18c4ed1ec491: Verifying Checksum 2025-08-14T21:26:32.0153515Z 18c4ed1ec491: Download complete 2025-08-14T21:26:32.2592306Z d7618c2df6cd: Verifying Checksum 2025-08-14T21:26:32.2592830Z d7618c2df6cd: Download complete 2025-08-14T21:26:32.3295117Z b7bdd9a6f789: Verifying Checksum 2025-08-14T21:26:32.3295789Z b7bdd9a6f789: Download complete 2025-08-14T21:26:32.4133638Z 6738ba83282e: Verifying Checksum 2025-08-14T21:26:32.4134200Z 6738ba83282e: Download complete 2025-08-14T21:26:32.4902783Z dfb0f2488639: Verifying Checksum 2025-08-14T21:26:32.4903144Z dfb0f2488639: Download complete 2025-08-14T21:26:32.5636577Z dc833b0762f2: Verifying Checksum 2025-08-14T21:26:32.5636892Z dc833b0762f2: Download complete 2025-08-14T21:26:32.6396467Z 8827df8ca2da: Download complete 2025-08-14T21:26:35.7374865Z 7c5050d8408d: Verifying Checksum 2025-08-14T21:26:35.7375179Z 7c5050d8408d: Download complete 2025-08-14T21:26:35.8260071Z d7cf7f140df3: Download complete 2025-08-14T21:26:38.6483513Z 733eedc8da8d: Verifying Checksum 2025-08-14T21:26:38.6484051Z 733eedc8da8d: Download complete 2025-08-14T21:26:42.5910053Z e5a28988c893: Pull complete 2025-08-14T21:26:42.9217348Z 76a69b57b683: Pull complete 2025-08-14T21:26:43.1709049Z 5c785dcb4cdb: Pull complete 2025-08-14T21:26:43.4515922Z 836ab08052e8: Pull complete 2025-08-14T21:26:43.7554305Z 53b11c77468c: Pull complete 2025-08-14T21:26:44.0093670Z e97311a6a967: Pull complete 2025-08-14T21:26:47.5683501Z 2c414689d31d: Pull complete 2025-08-14T21:26:47.8308849Z 6d89b5f065d5: Pull complete 2025-08-14T21:26:48.1089132Z 5a5cc76ada43: Pull complete 2025-08-14T21:26:48.4535905Z fc6b37d40530: Pull complete 2025-08-14T21:26:48.8676636Z 2e1657907860: Pull complete 2025-08-14T21:27:03.3703992Z 7b92d7a4b8c7: Verifying Checksum 2025-08-14T21:27:03.3704296Z 7b92d7a4b8c7: Download complete 2025-08-14T21:27:03.4711070Z bc5961031092: Download complete 2025-08-14T21:27:03.5581553Z 0531cc34c12a: Download complete 2025-08-14T21:27:03.6471237Z 38c303d3b62e: Verifying Checksum 2025-08-14T21:27:03.6471681Z 38c303d3b62e: Download complete 2025-08-14T21:27:03.7214918Z e06d15594a2a: Verifying Checksum 2025-08-14T21:27:03.7215324Z e06d15594a2a: Download complete 2025-08-14T21:27:03.7907646Z 0e55deb5cb38: Verifying Checksum 2025-08-14T21:27:03.7909624Z 0e55deb5cb38: Download complete 2025-08-14T21:27:03.8643240Z 4a53d66dce07: Verifying Checksum 2025-08-14T21:27:03.8644229Z 4a53d66dce07: Download complete 2025-08-14T21:27:03.9436349Z 1519daa051b8: Verifying Checksum 2025-08-14T21:27:03.9436675Z 1519daa051b8: Download complete 2025-08-14T21:27:04.0317586Z 381ed91d2119: Download complete 2025-08-14T21:27:04.1362631Z c6b0a01a96dd: Verifying Checksum 2025-08-14T21:27:04.1363098Z c6b0a01a96dd: Download complete 2025-08-14T21:27:04.2133625Z 62df6413daee: Download complete 2025-08-14T21:27:04.2850884Z 7a18bc2a6881: Verifying Checksum 2025-08-14T21:27:04.2851155Z 7a18bc2a6881: Download complete 2025-08-14T21:27:04.3655862Z 93359cd58a8c: Verifying Checksum 2025-08-14T21:27:04.3656157Z 93359cd58a8c: Download complete 2025-08-14T21:27:04.4497475Z c35ba0a1f353: Verifying Checksum 2025-08-14T21:27:04.4497792Z c35ba0a1f353: Download complete 2025-08-14T21:27:04.5361651Z dcf1e01c98d6: Verifying Checksum 2025-08-14T21:27:04.5361959Z dcf1e01c98d6: Download complete 2025-08-14T21:27:07.0304495Z bad0564f61fd: Verifying Checksum 2025-08-14T21:27:07.0305062Z bad0564f61fd: Download complete 2025-08-14T21:27:07.1515431Z 539ded905736: Verifying Checksum 2025-08-14T21:27:07.1515751Z 539ded905736: Download complete 2025-08-14T21:27:07.2348512Z 28d482062637: Verifying Checksum 2025-08-14T21:27:07.2348872Z 28d482062637: Download complete 2025-08-14T21:27:07.3158637Z 3245316ff51b: Verifying Checksum 2025-08-14T21:27:07.3158938Z 3245316ff51b: Download complete 2025-08-14T21:27:07.4424985Z b53167d1a6df: Download complete 2025-08-14T21:27:07.5371090Z 7f5277f69167: Download complete 2025-08-14T21:27:07.6098461Z 23dff10cdaa5: Verifying Checksum 2025-08-14T21:27:07.6098784Z 23dff10cdaa5: Download complete 2025-08-14T21:27:08.2005795Z 9fb73296da6a: Verifying Checksum 2025-08-14T21:27:08.2006100Z 9fb73296da6a: Download complete 2025-08-14T21:27:43.0522846Z 5b092eb06909: Verifying Checksum 2025-08-14T21:27:43.0523169Z 5b092eb06909: Download complete 2025-08-14T21:28:17.1633813Z 7b92d7a4b8c7: Pull complete 2025-08-14T21:28:17.4971275Z 4f4fb700ef54: Pull complete 2025-08-14T21:28:17.9011665Z d6226eb61f82: Pull complete 2025-08-14T21:28:18.3547839Z 83c70f4266a6: Pull complete 2025-08-14T21:28:18.7932360Z 60c725d21861: Pull complete 2025-08-14T21:28:19.3511088Z a504e76e66a4: Pull complete 2025-08-14T21:28:19.7041431Z fc1c200a4f77: Pull complete 2025-08-14T21:28:20.1164618Z 43273c22704f: Pull complete 2025-08-14T21:28:20.4218740Z 89df389d042a: Pull complete 2025-08-14T21:28:20.6911191Z 684349f50d94: Pull complete 2025-08-14T21:28:21.0362345Z 21d0eae87fb3: Pull complete 2025-08-14T21:28:21.1998067Z c9c2b424b8e0: Pull complete 2025-08-14T21:28:21.8335278Z 98dda28f3395: Pull complete 2025-08-14T21:28:22.3422159Z acf5babd87f2: Pull complete 2025-08-14T21:28:34.1548472Z 7c5050d8408d: Pull complete 2025-08-14T21:28:34.3320348Z 7ddd14e2b548: Pull complete 2025-08-14T21:28:34.5944958Z 4ba8e7a736c8: Pull complete 2025-08-14T21:28:35.4773130Z 907c320fee2f: Pull complete 2025-08-14T21:28:35.8677993Z 18c4ed1ec491: Pull complete 2025-08-14T21:28:36.5922490Z d7618c2df6cd: Pull complete 2025-08-14T21:28:37.1204423Z b7bdd9a6f789: Pull complete 2025-08-14T21:28:37.5879939Z 6738ba83282e: Pull complete 2025-08-14T21:28:38.5690511Z dfb0f2488639: Pull complete 2025-08-14T21:28:38.8717300Z dc833b0762f2: Pull complete 2025-08-14T21:28:39.1943572Z 8827df8ca2da: Pull complete 2025-08-14T21:29:29.5055369Z fac8f3bd0f85: Verifying Checksum 2025-08-14T21:29:29.5055888Z fac8f3bd0f85: Download complete 2025-08-14T21:33:22.3782394Z fac8f3bd0f85: Pull complete 2025-08-14T21:33:22.4052630Z d7cf7f140df3: Pull complete 2025-08-14T21:33:24.4710182Z 733eedc8da8d: Pull complete 2025-08-14T21:35:44.7672607Z 5b092eb06909: Pull complete 2025-08-14T21:35:44.7936632Z bc5961031092: Pull complete 2025-08-14T21:35:44.8201906Z 0531cc34c12a: Pull complete 2025-08-14T21:35:44.8688158Z 38c303d3b62e: Pull complete 2025-08-14T21:35:44.9201566Z e06d15594a2a: Pull complete 2025-08-14T21:35:44.9437738Z 0e55deb5cb38: Pull complete 2025-08-14T21:35:44.9917984Z 4a53d66dce07: Pull complete 2025-08-14T21:35:45.0399550Z 1519daa051b8: Pull complete 2025-08-14T21:35:45.0673174Z 381ed91d2119: Pull complete 2025-08-14T21:35:45.1187292Z c6b0a01a96dd: Pull complete 2025-08-14T21:35:45.1435417Z 62df6413daee: Pull complete 2025-08-14T21:35:45.1961254Z 7a18bc2a6881: Pull complete 2025-08-14T21:35:45.2214347Z 93359cd58a8c: Pull complete 2025-08-14T21:35:45.2730348Z c35ba0a1f353: Pull complete 2025-08-14T21:35:45.2987670Z dcf1e01c98d6: Pull complete 2025-08-14T21:35:54.7194259Z bad0564f61fd: Pull complete 2025-08-14T21:35:55.1396086Z 539ded905736: Pull complete 2025-08-14T21:35:55.5440002Z 28d482062637: Pull complete 2025-08-14T21:35:55.8570181Z 3245316ff51b: Pull complete 2025-08-14T21:35:56.1270533Z b53167d1a6df: Pull complete 2025-08-14T21:35:56.4286290Z 7f5277f69167: Pull complete 2025-08-14T21:35:57.4420712Z 23dff10cdaa5: Pull complete 2025-08-14T21:36:00.1381322Z 9fb73296da6a: Pull complete 2025-08-14T21:36:00.8691125Z Digest: sha256:4236794baba289041d240d08fd393bbd57497c3012e5e0ccd9fd98f61ebf35c6 2025-08-14T21:36:00.9475920Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:36:00.9924960Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:36:01.0011755Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:36:01.0012363Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:36:01.0021001Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:01.0021255Z env: 2025-08-14T21:36:01.0021427Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:01.0021638Z ##[endgroup] 2025-08-14T21:36:01.0101791Z Prepare all required actions 2025-08-14T21:36:01.0132206Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-08-14T21:36:01.0132508Z with: 2025-08-14T21:36:01.0133150Z github-token: *** 2025-08-14T21:36:01.0133350Z env: 2025-08-14T21:36:01.0133535Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:01.0133754Z ##[endgroup] 2025-08-14T21:36:01.0276350Z ##[group]Run set -eux 2025-08-14T21:36:01.0276593Z set -eux 2025-08-14T21:36:01.0276905Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-08-14T21:36:01.0282364Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:01.0282619Z env: 2025-08-14T21:36:01.0282810Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:01.0283297Z GITHUB_TOKEN: *** 2025-08-14T21:36:01.0283455Z ##[endgroup] 2025-08-14T21:36:01.0308442Z + python3 .github/scripts/get_workflow_job_id.py 16976338999 i-0aaf71856f9399359 2025-08-14T21:36:02.4293187Z Setting output job-id=48128301875 2025-08-14T21:36:02.4293860Z Setting output job-name=linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:36:02.4414201Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-08-14T21:36:02.4414657Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-08-14T21:36:02.4415292Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-08-14T21:36:02.4415776Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:36:02.4420322Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:02.4420558Z env: 2025-08-14T21:36:02.4420715Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:02.4420884Z JOB_ID: 48128301875 2025-08-14T21:36:02.4421275Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:36:02.4421687Z WORKFLOW_NAME: inductor-periodic 2025-08-14T21:36:02.4421935Z WORKFLOW_RUN_ID: 16976338999 2025-08-14T21:36:02.4422113Z MONITOR_LOG_INTERVAL: 5 2025-08-14T21:36:02.4422293Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-08-14T21:36:02.4422481Z ##[endgroup] 2025-08-14T21:36:03.0085777Z Defaulting to user installation because normal site-packages is not writeable 2025-08-14T21:36:03.2786698Z Collecting psutil==5.9.8 2025-08-14T21:36:03.2936305Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-08-14T21:36:03.3882368Z Collecting dataclasses_json==0.6.7 2025-08-14T21:36:03.3918519Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-08-14T21:36:03.4872981Z Collecting nvidia-ml-py==11.525.84 2025-08-14T21:36:03.4928907Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-08-14T21:36:03.6900631Z Collecting marshmallow<4.0.0,>=3.18.0 2025-08-14T21:36:03.6942541Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-08-14T21:36:03.8229043Z Collecting typing-inspect<1,>=0.4.0 2025-08-14T21:36:03.8259727Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-08-14T21:36:03.9558070Z Collecting packaging>=17.0 2025-08-14T21:36:03.9595144Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-08-14T21:36:04.0765411Z Collecting mypy-extensions>=0.3.0 2025-08-14T21:36:04.0799759Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-08-14T21:36:04.2156420Z Collecting typing-extensions>=3.7.4 2025-08-14T21:36:04.2191697Z Downloading typing_extensions-4.14.1-py3-none-any.whl (43 kB) 2025-08-14T21:36:04.5174000Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-08-14T21:36:05.1378934Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.14.1 typing-inspect-0.9.0 2025-08-14T21:36:05.3736884Z Prepare all required actions 2025-08-14T21:36:05.3737203Z Getting action download info 2025-08-14T21:36:05.4999401Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-08-14T21:36:06.2253924Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-08-14T21:36:08.7842958Z ##[group]Run ./.github/actions/download-build-artifacts 2025-08-14T21:36:08.7843222Z with: 2025-08-14T21:36:08.7843410Z name: linux-jammy-py3.9-gcc11-build 2025-08-14T21:36:08.7843631Z s3-bucket: gha-artifacts 2025-08-14T21:36:08.7843822Z env: 2025-08-14T21:36:08.7843981Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:08.7844157Z ##[endgroup] 2025-08-14T21:36:08.7871306Z ##[group]Run seemethere/download-artifact-s3@v4 2025-08-14T21:36:08.7871557Z with: 2025-08-14T21:36:08.7871748Z name: linux-jammy-py3.9-gcc11-build 2025-08-14T21:36:08.7871982Z s3-bucket: gha-artifacts 2025-08-14T21:36:08.7872231Z region: us-east-1 2025-08-14T21:36:08.7872408Z env: 2025-08-14T21:36:08.7872571Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:08.7872756Z ##[endgroup] 2025-08-14T21:36:09.4683642Z (node:48175) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-08-14T21:36:09.4689149Z 2025-08-14T21:36:09.4693677Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-08-14T21:36:09.4695678Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-08-14T21:36:09.4696075Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-08-14T21:36:10.9707380Z Found 1 objects with prefix pytorch/pytorch/16976338999/linux-jammy-py3.9-gcc11-build/ 2025-08-14T21:36:10.9708000Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-08-14T21:36:15.5790132Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-08-14T21:36:15.5794951Z Artifact download has finished successfully 2025-08-14T21:36:15.6003457Z ##[group]Run unzip -o artifacts.zip 2025-08-14T21:36:15.6003717Z unzip -o artifacts.zip 2025-08-14T21:36:15.6008594Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:15.6009129Z env: 2025-08-14T21:36:15.6009303Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:15.6009492Z ##[endgroup] 2025-08-14T21:36:15.6070778Z Archive: artifacts.zip 2025-08-14T21:36:15.6071095Z creating: dist/ 2025-08-14T21:36:16.7229378Z inflating: dist/torch-2.9.0a0+git1fc683c-cp39-cp39-linux_x86_64.whl 2025-08-14T21:36:16.7229768Z creating: dist/vision/ 2025-08-14T21:36:16.7309934Z inflating: dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:36:16.7310865Z creating: dist/audio/ 2025-08-14T21:36:16.7413836Z inflating: dist/audio/torchaudio-2.8.0a0+bdb88e1-cp39-cp39-linux_x86_64.whl 2025-08-14T21:36:16.7414380Z creating: dist/ao/ 2025-08-14T21:36:16.7453374Z inflating: dist/ao/torchao-0.7.0+git51c87b6e-py3-none-any.whl 2025-08-14T21:36:16.7566451Z inflating: dist/.ninja_log 2025-08-14T21:36:16.7566848Z creating: build/custom_test_artifacts/ 2025-08-14T21:36:16.7572666Z creating: build/custom_test_artifacts/custom-op-build/ 2025-08-14T21:36:16.7576472Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-08-14T21:36:16.7581321Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-08-14T21:36:16.7583608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-14T21:36:16.7584151Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/ 2025-08-14T21:36:16.7584633Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-14T21:36:16.7585092Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-14T21:36:16.7585978Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-14T21:36:16.7586521Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-14T21:36:16.7587061Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-14T21:36:16.7587516Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-14T21:36:16.7587992Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-14T21:36:16.7588458Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-14T21:36:16.7588985Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-14T21:36:16.7589518Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-14T21:36:16.7590030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-14T21:36:16.7590556Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-14T21:36:16.7591095Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-14T21:36:16.7591563Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-08-14T21:36:16.7591976Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-08-14T21:36:16.7592398Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-08-14T21:36:16.7592839Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-08-14T21:36:16.7593391Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-08-14T21:36:16.7593898Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-08-14T21:36:16.7594374Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-08-14T21:36:16.7594857Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-08-14T21:36:16.7595355Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-08-14T21:36:16.7595874Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-08-14T21:36:16.7596569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-08-14T21:36:16.7597063Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-08-14T21:36:16.7604190Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-08-14T21:36:16.7778007Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-08-14T21:36:16.7784130Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-08-14T21:36:16.7789070Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-08-14T21:36:16.7793659Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-08-14T21:36:16.7794257Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-08-14T21:36:16.7794786Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-08-14T21:36:16.7795288Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-08-14T21:36:16.7795811Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-08-14T21:36:16.7796810Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-08-14T21:36:16.7797402Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-08-14T21:36:16.7797960Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-08-14T21:36:16.7798502Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-08-14T21:36:16.7869812Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-08-14T21:36:16.7870380Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-14T21:36:16.7870906Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-08-14T21:36:16.7871324Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-08-14T21:36:16.7871749Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-08-14T21:36:16.7872156Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-08-14T21:36:16.7872590Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json 2025-08-14T21:36:16.7872988Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-08-14T21:36:16.7873346Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-08-14T21:36:16.7873709Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-08-14T21:36:16.8019142Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-08-14T21:36:16.8070983Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-08-14T21:36:16.8071380Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-08-14T21:36:16.8071718Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-08-14T21:36:16.8072126Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-08-14T21:36:16.8072590Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-14T21:36:16.8073021Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/ 2025-08-14T21:36:16.8073433Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-14T21:36:16.8073940Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-14T21:36:16.8074389Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-14T21:36:16.8075466Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-14T21:36:16.8076955Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-14T21:36:16.8077446Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-14T21:36:16.8077915Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-14T21:36:16.8078752Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-14T21:36:16.8085691Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-14T21:36:16.8087680Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-14T21:36:16.8088296Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-14T21:36:16.8091758Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-14T21:36:16.8092425Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-14T21:36:16.8094938Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-08-14T21:36:16.8095665Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-08-14T21:36:16.8100798Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-08-14T21:36:16.8102701Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-08-14T21:36:16.8103274Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-08-14T21:36:16.8103810Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-08-14T21:36:16.8104284Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-08-14T21:36:16.8104789Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-08-14T21:36:16.8105339Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-08-14T21:36:16.8105947Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-08-14T21:36:16.8106482Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-08-14T21:36:16.8106964Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-08-14T21:36:16.8109019Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-08-14T21:36:16.8165075Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-08-14T21:36:16.8165856Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-14T21:36:16.8166771Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-08-14T21:36:16.8167251Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-08-14T21:36:16.8167675Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-08-14T21:36:16.8168102Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-08-14T21:36:16.8168540Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json 2025-08-14T21:36:16.8168951Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-08-14T21:36:16.8169283Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-08-14T21:36:16.8169608Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-08-14T21:36:16.8205086Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-08-14T21:36:16.8205600Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-08-14T21:36:16.8206078Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-08-14T21:36:16.8206605Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-08-14T21:36:16.8211965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-14T21:36:16.8212587Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/ 2025-08-14T21:36:16.8213152Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-14T21:36:16.8213624Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-14T21:36:16.8214120Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-14T21:36:16.8214662Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-14T21:36:16.8215208Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-14T21:36:16.8215985Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-14T21:36:16.8216459Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-14T21:36:16.8216896Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-14T21:36:16.8217551Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-14T21:36:16.8218194Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-14T21:36:16.8218684Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-14T21:36:16.8219233Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-14T21:36:16.8219785Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-14T21:36:16.8220283Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-08-14T21:36:16.8220712Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-08-14T21:36:16.8221154Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-08-14T21:36:16.8221650Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-08-14T21:36:16.8222191Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-08-14T21:36:16.8222714Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-08-14T21:36:16.8223323Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-08-14T21:36:16.8223950Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-08-14T21:36:16.8224588Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-08-14T21:36:16.8225629Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-08-14T21:36:16.8226185Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-08-14T21:36:16.8226690Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-08-14T21:36:16.8227227Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-08-14T21:36:16.8337967Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-08-14T21:36:16.8338710Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-08-14T21:36:16.8339305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-08-14T21:36:16.8340241Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-08-14T21:36:16.8340836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-08-14T21:36:16.8341376Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-08-14T21:36:16.8341956Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-08-14T21:36:16.8342523Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-08-14T21:36:16.8343077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-08-14T21:36:16.8343720Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-08-14T21:36:16.8344279Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-08-14T21:36:16.8360408Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-08-14T21:36:16.8413729Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-08-14T21:36:16.8414390Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-14T21:36:16.8414932Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-08-14T21:36:16.8415465Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-08-14T21:36:16.8415911Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-08-14T21:36:16.8416355Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-08-14T21:36:16.8416825Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json 2025-08-14T21:36:16.8417258Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-08-14T21:36:16.8417625Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-08-14T21:36:16.8418011Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-08-14T21:36:16.8510000Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-08-14T21:36:16.8545159Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-08-14T21:36:16.8545710Z creating: build/lib/ 2025-08-14T21:36:16.8621796Z inflating: build/lib/libprotobuf-lite.a 2025-08-14T21:36:16.9034807Z inflating: build/lib/libprotobuf.a 2025-08-14T21:36:16.9511705Z inflating: build/lib/libprotoc.a 2025-08-14T21:36:16.9522649Z inflating: build/lib/libpthreadpool.a 2025-08-14T21:36:16.9528317Z inflating: build/lib/libcpuinfo.a 2025-08-14T21:36:16.9535237Z inflating: build/lib/libcpuinfo_internals.a 2025-08-14T21:36:16.9535536Z inflating: build/lib/libclog.a 2025-08-14T21:36:16.9553974Z inflating: build/lib/libpytorch_qnnpack.a 2025-08-14T21:36:16.9554316Z inflating: build/lib/libnnpack_reference_layers.a 2025-08-14T21:36:16.9731347Z inflating: build/lib/libmicrokernels-prod.a 2025-08-14T21:36:16.9745455Z inflating: build/lib/libnnpack.a 2025-08-14T21:36:17.0584521Z inflating: build/lib/libmicrokernels-all.a 2025-08-14T21:36:17.0651727Z inflating: build/lib/libgtest.a 2025-08-14T21:36:17.0667561Z inflating: build/lib/libgmock.a 2025-08-14T21:36:17.0667856Z inflating: build/lib/libgmock_main.a 2025-08-14T21:36:17.0668081Z inflating: build/lib/libgtest_main.a 2025-08-14T21:36:17.0752602Z inflating: build/lib/libXNNPACK.a 2025-08-14T21:36:17.0822340Z inflating: build/lib/libbenchmark.a 2025-08-14T21:36:17.0823004Z inflating: build/lib/libbenchmark_main.a 2025-08-14T21:36:17.0823291Z inflating: build/lib/libjitprofiling.a 2025-08-14T21:36:17.0884058Z inflating: build/lib/libasmjit.a 2025-08-14T21:36:17.0889682Z inflating: build/lib/libittnotify.a 2025-08-14T21:36:17.1964154Z inflating: build/lib/libfbgemm.a 2025-08-14T21:36:17.1993488Z inflating: build/lib/libtensorpipe_uv.a 2025-08-14T21:36:17.2503639Z inflating: build/lib/libtensorpipe.a 2025-08-14T21:36:17.2620211Z inflating: build/lib/libgloo.a 2025-08-14T21:36:17.2661051Z inflating: build/lib/libonnx_proto.a 2025-08-14T21:36:17.3331057Z inflating: build/lib/libonnx.a 2025-08-14T21:36:18.2669427Z inflating: build/lib/libdnnl.a 2025-08-14T21:36:18.2689515Z inflating: build/lib/libfmt.a 2025-08-14T21:36:18.2935538Z inflating: build/lib/libkineto.a 2025-08-14T21:36:18.3038164Z inflating: build/lib/libc10.so 2025-08-14T21:36:18.3038855Z inflating: build/lib/libtorch_global_deps.so 2025-08-14T21:36:21.1094036Z inflating: build/lib/libtorch_cpu.so 2025-08-14T21:36:21.1094589Z inflating: build/lib/libtorch.so 2025-08-14T21:36:21.1161948Z inflating: build/lib/libtorchbind_test.so 2025-08-14T21:36:21.1176622Z inflating: build/lib/libjitbackend_test.so 2025-08-14T21:36:21.1200287Z inflating: build/lib/libbackend_with_compiler.so 2025-08-14T21:36:21.1223418Z inflating: build/lib/libaoti_custom_ops.so 2025-08-14T21:36:21.1228639Z inflating: build/lib/libshm.so 2025-08-14T21:36:21.3091762Z inflating: build/lib/libtorch_python.so 2025-08-14T21:36:21.3125075Z inflating: build/lib/libnnapi_backend.so 2025-08-14T21:36:21.3128096Z creating: build/bin/ 2025-08-14T21:36:21.3131661Z creating: build/bin/CMakeFiles/ 2025-08-14T21:36:21.3132066Z inflating: build/bin/cmake_install.cmake 2025-08-14T21:36:21.3137023Z inflating: build/bin/CTestTestfile.cmake 2025-08-14T21:36:21.3561994Z inflating: build/bin/protoc-3.13.0.0 2025-08-14T21:36:21.3992971Z inflating: build/bin/protoc 2025-08-14T21:36:21.4051293Z inflating: build/bin/c10_AllocatorConfig_test 2025-08-14T21:36:21.4104317Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-08-14T21:36:21.4160792Z inflating: build/bin/c10_DeviceGuard_test 2025-08-14T21:36:21.4217936Z inflating: build/bin/c10_Device_test 2025-08-14T21:36:21.4267943Z inflating: build/bin/c10_StreamGuard_test 2025-08-14T21:36:21.4333959Z inflating: build/bin/c10_DispatchKeySet_test 2025-08-14T21:36:21.4386035Z inflating: build/bin/c10_SymInt_test 2025-08-14T21:36:21.4442067Z inflating: build/bin/c10_Scalar_test 2025-08-14T21:36:21.4501526Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-08-14T21:36:21.4563297Z inflating: build/bin/c10_InlineStreamGuard_test 2025-08-14T21:36:21.4622804Z inflating: build/bin/c10_SizesAndStrides_test 2025-08-14T21:36:21.4679185Z inflating: build/bin/c10_Bitset_test 2025-08-14T21:36:21.4750687Z inflating: build/bin/c10_cow_test 2025-08-14T21:36:21.4802825Z inflating: build/bin/c10_ArrayRef_test 2025-08-14T21:36:21.4856822Z inflating: build/bin/c10_ConstexprCrc_test 2025-08-14T21:36:21.4907997Z inflating: build/bin/c10_DeadlockDetection_test 2025-08-14T21:36:21.4969048Z inflating: build/bin/c10_Enumerate_test 2025-08-14T21:36:21.5026292Z inflating: build/bin/c10_Half_test 2025-08-14T21:36:21.5084489Z inflating: build/bin/c10_IntrusiveList_test 2025-08-14T21:36:21.5142122Z inflating: build/bin/c10_LeftRight_test 2025-08-14T21:36:21.5199211Z inflating: build/bin/c10_Metaprogramming_test 2025-08-14T21:36:21.5257553Z inflating: build/bin/c10_NetworkFlow_test 2025-08-14T21:36:21.5310909Z inflating: build/bin/c10_Synchronized_test 2025-08-14T21:36:21.5363296Z inflating: build/bin/c10_Semaphore_test 2025-08-14T21:36:21.5419774Z inflating: build/bin/c10_TypeIndex_test 2025-08-14T21:36:21.5474771Z inflating: build/bin/c10_ThreadLocal_test 2025-08-14T21:36:21.5535296Z inflating: build/bin/c10_TypeList_test 2025-08-14T21:36:21.5586292Z inflating: build/bin/c10_TypeTraits_test 2025-08-14T21:36:21.5636222Z inflating: build/bin/c10_accumulate_test 2025-08-14T21:36:21.5694116Z inflating: build/bin/c10_bfloat16_test 2025-08-14T21:36:21.5754122Z inflating: build/bin/c10_complex_test 2025-08-14T21:36:21.5813069Z inflating: build/bin/c10_complex_math_test 2025-08-14T21:36:21.5864470Z inflating: build/bin/c10_bit_cast_test 2025-08-14T21:36:21.5922517Z inflating: build/bin/c10_error_test 2025-08-14T21:36:21.5974738Z inflating: build/bin/c10_exception_test 2025-08-14T21:36:21.6026329Z inflating: build/bin/c10_flags_test 2025-08-14T21:36:21.6083305Z inflating: build/bin/c10_irange_test 2025-08-14T21:36:21.6133865Z inflating: build/bin/c10_generic_math_test 2025-08-14T21:36:21.6296425Z inflating: build/bin/c10_intrusive_ptr_test 2025-08-14T21:36:21.6350803Z inflating: build/bin/c10_lazy_test 2025-08-14T21:36:21.6414688Z inflating: build/bin/c10_logging_test 2025-08-14T21:36:21.6475929Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-08-14T21:36:21.6558547Z inflating: build/bin/c10_optional_test 2025-08-14T21:36:21.6613633Z inflating: build/bin/c10_registry_test 2025-08-14T21:36:21.6770781Z inflating: build/bin/c10_small_vector_test 2025-08-14T21:36:21.6830706Z inflating: build/bin/c10_string_util_test 2025-08-14T21:36:21.6884421Z inflating: build/bin/c10_ssize_test 2025-08-14T21:36:21.6937114Z inflating: build/bin/c10_string_view_test 2025-08-14T21:36:21.6990008Z inflating: build/bin/c10_tempfile_test 2025-08-14T21:36:21.7048498Z inflating: build/bin/c10_typeid_test 2025-08-14T21:36:21.7092736Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-08-14T21:36:21.7656811Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-08-14T21:36:21.8246826Z inflating: build/bin/vec_test_all_types_AVX512 2025-08-14T21:36:21.8832964Z inflating: build/bin/vec_test_all_types_AVX2 2025-08-14T21:36:21.8892195Z inflating: build/bin/static_runtime_bench 2025-08-14T21:36:21.9133730Z inflating: build/bin/static_runtime_test 2025-08-14T21:36:21.9211963Z inflating: build/bin/Dict_test 2025-08-14T21:36:21.9262473Z inflating: build/bin/Dimname_test 2025-08-14T21:36:21.9330916Z inflating: build/bin/MaybeOwned_test 2025-08-14T21:36:21.9386993Z inflating: build/bin/NamedTensor_test 2025-08-14T21:36:21.9447754Z inflating: build/bin/apply_utils_test 2025-08-14T21:36:21.9510853Z inflating: build/bin/atest 2025-08-14T21:36:21.9577230Z inflating: build/bin/basic 2025-08-14T21:36:21.9632567Z inflating: build/bin/broadcast_test 2025-08-14T21:36:21.9688295Z inflating: build/bin/cpu_allocator_test 2025-08-14T21:36:21.9746630Z inflating: build/bin/cpu_generator_test 2025-08-14T21:36:21.9804062Z inflating: build/bin/cpu_profiling_allocator_test 2025-08-14T21:36:21.9898974Z inflating: build/bin/cpu_rng_test 2025-08-14T21:36:21.9951711Z inflating: build/bin/dlconvertor_test 2025-08-14T21:36:22.0014126Z inflating: build/bin/extension_backend_test 2025-08-14T21:36:22.0072305Z inflating: build/bin/half_test 2025-08-14T21:36:22.0172194Z inflating: build/bin/ivalue_test 2025-08-14T21:36:22.0229605Z inflating: build/bin/lazy_tensor_test 2025-08-14T21:36:22.0287617Z inflating: build/bin/math_kernel_test 2025-08-14T21:36:22.0343555Z inflating: build/bin/memory_format_test 2025-08-14T21:36:22.0401964Z inflating: build/bin/memory_overlapping_test 2025-08-14T21:36:22.0462877Z inflating: build/bin/mobile_memory_cleanup 2025-08-14T21:36:22.0525859Z inflating: build/bin/native_test 2025-08-14T21:36:22.0576264Z inflating: build/bin/operator_name_test 2025-08-14T21:36:22.0631097Z inflating: build/bin/operators_test 2025-08-14T21:36:22.0688864Z inflating: build/bin/packedtensoraccessor_test 2025-08-14T21:36:22.0756394Z inflating: build/bin/pow_test 2025-08-14T21:36:22.0816223Z inflating: build/bin/quantized_test 2025-08-14T21:36:22.0869065Z inflating: build/bin/reduce_ops_test 2025-08-14T21:36:22.0922495Z inflating: build/bin/reportMemoryUsage_test 2025-08-14T21:36:22.0982714Z inflating: build/bin/scalar_tensor_test 2025-08-14T21:36:22.1041601Z inflating: build/bin/scalar_test 2025-08-14T21:36:22.1097377Z inflating: build/bin/StorageUtils_test 2025-08-14T21:36:22.1150409Z inflating: build/bin/stride_properties_test 2025-08-14T21:36:22.1230701Z inflating: build/bin/tensor_iterator_test 2025-08-14T21:36:22.1289974Z inflating: build/bin/test_parallel 2025-08-14T21:36:22.1345649Z inflating: build/bin/thread_init_test 2025-08-14T21:36:22.1403664Z inflating: build/bin/type_ptr_test 2025-08-14T21:36:22.1461776Z inflating: build/bin/type_test 2025-08-14T21:36:22.1517622Z inflating: build/bin/undefined_tensor_test 2025-08-14T21:36:22.1569220Z inflating: build/bin/verify_api_visibility 2025-08-14T21:36:22.1646292Z inflating: build/bin/legacy_vmap_test 2025-08-14T21:36:22.1696604Z inflating: build/bin/weakref_test 2025-08-14T21:36:22.1752089Z inflating: build/bin/wrapdim_test 2025-08-14T21:36:22.1807163Z inflating: build/bin/xla_tensor_test 2025-08-14T21:36:22.1866904Z inflating: build/bin/IListRef_test 2025-08-14T21:36:22.1975276Z inflating: build/bin/List_test 2025-08-14T21:36:22.2042216Z inflating: build/bin/KernelFunction_test 2025-08-14T21:36:22.2164784Z inflating: build/bin/kernel_function_legacy_test 2025-08-14T21:36:22.2260666Z inflating: build/bin/kernel_function_test 2025-08-14T21:36:22.2384152Z inflating: build/bin/kernel_lambda_legacy_test 2025-08-14T21:36:22.2487291Z inflating: build/bin/kernel_lambda_test 2025-08-14T21:36:22.2551285Z inflating: build/bin/kernel_stackbased_test 2025-08-14T21:36:22.2647251Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-08-14T21:36:22.2702954Z inflating: build/bin/CppSignature_test 2025-08-14T21:36:22.2760346Z inflating: build/bin/backend_fallback_test 2025-08-14T21:36:22.2810448Z inflating: build/bin/op_allowlist_test 2025-08-14T21:36:22.3113302Z inflating: build/bin/op_registration_test 2025-08-14T21:36:22.3184279Z inflating: build/bin/inline_container_test 2025-08-14T21:36:22.4259548Z inflating: build/bin/test_jit 2025-08-14T21:36:22.4568684Z inflating: build/bin/test_nativert 2025-08-14T21:36:22.4623914Z inflating: build/bin/BackoffTest 2025-08-14T21:36:22.4681754Z inflating: build/bin/FileStoreTest 2025-08-14T21:36:22.4744060Z inflating: build/bin/TCPStoreTest 2025-08-14T21:36:22.4797341Z inflating: build/bin/HashStoreTest 2025-08-14T21:36:22.4865459Z inflating: build/bin/ProcessGroupGlooTest 2025-08-14T21:36:22.4869836Z inflating: build/bin/example_allreduce 2025-08-14T21:36:22.4927885Z inflating: build/bin/test_dist_autograd 2025-08-14T21:36:22.4998290Z inflating: build/bin/test_cpp_rpc 2025-08-14T21:36:22.6101001Z inflating: build/bin/test_api 2025-08-14T21:36:22.6105251Z inflating: build/bin/parallel_benchmark 2025-08-14T21:36:22.6441086Z inflating: build/bin/test_lazy 2025-08-14T21:36:22.6446330Z inflating: build/bin/torch_shm_manager 2025-08-14T21:36:22.6452247Z creating: .additional_ci_files/ 2025-08-14T21:36:22.6516847Z inflating: .additional_ci_files/test-times.json 2025-08-14T21:36:22.6803447Z inflating: .additional_ci_files/test-class-times.json 2025-08-14T21:36:22.6879664Z ##[group]Run rm artifacts.zip 2025-08-14T21:36:22.6879909Z rm artifacts.zip 2025-08-14T21:36:22.6884783Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:22.6885036Z env: 2025-08-14T21:36:22.6885199Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:22.6885386Z ##[endgroup] 2025-08-14T21:36:22.8739109Z ##[group]Run df -H 2025-08-14T21:36:22.8739312Z df -H 2025-08-14T21:36:22.8744293Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:22.8744562Z env: 2025-08-14T21:36:22.8744753Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:22.8745136Z ##[endgroup] 2025-08-14T21:36:22.8788591Z Filesystem Size Used Avail Use% Mounted on 2025-08-14T21:36:22.8788928Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-08-14T21:36:22.8789185Z tmpfs 67G 0 67G 0% /dev/shm 2025-08-14T21:36:22.8789428Z tmpfs 27G 791k 27G 1% /run 2025-08-14T21:36:22.8789688Z /dev/nvme0n1p1 215G 69G 147G 32% / 2025-08-14T21:36:22.8789932Z tmpfs 67G 13k 67G 1% /tmp 2025-08-14T21:36:22.8790170Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-08-14T21:36:22.8815154Z Prepare all required actions 2025-08-14T21:36:22.8816085Z Getting action download info 2025-08-14T21:36:23.0371001Z ##[group]Run ./.github/actions/download-td-artifacts 2025-08-14T21:36:23.0371259Z with: 2025-08-14T21:36:23.0371412Z env: 2025-08-14T21:36:23.0371575Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:23.0371761Z ##[endgroup] 2025-08-14T21:36:23.0451081Z ##[group]Run seemethere/download-artifact-s3@v4 2025-08-14T21:36:23.0451331Z with: 2025-08-14T21:36:23.0451502Z name: td_results 2025-08-14T21:36:23.0451674Z s3-bucket: gha-artifacts 2025-08-14T21:36:23.0451850Z region: us-east-1 2025-08-14T21:36:23.0452005Z env: 2025-08-14T21:36:23.0452177Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:23.0452353Z ##[endgroup] 2025-08-14T21:36:23.3956833Z (node:48193) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-08-14T21:36:23.3957260Z 2025-08-14T21:36:23.3957813Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-08-14T21:36:23.3958207Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-08-14T21:36:23.3958612Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-08-14T21:36:23.4861392Z Found 0 objects with prefix pytorch/pytorch/16976338999/td_results/ 2025-08-14T21:36:23.4869466Z Artifact download has finished successfully 2025-08-14T21:36:23.5042094Z ##[group]Run mkdir -p .additional_ci_files 2025-08-14T21:36:23.5042355Z mkdir -p .additional_ci_files 2025-08-14T21:36:23.5042645Z mv td_results.json .additional_ci_files/td_results.json || true 2025-08-14T21:36:23.5047334Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:23.5047573Z env: 2025-08-14T21:36:23.5047730Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:23.5047903Z ##[endgroup] 2025-08-14T21:36:23.5101452Z mv: cannot stat 'td_results.json': No such file or directory 2025-08-14T21:36:23.5131228Z ##[group]Run .github/scripts/parse_ref.py 2025-08-14T21:36:23.5131507Z .github/scripts/parse_ref.py 2025-08-14T21:36:23.5136110Z shell: /usr/bin/bash -e {0} 2025-08-14T21:36:23.5136309Z env: 2025-08-14T21:36:23.5136473Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:23.5136660Z ##[endgroup] 2025-08-14T21:36:23.6028245Z Setting output branch=main 2025-08-14T21:36:23.6120849Z Prepare all required actions 2025-08-14T21:36:23.6121187Z Getting action download info 2025-08-14T21:36:23.7471188Z ##[group]Run ./.github/actions/filter-test-configs 2025-08-14T21:36:23.7471510Z with: 2025-08-14T21:36:23.7472007Z github-token: *** 2025-08-14T21:36:23.7474611Z test-matrix: {"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]} 2025-08-14T21:36:23.7477945Z job-name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:36:23.7478438Z env: 2025-08-14T21:36:23.7478603Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:23.7478809Z ##[endgroup] 2025-08-14T21:36:24.7800122Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-14T21:36:24.7800363Z with: 2025-08-14T21:36:24.7800571Z shell: bash 2025-08-14T21:36:24.7800744Z timeout_minutes: 10 2025-08-14T21:36:24.7800915Z max_attempts: 5 2025-08-14T21:36:24.7801118Z retry_wait_seconds: 30 2025-08-14T21:36:24.7801652Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-08-14T21:36:24.7802202Z polling_interval_seconds: 1 2025-08-14T21:36:24.7802400Z warning_on_retry: true 2025-08-14T21:36:24.7802588Z continue_on_error: false 2025-08-14T21:36:24.7802775Z env: 2025-08-14T21:36:24.7802925Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:24.7803330Z GITHUB_TOKEN: *** 2025-08-14T21:36:24.7803509Z ##[endgroup] 2025-08-14T21:36:25.0026288Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-08-14T21:36:25.1882135Z Defaulting to user installation because normal site-packages is not writeable 2025-08-14T21:36:25.2855800Z Collecting requests==2.27.1 2025-08-14T21:36:25.3013540Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-08-14T21:36:25.4672881Z Collecting pyyaml==6.0.2 2025-08-14T21:36:25.4711508Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-08-14T21:36:25.5396264Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-08-14T21:36:25.6157426Z Collecting certifi>=2017.4.17 2025-08-14T21:36:25.6194163Z Downloading certifi-2025.8.3-py3-none-any.whl (161 kB) 2025-08-14T21:36:25.6717720Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-08-14T21:36:25.9587318Z Collecting charset-normalizer~=2.0.0 2025-08-14T21:36:25.9620268Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-08-14T21:36:26.0739440Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-08-14T21:36:26.4353319Z Successfully installed certifi-2025.8.3 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-08-14T21:36:26.8456087Z Command completed after 1 attempt(s). 2025-08-14T21:36:26.8503751Z ##[group]Run set -x 2025-08-14T21:36:26.8503972Z set -x 2025-08-14T21:36:26.8504144Z  2025-08-14T21:36:26.8504413Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-08-14T21:36:26.8504755Z # in runner workspace 2025-08-14T21:36:26.8505036Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-08-14T21:36:26.8510376Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:26.8510632Z env: 2025-08-14T21:36:26.8510799Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:26.8510989Z ##[endgroup] 2025-08-14T21:36:26.8535014Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-08-14T21:36:26.8678091Z Setting output branch=main 2025-08-14T21:36:26.8724633Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-08-14T21:36:26.8724931Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-08-14T21:36:26.8725186Z echo "Job name: ${JOB_NAME}" 2025-08-14T21:36:26.8725389Z  2025-08-14T21:36:26.8725650Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-08-14T21:36:26.8726082Z # in runner workspace 2025-08-14T21:36:26.8726379Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-08-14T21:36:26.8726694Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-08-14T21:36:26.8726908Z  --job-name "${JOB_NAME}" \ 2025-08-14T21:36:26.8729008Z  --test-matrix "{"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]}" \ 2025-08-14T21:36:26.8731347Z  --selected-test-configs "" \ 2025-08-14T21:36:26.8731581Z  --pr-number "${PR_NUMBER}" \ 2025-08-14T21:36:26.8731806Z  --tag "${TAG}" \ 2025-08-14T21:36:26.8732023Z  --event-name "${EVENT_NAME}" \ 2025-08-14T21:36:26.8732258Z  --schedule "${SCHEDULE}" \ 2025-08-14T21:36:26.8732472Z  --branch "${HEAD_BRANCH}" 2025-08-14T21:36:26.8737182Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:26.8737442Z env: 2025-08-14T21:36:26.8737602Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:26.8738112Z GITHUB_TOKEN: *** 2025-08-14T21:36:26.8738554Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:36:26.8738981Z PR_NUMBER: 2025-08-14T21:36:26.8739141Z TAG: 2025-08-14T21:36:26.8739303Z EVENT_NAME: schedule 2025-08-14T21:36:26.8739497Z SCHEDULE: 45 0,4,8,12,16,20 * * 1-5 2025-08-14T21:36:26.8739693Z HEAD_BRANCH: main 2025-08-14T21:36:26.8739868Z ##[endgroup] 2025-08-14T21:36:26.8767858Z Workflow: inductor-periodic 2025-08-14T21:36:26.8770105Z Job name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:36:27.0346240Z Setting output keep-going=True 2025-08-14T21:36:27.0346598Z Setting output ci-verbose-test-logs=False 2025-08-14T21:36:27.0346896Z Setting output ci-test-showlocals=False 2025-08-14T21:36:27.0347129Z Setting output ci-no-test-timeout=False 2025-08-14T21:36:27.0347362Z Setting output ci-no-td=False 2025-08-14T21:36:27.0347586Z Setting output ci-td-distributed=False 2025-08-14T21:36:27.0347821Z Setting output is-unstable=False 2025-08-14T21:36:27.0348024Z Setting output reenabled-issues= 2025-08-14T21:36:27.0350298Z Setting output test-matrix={"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]} 2025-08-14T21:36:27.0352840Z Setting output is-test-matrix-empty=False 2025-08-14T21:36:27.0466423Z ##[group]Run echo "Filtered matrix:" 2025-08-14T21:36:27.0466677Z echo "Filtered matrix:" 2025-08-14T21:36:27.0468837Z echo "{"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]}" 2025-08-14T21:36:27.0471251Z  2025-08-14T21:36:27.0471406Z echo 2025-08-14T21:36:27.0471610Z echo "Is the current job unstable? False" 2025-08-14T21:36:27.0471840Z  2025-08-14T21:36:27.0471993Z echo 2025-08-14T21:36:27.0472181Z echo "Is keep-going label set? True" 2025-08-14T21:36:27.0472396Z  2025-08-14T21:36:27.0472549Z echo 2025-08-14T21:36:27.0472725Z echo "Reenabled issues? " 2025-08-14T21:36:27.0477991Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:27.0478271Z env: 2025-08-14T21:36:27.0478450Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:27.0478637Z ##[endgroup] 2025-08-14T21:36:27.0500551Z Filtered matrix: 2025-08-14T21:36:27.0503223Z {include: [{config: cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_avx2_huggingface, shard: 1, num_shards: 1, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_torchbench, shard: 1, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_torchbench, shard: 2, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_timm, shard: 1, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_timm, shard: 2, num_shards: 2, runner: linux.10xlarge.avx2}]} 2025-08-14T21:36:27.0505626Z 2025-08-14T21:36:27.0505720Z Is the current job unstable? False 2025-08-14T21:36:27.0505875Z 2025-08-14T21:36:27.0505969Z Is keep-going label set? True 2025-08-14T21:36:27.0506105Z 2025-08-14T21:36:27.0506186Z Reenabled issues? 2025-08-14T21:36:27.0573965Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-08-14T21:36:27.0574483Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-08-14T21:36:27.0579182Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:27.0579441Z env: 2025-08-14T21:36:27.0579608Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:27.0579804Z JOB_TIMEOUT: 240 2025-08-14T21:36:27.0579976Z ##[endgroup] 2025-08-14T21:36:27.0620902Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:36:27.0621239Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:36:27.0621520Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:36:27.0625621Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:36:27.0625861Z env: 2025-08-14T21:36:27.0626014Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:27.0626189Z ##[endgroup] 2025-08-14T21:36:27.0709232Z ##[group]Run set -x 2025-08-14T21:36:27.0709509Z set -x 2025-08-14T21:36:27.0709672Z  2025-08-14T21:36:27.0709882Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-08-14T21:36:27.0710163Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-08-14T21:36:27.0710435Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-08-14T21:36:27.0710688Z  TEST_COMMAND=.ci/onnx/test.sh 2025-08-14T21:36:27.0710904Z else 2025-08-14T21:36:27.0711094Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-08-14T21:36:27.0711304Z fi 2025-08-14T21:36:27.0711464Z  2025-08-14T21:36:27.0711660Z # Leaving 1GB for the runner and other things 2025-08-14T21:36:27.0712042Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-08-14T21:36:27.0712628Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-08-14T21:36:27.0713083Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-08-14T21:36:27.0713431Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-08-14T21:36:27.0713710Z  2025-08-14T21:36:27.0713911Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-08-14T21:36:27.0714142Z  SHM_OPTS= 2025-08-14T21:36:27.0714327Z  JENKINS_USER= 2025-08-14T21:36:27.0714575Z  # ensure that docker container cleanly exits in 12 hours 2025-08-14T21:36:27.0714885Z  # if for some reason cleanup action doesn't stop container 2025-08-14T21:36:27.0715154Z  # when job is cancelled 2025-08-14T21:36:27.0715376Z  DOCKER_SHELL_CMD="sleep 12h" 2025-08-14T21:36:27.0715587Z else 2025-08-14T21:36:27.0715770Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-08-14T21:36:27.0716118Z  JENKINS_USER="--user jenkins" 2025-08-14T21:36:27.0716414Z  DOCKER_SHELL_CMD= 2025-08-14T21:36:27.0716601Z fi 2025-08-14T21:36:27.0716759Z  2025-08-14T21:36:27.0717002Z # detached container should get cleaned up by teardown_ec2_linux 2025-08-14T21:36:27.0717360Z # TODO: Stop building test binaries as part of the build phase 2025-08-14T21:36:27.0717753Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-08-14T21:36:27.0718100Z # shellcheck disable=SC2086,SC2090 2025-08-14T21:36:27.0718335Z container_name=$(docker run \ 2025-08-14T21:36:27.0718579Z  ${GPU_FLAG:-} \ 2025-08-14T21:36:27.0718794Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-08-14T21:36:27.0719024Z  -e BUILD_ENVIRONMENT \ 2025-08-14T21:36:27.0719238Z  -e PR_NUMBER \ 2025-08-14T21:36:27.0719433Z  -e GITHUB_ACTIONS \ 2025-08-14T21:36:27.0719631Z  -e GITHUB_REPOSITORY \ 2025-08-14T21:36:27.0719842Z  -e GITHUB_WORKFLOW \ 2025-08-14T21:36:27.0720046Z  -e GITHUB_JOB \ 2025-08-14T21:36:27.0720230Z  -e GITHUB_RUN_ID \ 2025-08-14T21:36:27.0720544Z  -e GITHUB_RUN_NUMBER \ 2025-08-14T21:36:27.0720749Z  -e GITHUB_RUN_ATTEMPT \ 2025-08-14T21:36:27.0720955Z  -e JOB_ID \ 2025-08-14T21:36:27.0721129Z  -e JOB_NAME \ 2025-08-14T21:36:27.0721313Z  -e BASE_SHA \ 2025-08-14T21:36:27.0721497Z  -e BRANCH \ 2025-08-14T21:36:27.0721667Z  -e SHA1 \ 2025-08-14T21:36:27.0721851Z  -e AWS_DEFAULT_REGION \ 2025-08-14T21:36:27.0722058Z  -e IN_WHEEL_TEST \ 2025-08-14T21:36:27.0722245Z  -e SHARD_NUMBER \ 2025-08-14T21:36:27.0722435Z  -e TEST_CONFIG \ 2025-08-14T21:36:27.0722625Z  -e NUM_TEST_SHARDS \ 2025-08-14T21:36:27.0722821Z  -e REENABLED_ISSUES \ 2025-08-14T21:36:27.0723029Z  -e CONTINUE_THROUGH_ERROR \ 2025-08-14T21:36:27.0723327Z  -e VERBOSE_TEST_LOGS \ 2025-08-14T21:36:27.0723530Z  -e TEST_SHOWLOCALS \ 2025-08-14T21:36:27.0723723Z  -e NO_TEST_TIMEOUT \ 2025-08-14T21:36:27.0723918Z  -e NO_TD \ 2025-08-14T21:36:27.0724097Z  -e TD_DISTRIBUTED \ 2025-08-14T21:36:27.0724281Z  -e PR_LABELS \ 2025-08-14T21:36:27.0724492Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-08-14T21:36:27.0724725Z  -e SCCACHE_BUCKET \ 2025-08-14T21:36:27.0724916Z  -e SCCACHE_REGION \ 2025-08-14T21:36:27.0725098Z  -e XLA_CUDA \ 2025-08-14T21:36:27.0725294Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-08-14T21:36:27.0725532Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-08-14T21:36:27.0725762Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-08-14T21:36:27.0726000Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-08-14T21:36:27.0726220Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-08-14T21:36:27.0726432Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-08-14T21:36:27.0726641Z  -e DASHBOARD_TAG \ 2025-08-14T21:36:27.0726836Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-08-14T21:36:27.0727070Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-08-14T21:36:27.0727336Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-08-14T21:36:27.0727599Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-08-14T21:36:27.0727852Z  --security-opt seccomp=unconfined \ 2025-08-14T21:36:27.0728068Z  --cap-add=SYS_PTRACE \ 2025-08-14T21:36:27.0728264Z  --ipc=host \ 2025-08-14T21:36:27.0728441Z  ${SHM_OPTS} \ 2025-08-14T21:36:27.0728602Z  --tty \ 2025-08-14T21:36:27.0728762Z  --detach \ 2025-08-14T21:36:27.0728947Z  --name="${container_name}" \ 2025-08-14T21:36:27.0729143Z  ${JENKINS_USER} \ 2025-08-14T21:36:27.0729386Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-08-14T21:36:27.0729634Z  -w /var/lib/jenkins/workspace \ 2025-08-14T21:36:27.0729833Z  "${DOCKER_IMAGE}" \ 2025-08-14T21:36:27.0730008Z  ${DOCKER_SHELL_CMD} 2025-08-14T21:36:27.0730183Z ) 2025-08-14T21:36:27.0730379Z # Propagate download.pytorch.org IP to container 2025-08-14T21:36:27.0730773Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2025-08-14T21:36:27.0731196Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-08-14T21:36:27.0731448Z  2025-08-14T21:36:27.0731628Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-08-14T21:36:27.0731974Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-08-14T21:36:27.0732284Z fi 2025-08-14T21:36:27.0732432Z  2025-08-14T21:36:27.0732729Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-08-14T21:36:27.0737157Z shell: /usr/bin/bash -e {0} 2025-08-14T21:36:27.0737345Z env: 2025-08-14T21:36:27.0737568Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:36:27.0737779Z BUILD_ENVIRONMENT: linux-jammy-py3.9-gcc11-build 2025-08-14T21:36:27.0738004Z PR_NUMBER: 2025-08-14T21:36:27.0738173Z GITHUB_REPOSITORY: pytorch/pytorch 2025-08-14T21:36:27.0738376Z GITHUB_WORKFLOW: inductor-periodic 2025-08-14T21:36:27.0738567Z GITHUB_JOB: test 2025-08-14T21:36:27.0738732Z GITHUB_RUN_ID: 16976338999 2025-08-14T21:36:27.0738905Z GITHUB_RUN_NUMBER: 66307 2025-08-14T21:36:27.0739079Z GITHUB_RUN_ATTEMPT: 1 2025-08-14T21:36:27.0739247Z JOB_ID: 48128301875 2025-08-14T21:36:27.0739630Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:36:27.0740031Z BRANCH: main 2025-08-14T21:36:27.0740213Z SHA1: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:27.0740526Z BASE_SHA: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:27.0740759Z TEST_CONFIG: cpu_inductor_huggingface 2025-08-14T21:36:27.0740968Z SHARD_NUMBER: 1 2025-08-14T21:36:27.0741131Z NUM_TEST_SHARDS: 1 2025-08-14T21:36:27.0741286Z REENABLED_ISSUES: 2025-08-14T21:36:27.0741461Z CONTINUE_THROUGH_ERROR: True 2025-08-14T21:36:27.0741651Z VERBOSE_TEST_LOGS: False 2025-08-14T21:36:27.0741826Z TEST_SHOWLOCALS: False 2025-08-14T21:36:27.0742006Z NO_TEST_TIMEOUT: False 2025-08-14T21:36:27.0742176Z NO_TD: False 2025-08-14T21:36:27.0742337Z TD_DISTRIBUTED: False 2025-08-14T21:36:27.0742543Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-08-14T21:36:27.0742779Z SCCACHE_REGION: us-east-1 2025-08-14T21:36:27.0742957Z SHM_SIZE: 1g 2025-08-14T21:36:27.0743481Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:36:27.0744042Z XLA_CUDA: 2025-08-14T21:36:27.0744304Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-08-14T21:36:27.0744620Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-08-14T21:36:27.0744856Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-08-14T21:36:27.0745064Z DASHBOARD_TAG: 2025-08-14T21:36:27.0745408Z HUGGING_FACE_HUB_TOKEN: *** 2025-08-14T21:36:27.0745696Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-08-14T21:36:27.0746019Z ARTIFACTS_FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875 2025-08-14T21:36:27.0746344Z ##[endgroup] 2025-08-14T21:36:27.0768071Z + [[ cpu_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2025-08-14T21:36:27.0768387Z + [[ linux-jammy-py3.9-gcc11-build == *onnx* ]] 2025-08-14T21:36:27.0768646Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-08-14T21:36:27.0772137Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-08-14T21:36:27.0790220Z + TOTAL_AVAILABLE_MEMORY_IN_GB='122.780 ' 2025-08-14T21:36:27.0790491Z + TOTAL_MEMORY_WITH_SWAP=125 2025-08-14T21:36:27.0790774Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]] 2025-08-14T21:36:27.0791033Z + SHM_OPTS=--shm-size=1g 2025-08-14T21:36:27.0791239Z + JENKINS_USER='--user jenkins' 2025-08-14T21:36:27.0791441Z + DOCKER_SHELL_CMD= 2025-08-14T21:36:27.0799070Z +++ nproc --ignore=2 2025-08-14T21:36:27.0829454Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=122g --memory-swap=125g --env-file=/tmp/github_env_16976338999 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:36:37.7072473Z + container_name=ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T21:36:37.7073343Z + grep download.pytorch.org /etc/hosts 2025-08-14T21:36:37.7074481Z + docker exec -i ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb sudo bash -c '/bin/cat >> /etc/hosts' 2025-08-14T21:36:37.8604435Z + echo DOCKER_CONTAINER_ID=ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T21:36:37.8605292Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]] 2025-08-14T21:36:37.8613401Z ++ echo dist/torch-2.9.0a0+git1fc683c-cp39-cp39-linux_x86_64.whl 2025-08-14T21:36:37.8614343Z + docker exec -t ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb sh -c 'python3 -m pip install dist/torch-2.9.0a0+git1fc683c-cp39-cp39-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-08-14T21:36:38.2136534Z Processing ./dist/torch-2.9.0a0+git1fc683c-cp39-cp39-linux_x86_64.whl (from torch==2.9.0a0+git1fc683c) 2025-08-14T21:36:38.4327816Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (3.18.0) 2025-08-14T21:36:38.4328960Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (4.14.1) 2025-08-14T21:36:38.4329743Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (1.13.3) 2025-08-14T21:36:38.4330491Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (2.8.8) 2025-08-14T21:36:38.4331194Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (3.1.6) 2025-08-14T21:36:38.4336435Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (2025.3.0) 2025-08-14T21:36:38.4344409Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (3.3.0) 2025-08-14T21:36:38.4646271Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from opt-einsum>=3.3->torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (1.22.4) 2025-08-14T21:36:38.4665699Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from sympy>=1.13.3->torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (1.3.0) 2025-08-14T21:36:38.4709113Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from jinja2->torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (3.0.2) 2025-08-14T21:36:39.2403825Z Installing collected packages: torch 2025-08-14T21:36:46.6140988Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-08-14T21:36:46.6144498Z dall-e 0.1 requires torchvision, which is not installed. 2025-08-14T21:36:46.6144818Z effdet 0.4.1 requires torchvision, which is not installed. 2025-08-14T21:36:46.6145205Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed. 2025-08-14T21:36:46.6145879Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed. 2025-08-14T21:36:46.6151422Z timm 1.0.14 requires torchvision, which is not installed. 2025-08-14T21:36:46.6152666Z Successfully installed torch-2.9.0a0+git1fc683c 2025-08-14T21:36:46.7156214Z + export TERM=vt100 2025-08-14T21:36:46.7156496Z + TERM=vt100 2025-08-14T21:36:46.7156687Z ++ dirname .ci/pytorch/test.sh 2025-08-14T21:36:46.7166838Z + source .ci/pytorch/common.sh 2025-08-14T21:36:46.7167621Z +++ dirname .ci/pytorch/common.sh 2025-08-14T21:36:46.7175321Z ++ source .ci/pytorch/common_utils.sh 2025-08-14T21:36:46.7175607Z +++ declare -f -t trap_add 2025-08-14T21:36:46.7181419Z ++ set -ex -o pipefail 2025-08-14T21:36:46.7181696Z ++ [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-08-14T21:36:46.7181941Z ++ BUILD_TEST_LIBTORCH=0 2025-08-14T21:36:46.7182143Z ++ dirname .ci/pytorch/test.sh 2025-08-14T21:36:46.7188940Z + source .ci/pytorch/common-build.sh 2025-08-14T21:36:46.7189708Z ++ [[ linux-jammy-py3.9-gcc11-build != *win-* ]] 2025-08-14T21:36:46.7192785Z ++++ dirname .ci/pytorch/common-build.sh 2025-08-14T21:36:46.7204892Z +++ cd .ci/pytorch 2025-08-14T21:36:46.7205139Z +++ pwd -P 2025-08-14T21:36:46.7205366Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-08-14T21:36:46.7205700Z ++ [[ linux-jammy-py3.9-gcc11-build == *-pch* ]] 2025-08-14T21:36:46.7205936Z ++ which sccache 2025-08-14T21:36:46.7227964Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-08-14T21:36:46.7232156Z ++ sccache --stop-server 2025-08-14T21:36:46.7260021Z ++ true 2025-08-14T21:36:46.7265112Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-08-14T21:36:46.7271357Z ++ trap_add sccache_epilogue EXIT 2025-08-14T21:36:46.7273560Z ++ trap_add_cmd=sccache_epilogue 2025-08-14T21:36:46.7273826Z ++ shift 2025-08-14T21:36:46.7274012Z ++ for trap_add_name in "$@" 2025-08-14T21:36:46.7274224Z ++++ trap -p EXIT 2025-08-14T21:36:46.7274406Z +++ eval 'extract_trap_cmd ' 2025-08-14T21:36:46.7274629Z ++++ extract_trap_cmd 2025-08-14T21:36:46.7274815Z ++++ printf '%s\n' '' 2025-08-14T21:36:46.7275000Z +++ printf '%s\n' sccache_epilogue 2025-08-14T21:36:46.7275394Z ++ trap -- ' 2025-08-14T21:36:46.7275793Z sccache_epilogue' EXIT 2025-08-14T21:36:46.7276198Z ++ [[ -n 1 ]] 2025-08-14T21:36:46.7276620Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-08-14T21:36:46.7277167Z Skipping sccache server initialization, setting environment variables 2025-08-14T21:36:46.7277452Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-08-14T21:36:46.7277659Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-08-14T21:36:46.7277901Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-14T21:36:46.7278202Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-14T21:36:46.7278522Z ++ export RUST_LOG=sccache::server=error 2025-08-14T21:36:46.7278745Z ++ RUST_LOG=sccache::server=error 2025-08-14T21:36:46.7278958Z ++ sccache --zero-stats 2025-08-14T21:36:46.8642308Z Statistics zeroed. 2025-08-14T21:36:46.8647941Z ++ which ccache 2025-08-14T21:36:46.8680863Z + [[ linux-jammy-py3.9-gcc11-build != *rocm* ]] 2025-08-14T21:36:46.8681602Z + [[ linux-jammy-py3.9-gcc11-build != *s390x* ]] 2025-08-14T21:36:46.8681958Z + [[ -d /var/lib/jenkins/workspace ]] 2025-08-14T21:36:46.8682189Z ++ stat -c %u /var/lib/jenkins/workspace 2025-08-14T21:36:46.8694590Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-08-14T21:36:46.8694878Z + trap_add cleanup_workspace EXIT 2025-08-14T21:36:46.8695111Z + trap_add_cmd=cleanup_workspace 2025-08-14T21:36:46.8695308Z + shift 2025-08-14T21:36:46.8695497Z + for trap_add_name in "$@" 2025-08-14T21:36:46.8704752Z +++ trap -p EXIT 2025-08-14T21:36:46.8705502Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-08-14T21:36:46.8705800Z sccache_epilogue'\'' EXIT' 2025-08-14T21:36:46.8706199Z +++ extract_trap_cmd trap -- ' 2025-08-14T21:36:46.8706411Z sccache_epilogue' EXIT 2025-08-14T21:36:46.8706604Z +++ printf '%s\n' ' 2025-08-14T21:36:46.8706793Z sccache_epilogue' 2025-08-14T21:36:46.8707007Z ++ printf '%s\n' cleanup_workspace 2025-08-14T21:36:46.8707219Z + trap -- ' 2025-08-14T21:36:46.8707391Z sccache_epilogue 2025-08-14T21:36:46.8707571Z cleanup_workspace' EXIT 2025-08-14T21:36:46.8708087Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-08-14T21:36:47.3106005Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-08-14T21:36:47.3133946Z + echo 'Environment variables:' 2025-08-14T21:36:47.3137019Z Environment variables: 2025-08-14T21:36:47.3137543Z + env 2025-08-14T21:36:47.3144824Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T21:36:47.3145249Z CONTINUE_THROUGH_ERROR=True 2025-08-14T21:36:47.3145575Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build 2025-08-14T21:36:47.3145934Z HOSTNAME=ec43c4531511 2025-08-14T21:36:47.3146399Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3147012Z GITHUB_ACTION=__run_2 2025-08-14T21:36:47.3147482Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-08-14T21:36:47.3147824Z GITHUB_RUN_NUMBER=66307 2025-08-14T21:36:47.3148107Z TEST_CONFIG=cpu_inductor_huggingface 2025-08-14T21:36:47.3148714Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-08-14T21:36:47.3149035Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-08-14T21:36:47.3149257Z SCCACHE_IDLE_TIMEOUT=0 2025-08-14T21:36:47.3149662Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-08-14T21:36:47.3149887Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-08-14T21:36:47.3150094Z GITHUB_REF_TYPE=branch 2025-08-14T21:36:47.3150280Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-08-14T21:36:47.3150503Z BASE_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:47.3150717Z XLA_CUDA= 2025-08-14T21:36:47.3150927Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-08-14T21:36:47.3151243Z HUGGING_FACE_HUB_TOKEN=*** 2025-08-14T21:36:47.3157730Z *** 2025-08-14T21:36:47.3157939Z GITHUB_REPOSITORY_ID=65600975 2025-08-14T21:36:47.3158163Z GITHUB_ACTIONS=true 2025-08-14T21:36:47.3158386Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-14T21:36:47.3158711Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:47.3158974Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:47.3159383Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/heads/main 2025-08-14T21:36:47.3159721Z UCC_HOME=/usr 2025-08-14T21:36:47.3159893Z VERBOSE_TEST_LOGS=False 2025-08-14T21:36:47.3160087Z GITHUB_REF=refs/heads/main 2025-08-14T21:36:47.3160271Z SHARD_NUMBER=1 2025-08-14T21:36:47.3160445Z GITHUB_REF_PROTECTED=true 2025-08-14T21:36:47.3160638Z HOME=/var/lib/jenkins 2025-08-14T21:36:47.3160847Z GITHUB_API_URL=https://api.github.com 2025-08-14T21:36:47.3161092Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-08-14T21:36:47.3161299Z UCX_COMMIT= 2025-08-14T21:36:47.3161457Z USE_SYSTEM_NCCL=1 2025-08-14T21:36:47.3161623Z NUM_TEST_SHARDS=1 2025-08-14T21:36:47.3161784Z UCX_HOME=/usr 2025-08-14T21:36:47.3162176Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3162805Z JOB_NAME=linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:36:47.3163428Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3164013Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-08-14T21:36:47.3164333Z GITHUB_EVENT_NAME=schedule 2025-08-14T21:36:47.3164514Z DASHBOARD_TAG= 2025-08-14T21:36:47.3164685Z GITHUB_RUN_ID=16976338999 2025-08-14T21:36:47.3164872Z INSTALLED_OPENBLAS= 2025-08-14T21:36:47.3165256Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3165693Z GITHUB_ACTOR=pytorchmergebot 2025-08-14T21:36:47.3165889Z PR_NUMBER= 2025-08-14T21:36:47.3166047Z DESIRED_CUDA= 2025-08-14T21:36:47.3166205Z GITHUB_RUN_ATTEMPT=1 2025-08-14T21:36:47.3166398Z ANACONDA_PYTHON_VERSION=3.9 2025-08-14T21:36:47.3166632Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-08-14T21:36:47.3166860Z TERM=vt100 2025-08-14T21:36:47.3167224Z INSTALLED_VISION=yes 2025-08-14T21:36:47.3167396Z BRANCH=main 2025-08-14T21:36:47.3167554Z SCCACHE_REGION=us-east-1 2025-08-14T21:36:47.3167752Z OPENSSL_ROOT_DIR=/opt/openssl 2025-08-14T21:36:47.3167953Z CUDA_PATH=/usr/local/cuda 2025-08-14T21:36:47.3168280Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-08-14T21:36:47.3168651Z GITHUB_SERVER_URL=https://github.com 2025-08-14T21:36:47.3168852Z UCC_COMMIT= 2025-08-14T21:36:47.3168991Z REENABLED_ISSUES= 2025-08-14T21:36:47.3169158Z DOCS=yes 2025-08-14T21:36:47.3169297Z SHLVL=1 2025-08-14T21:36:47.3169426Z MAX_JOBS=30 2025-08-14T21:36:47.3169573Z GITHUB_ACTOR_ID=97764156 2025-08-14T21:36:47.3169792Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:47.3170014Z GITHUB_REF_NAME=main 2025-08-14T21:36:47.3170309Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-08-14T21:36:47.3170573Z GITHUB_JOB=test 2025-08-14T21:36:47.3170728Z NO_TEST_TIMEOUT=False 2025-08-14T21:36:47.3170889Z TD_DISTRIBUTED=False 2025-08-14T21:36:47.3171060Z GITHUB_REPOSITORY=pytorch/pytorch 2025-08-14T21:36:47.3171254Z GITHUB_RETENTION_DAYS=90 2025-08-14T21:36:47.3171416Z OPENSSL_DIR=/opt/openssl 2025-08-14T21:36:47.3171588Z GITHUB_ACTION_REPOSITORY= 2025-08-14T21:36:47.3172043Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:36:47.3172486Z GITHUB_BASE_REF= 2025-08-14T21:36:47.3172666Z INSTALLED_ACL= 2025-08-14T21:36:47.3172934Z ARTIFACTS_FILE_SUFFIX=test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875 2025-08-14T21:36:47.3173232Z CI=true 2025-08-14T21:36:47.3173396Z GITHUB_REPOSITORY_OWNER=pytorch 2025-08-14T21:36:47.3173636Z RUST_LOG=sccache::server=error 2025-08-14T21:36:47.3173815Z JOB_ID=48128301875 2025-08-14T21:36:47.3173966Z GITHUB_HEAD_REF= 2025-08-14T21:36:47.3174115Z GITHUB_ACTION_REF= 2025-08-14T21:36:47.3174314Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-08-14T21:36:47.3174544Z TEST_SHOWLOCALS=False 2025-08-14T21:36:47.3174719Z GITHUB_WORKFLOW=inductor-periodic 2025-08-14T21:36:47.3174923Z DEBIAN_FRONTEND=noninteractive 2025-08-14T21:36:47.3175296Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3175656Z NO_TD=False 2025-08-14T21:36:47.3175814Z SKIP_SCCACHE_INITIALIZATION=1 2025-08-14T21:36:47.3176022Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-08-14T21:36:47.3176215Z _=/usr/bin/env 2025-08-14T21:36:47.3176418Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-08-14T21:36:47.3414096Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch 2025-08-14T21:36:47.3419276Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin 2025-08-14T21:36:47.3423584Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib 2025-08-14T21:36:47.3426061Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/test 2025-08-14T21:36:47.3426604Z + BUILD_DIR=build 2025-08-14T21:36:47.3426907Z + BUILD_RENAMED_DIR=build_renamed 2025-08-14T21:36:47.3427296Z + BUILD_BIN_DIR=build/bin 2025-08-14T21:36:47.3428010Z + SHARD_NUMBER=1 2025-08-14T21:36:47.3428254Z + NUM_TEST_SHARDS=1 2025-08-14T21:36:47.3428447Z + export TORCH_SERIALIZATION_DEBUG=1 2025-08-14T21:36:47.3428665Z + TORCH_SERIALIZATION_DEBUG=1 2025-08-14T21:36:47.3428853Z + export VALGRIND=ON 2025-08-14T21:36:47.3429021Z + VALGRIND=ON 2025-08-14T21:36:47.3429209Z + [[ linux-jammy-py3.9-gcc11-build == *clang9* ]] 2025-08-14T21:36:47.3429464Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-08-14T21:36:47.3429708Z + [[ linux-jammy-py3.9-gcc11-build == *s390x* ]] 2025-08-14T21:36:47.3429913Z + [[ 0 == \1 ]] 2025-08-14T21:36:47.3430090Z + [[ True == \1 ]] 2025-08-14T21:36:47.3430277Z + [[ linux-jammy-py3.9-gcc11-build != *bazel* ]] 2025-08-14T21:36:47.3430510Z ++ realpath build/custom_test_artifacts 2025-08-14T21:36:47.3431158Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-08-14T21:36:47.3431486Z + [[ -n '' ]] 2025-08-14T21:36:47.3431667Z + echo 'Environment variables' 2025-08-14T21:36:47.3431865Z Environment variables 2025-08-14T21:36:47.3432041Z + env 2025-08-14T21:36:47.3448633Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T21:36:47.3449054Z CONTINUE_THROUGH_ERROR=True 2025-08-14T21:36:47.3449402Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build 2025-08-14T21:36:47.3449772Z HOSTNAME=ec43c4531511 2025-08-14T21:36:47.3450628Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3451134Z GITHUB_ACTION=__run_2 2025-08-14T21:36:47.3451335Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-08-14T21:36:47.3451830Z GITHUB_RUN_NUMBER=66307 2025-08-14T21:36:47.3452051Z TEST_CONFIG=cpu_inductor_huggingface 2025-08-14T21:36:47.3452287Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-08-14T21:36:47.3452528Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-08-14T21:36:47.3452749Z SCCACHE_IDLE_TIMEOUT=0 2025-08-14T21:36:47.3453185Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-08-14T21:36:47.3453410Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-08-14T21:36:47.3453621Z GITHUB_REF_TYPE=branch 2025-08-14T21:36:47.3453812Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-08-14T21:36:47.3454026Z BASE_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:47.3454234Z XLA_CUDA= 2025-08-14T21:36:47.3454397Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-08-14T21:36:47.3454662Z HUGGING_FACE_HUB_TOKEN=*** 2025-08-14T21:36:47.3454899Z *** 2025-08-14T21:36:47.3455045Z GITHUB_REPOSITORY_ID=65600975 2025-08-14T21:36:47.3455228Z GITHUB_ACTIONS=true 2025-08-14T21:36:47.3455425Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-14T21:36:47.3455666Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:47.3455899Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:47.3456256Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/heads/main 2025-08-14T21:36:47.3456562Z UCC_HOME=/usr 2025-08-14T21:36:47.3456724Z TORCH_SERIALIZATION_DEBUG=1 2025-08-14T21:36:47.3456906Z VERBOSE_TEST_LOGS=False 2025-08-14T21:36:47.3457072Z GITHUB_REF=refs/heads/main 2025-08-14T21:36:47.3457247Z SHARD_NUMBER=1 2025-08-14T21:36:47.3457411Z GITHUB_REF_PROTECTED=true 2025-08-14T21:36:47.3457587Z HOME=/var/lib/jenkins 2025-08-14T21:36:47.3457776Z GITHUB_API_URL=https://api.github.com 2025-08-14T21:36:47.3457994Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-08-14T21:36:47.3458184Z UCX_COMMIT= 2025-08-14T21:36:47.3458323Z USE_SYSTEM_NCCL=1 2025-08-14T21:36:47.3458478Z NUM_TEST_SHARDS=1 2025-08-14T21:36:47.3458633Z UCX_HOME=/usr 2025-08-14T21:36:47.3458977Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3459565Z JOB_NAME=linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:36:47.3460136Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3460612Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-08-14T21:36:47.3460921Z GITHUB_EVENT_NAME=schedule 2025-08-14T21:36:47.3461106Z DASHBOARD_TAG= 2025-08-14T21:36:47.3461274Z GITHUB_RUN_ID=16976338999 2025-08-14T21:36:47.3461454Z INSTALLED_OPENBLAS= 2025-08-14T21:36:47.3461847Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3462284Z GITHUB_ACTOR=pytorchmergebot 2025-08-14T21:36:47.3462471Z PR_NUMBER= 2025-08-14T21:36:47.3462627Z DESIRED_CUDA= 2025-08-14T21:36:47.3462794Z GITHUB_RUN_ATTEMPT=1 2025-08-14T21:36:47.3462962Z VALGRIND=ON 2025-08-14T21:36:47.3463128Z ANACONDA_PYTHON_VERSION=3.9 2025-08-14T21:36:47.3463438Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-08-14T21:36:47.3463817Z TERM=vt100 2025-08-14T21:36:47.3463978Z INSTALLED_VISION=yes 2025-08-14T21:36:47.3464150Z BRANCH=main 2025-08-14T21:36:47.3464315Z SCCACHE_REGION=us-east-1 2025-08-14T21:36:47.3464507Z OPENSSL_ROOT_DIR=/opt/openssl 2025-08-14T21:36:47.3464708Z CUDA_PATH=/usr/local/cuda 2025-08-14T21:36:47.3465051Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-08-14T21:36:47.3465411Z GITHUB_SERVER_URL=https://github.com 2025-08-14T21:36:47.3465619Z UCC_COMMIT= 2025-08-14T21:36:47.3465773Z REENABLED_ISSUES= 2025-08-14T21:36:47.3465930Z DOCS=yes 2025-08-14T21:36:47.3466079Z SHLVL=1 2025-08-14T21:36:47.3466233Z MAX_JOBS=30 2025-08-14T21:36:47.3466373Z GITHUB_ACTOR_ID=97764156 2025-08-14T21:36:47.3466596Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:36:47.3466885Z GITHUB_REF_NAME=main 2025-08-14T21:36:47.3467130Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-08-14T21:36:47.3467398Z GITHUB_JOB=test 2025-08-14T21:36:47.3467554Z NO_TEST_TIMEOUT=False 2025-08-14T21:36:47.3467713Z TD_DISTRIBUTED=False 2025-08-14T21:36:47.3467892Z GITHUB_REPOSITORY=pytorch/pytorch 2025-08-14T21:36:47.3468089Z GITHUB_RETENTION_DAYS=90 2025-08-14T21:36:47.3468263Z OPENSSL_DIR=/opt/openssl 2025-08-14T21:36:47.3468431Z GITHUB_ACTION_REPOSITORY= 2025-08-14T21:36:47.3468889Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:36:47.3469341Z GITHUB_BASE_REF= 2025-08-14T21:36:47.3469491Z INSTALLED_ACL= 2025-08-14T21:36:47.3469762Z ARTIFACTS_FILE_SUFFIX=test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875 2025-08-14T21:36:47.3470060Z CI=true 2025-08-14T21:36:47.3470206Z GITHUB_REPOSITORY_OWNER=pytorch 2025-08-14T21:36:47.3470459Z RUST_LOG=sccache::server=error 2025-08-14T21:36:47.3470643Z JOB_ID=48128301875 2025-08-14T21:36:47.3470792Z GITHUB_HEAD_REF= 2025-08-14T21:36:47.3470953Z GITHUB_ACTION_REF= 2025-08-14T21:36:47.3471157Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-08-14T21:36:47.3471391Z TEST_SHOWLOCALS=False 2025-08-14T21:36:47.3471583Z GITHUB_WORKFLOW=inductor-periodic 2025-08-14T21:36:47.3471802Z DEBIAN_FRONTEND=noninteractive 2025-08-14T21:36:47.3472210Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_40aeeabb-1ef0-48f7-8690-ac8e3f31ea48 2025-08-14T21:36:47.3472607Z NO_TD=False 2025-08-14T21:36:47.3472782Z SKIP_SCCACHE_INITIALIZATION=1 2025-08-14T21:36:47.3473002Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-08-14T21:36:47.3473213Z _=/usr/bin/env 2025-08-14T21:36:47.3473386Z + echo 'Testing pytorch' 2025-08-14T21:36:47.3473575Z Testing pytorch 2025-08-14T21:36:47.3473765Z + export LANG=C.UTF-8 2025-08-14T21:36:47.3473943Z + LANG=C.UTF-8 2025-08-14T21:36:47.3474122Z + PR_NUMBER= 2025-08-14T21:36:47.3474312Z + [[ cpu_inductor_huggingface == \d\e\f\a\u\l\t ]] 2025-08-14T21:36:47.3474589Z + [[ cpu_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-08-14T21:36:47.3474858Z + [[ cpu_inductor_huggingface == \s\l\o\w ]] 2025-08-14T21:36:47.3475135Z + [[ linux-jammy-py3.9-gcc11-build == *slow-gradcheck* ]] 2025-08-14T21:36:47.3475411Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-08-14T21:36:47.3475669Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-08-14T21:36:47.3475924Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-08-14T21:36:47.3476405Z + [[ cpu_inductor_huggingface == *crossref* ]] 2025-08-14T21:36:47.3476657Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-08-14T21:36:47.3476910Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-08-14T21:36:47.3477161Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]] 2025-08-14T21:36:47.3477413Z + pip_install ninja==1.10.2 2025-08-14T21:36:47.3477685Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-08-14T21:36:47.3478007Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-08-14T21:36:47.7303775Z Collecting ninja==1.10.2 2025-08-14T21:36:47.7392868Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-08-14T21:36:47.7506091Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-08-14T21:36:48.5258576Z Installing collected packages: ninja 2025-08-14T21:36:48.5258882Z Attempting uninstall: ninja 2025-08-14T21:36:48.5264852Z Found existing installation: ninja 1.11.1.3 2025-08-14T21:36:48.5290008Z Uninstalling ninja-1.11.1.3: 2025-08-14T21:36:48.5342516Z Successfully uninstalled ninja-1.11.1.3 2025-08-14T21:36:48.6345828Z Successfully installed ninja-1.10.2 2025-08-14T21:36:48.7345296Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:36:48.7346312Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:36:48.7346930Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]] 2025-08-14T21:36:48.7347214Z + [[ linux-jammy-py3.9-gcc11-build == *asan* ]] 2025-08-14T21:36:48.7347474Z + [[ linux-jammy-py3.9-gcc11-build == *-debug* ]] 2025-08-14T21:36:48.7347741Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]] 2025-08-14T21:36:48.7348108Z + echo 'We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass' 2025-08-14T21:36:48.7348555Z We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass 2025-08-14T21:36:48.7349101Z + cd test 2025-08-14T21:36:48.7349520Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-08-14T21:36:49.9674234Z + [[ cpu_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-08-14T21:36:49.9674771Z + [[ cpu_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-08-14T21:36:49.9675293Z + [[ cpu_inductor_huggingface == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-08-14T21:36:49.9675619Z + DYNAMO_BENCHMARK_FLAGS=() 2025-08-14T21:36:49.9676495Z + [[ cpu_inductor_huggingface == *pr_time_benchmarks* ]] 2025-08-14T21:36:49.9676874Z + [[ cpu_inductor_huggingface == *dynamo_eager* ]] 2025-08-14T21:36:49.9677143Z + [[ cpu_inductor_huggingface == *aot_eager* ]] 2025-08-14T21:36:49.9677410Z + [[ cpu_inductor_huggingface == *aot_inductor* ]] 2025-08-14T21:36:49.9677713Z + [[ cpu_inductor_huggingface == *max_autotune_inductor* ]] 2025-08-14T21:36:49.9677959Z + [[ cpu_inductor_huggingface == *inductor* ]] 2025-08-14T21:36:49.9678188Z + [[ cpu_inductor_huggingface != *perf* ]] 2025-08-14T21:36:49.9678439Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-08-14T21:36:49.9678648Z + [[ cpu_inductor_huggingface == *dynamic* ]] 2025-08-14T21:36:49.9678923Z + [[ cpu_inductor_huggingface == *cpu* ]] 2025-08-14T21:36:49.9679141Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2025-08-14T21:36:50.0064885Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]] 2025-08-14T21:36:50.0065260Z + [[ linux-jammy-py3.9-gcc11-build == *-bazel-* ]] 2025-08-14T21:36:50.0072305Z + cd test 2025-08-14T21:36:50.0072733Z + python -c 'import torch; print(torch.__config__.show())' 2025-08-14T21:36:50.9722405Z PyTorch built with: 2025-08-14T21:36:50.9722821Z - GCC 11.4 2025-08-14T21:36:50.9726127Z - C++ Version: 201703 2025-08-14T21:36:50.9726527Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-08-14T21:36:50.9727007Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-08-14T21:36:50.9727356Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-08-14T21:36:50.9733176Z - LAPACK is enabled (usually provided by MKL) 2025-08-14T21:36:50.9737741Z - NNPACK is enabled 2025-08-14T21:36:50.9742767Z - CPU capability usage: AVX512 2025-08-14T21:36:50.9751226Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=1fc683cf17c8c673044538d10266c00f92987be2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-08-14T21:36:50.9754201Z 2025-08-14T21:36:51.2005018Z + cd test 2025-08-14T21:36:51.2005375Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-08-14T21:36:52.1714814Z ATen/Parallel: 2025-08-14T21:36:52.1715161Z at::get_num_threads() : 16 2025-08-14T21:36:52.1715405Z at::get_num_interop_threads() : 16 2025-08-14T21:36:52.1715654Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-08-14T21:36:52.1715868Z omp_get_max_threads() : 16 2025-08-14T21:36:52.1716327Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-08-14T21:36:52.1716720Z mkl_get_max_threads() : 16 2025-08-14T21:36:52.1717003Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-08-14T21:36:52.1717321Z std::thread::hardware_concurrency() : 32 2025-08-14T21:36:52.1717570Z Environment variables: 2025-08-14T21:36:52.1717769Z OMP_NUM_THREADS : [not set] 2025-08-14T21:36:52.1717965Z MKL_NUM_THREADS : [not set] 2025-08-14T21:36:52.1718170Z ATen parallel backend: OpenMP 2025-08-14T21:36:52.1718309Z 2025-08-14T21:36:52.3894659Z + [[ cpu_inductor_huggingface == *numpy_2* ]] 2025-08-14T21:36:52.3895050Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]] 2025-08-14T21:36:52.3895349Z + [[ cpu_inductor_huggingface == *backward* ]] 2025-08-14T21:36:52.3895630Z + [[ cpu_inductor_huggingface == *xla* ]] 2025-08-14T21:36:52.3895889Z + [[ cpu_inductor_huggingface == *executorch* ]] 2025-08-14T21:36:52.3896143Z + [[ cpu_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2025-08-14T21:36:52.3896420Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]] 2025-08-14T21:36:52.3896681Z + [[ cpu_inductor_huggingface == distributed ]] 2025-08-14T21:36:52.3896938Z + [[ cpu_inductor_huggingface == *operator_benchmark* ]] 2025-08-14T21:36:52.3897221Z + [[ cpu_inductor_huggingface == *inductor_distributed* ]] 2025-08-14T21:36:52.3897526Z + [[ cpu_inductor_huggingface == *inductor-halide* ]] 2025-08-14T21:36:52.3897803Z + [[ cpu_inductor_huggingface == *inductor-triton-cpu* ]] 2025-08-14T21:36:52.3898110Z + [[ cpu_inductor_huggingface == *inductor-micro-benchmark* ]] 2025-08-14T21:36:52.3898402Z + [[ cpu_inductor_huggingface == *huggingface* ]] 2025-08-14T21:36:52.3898627Z + install_torchvision 2025-08-14T21:36:52.3898801Z + local orig_preload 2025-08-14T21:36:52.3898976Z + local commit 2025-08-14T21:36:52.3899161Z ++ get_pinned_commit vision 2025-08-14T21:36:52.3899367Z ++ cat .github/ci_commit_pins/vision.txt 2025-08-14T21:36:52.4568775Z + commit=966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-08-14T21:36:52.4569221Z + orig_preload= 2025-08-14T21:36:52.4569477Z + '[' -n '' ']' 2025-08-14T21:36:52.4569700Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-08-14T21:36:52.4570276Z + pip_build_and_install git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 dist/vision 2025-08-14T21:36:52.4570997Z + local build_target=git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-08-14T21:36:52.4571443Z + local wheel_dir=dist/vision 2025-08-14T21:36:52.4571997Z + local found_whl=0 2025-08-14T21:36:52.4572216Z + for file in "${wheel_dir}"/*.whl 2025-08-14T21:36:52.4572565Z + [[ -f dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl ]] 2025-08-14T21:36:52.4572847Z + found_whl=1 2025-08-14T21:36:52.4572998Z + break 2025-08-14T21:36:52.4573144Z + '[' 1 == 0 ']' 2025-08-14T21:36:52.4573308Z + for file in "${wheel_dir}"/*.whl 2025-08-14T21:36:52.4573614Z + pip_install_whl dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:36:52.4574054Z + args=('dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl') 2025-08-14T21:36:52.4574329Z + local args 2025-08-14T21:36:52.4574592Z + [[ dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl == *\ * ]] 2025-08-14T21:36:52.4574899Z + for path in "${args[@]}" 2025-08-14T21:36:52.4575296Z + echo 'Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl' 2025-08-14T21:36:52.4575706Z Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:36:52.4576170Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:36:52.7478070Z Processing ./dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:36:52.7554675Z Installing collected packages: torchvision 2025-08-14T21:36:53.2731642Z Successfully installed torchvision-0.22.0a0+966da7e 2025-08-14T21:36:53.3153210Z + '[' -n '' ']' 2025-08-14T21:36:53.3153488Z + id=0 2025-08-14T21:36:53.3153679Z + test_dynamo_benchmark huggingface 0 2025-08-14T21:36:53.3153917Z ++ pwd 2025-08-14T21:36:53.3154156Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-08-14T21:36:53.3154446Z + local suite=huggingface 2025-08-14T21:36:53.3154645Z + shift 2025-08-14T21:36:53.3154836Z + local shard_id=0 2025-08-14T21:36:53.3155003Z + shift 2025-08-14T21:36:53.3155198Z + [[ cpu_inductor_huggingface == *perf_compare* ]] 2025-08-14T21:36:53.3155477Z + [[ cpu_inductor_huggingface == *perf* ]] 2025-08-14T21:36:53.3155713Z + [[ cpu_inductor_huggingface == *cpu* ]] 2025-08-14T21:36:53.3155944Z + local dt=float32 2025-08-14T21:36:53.3156300Z + [[ cpu_inductor_huggingface == *amp* ]] 2025-08-14T21:36:53.3156548Z + [[ cpu_inductor_huggingface == *freezing* ]] 2025-08-14T21:36:53.3156873Z + test_single_dynamo_benchmark inference huggingface 0 --inference --float32 2025-08-14T21:36:53.3163478Z ++ pwd 2025-08-14T21:36:53.3164299Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-08-14T21:36:53.3164719Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-08-14T21:36:53.3180552Z + local name=inference 2025-08-14T21:36:53.3180753Z + shift 2025-08-14T21:36:53.3180941Z + local suite=huggingface 2025-08-14T21:36:53.3181123Z + shift 2025-08-14T21:36:53.3181271Z + local shard_id=0 2025-08-14T21:36:53.3181457Z + shift 2025-08-14T21:36:53.3181608Z + partition_flags=() 2025-08-14T21:36:53.3181789Z + local partition_flags 2025-08-14T21:36:53.3181971Z + [[ -n 1 ]] 2025-08-14T21:36:53.3182121Z + [[ -n 0 ]] 2025-08-14T21:36:53.3182395Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-08-14T21:36:53.3182751Z + [[ cpu_inductor_huggingface == *perf_compare* ]] 2025-08-14T21:36:53.3183003Z + [[ cpu_inductor_huggingface == *perf* ]] 2025-08-14T21:36:53.3183259Z + [[ cpu_inductor_huggingface == *_avx2* ]] 2025-08-14T21:36:53.3183483Z + [[ cpu_inductor_huggingface == *_avx512* ]] 2025-08-14T21:36:53.3184229Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --print-compilation-time --inductor --device cpu --inference --float32 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2025-08-14T21:36:56.8033865Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:36:56.8035216Z from pkg_resources import resource_filename 2025-08-14T21:36:57.1995455Z 2025-08-14T21:36:57.2036191Z config.json: 0% 0.00/694 [00:00bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8479170Z 2025-08-14T21:39:09.8479284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8479842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8480360Z layer_outputs = layer_module( 2025-08-14T21:39:09.8480737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8481128Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8481584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8482024Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8482480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8482923Z self_outputs = self.self( 2025-08-14T21:39:09.8483345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8483870Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8484423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8485050Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8485314Z 2025-08-14T21:39:09.8485428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8485997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8486505Z layer_outputs = layer_module( 2025-08-14T21:39:09.8486872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8487257Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8487700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8488148Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8488582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8489014Z self_outputs = self.self( 2025-08-14T21:39:09.8489442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8489923Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8490449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8491173Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8491439Z 2025-08-14T21:39:09.8491551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8492093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8492597Z layer_outputs = layer_module( 2025-08-14T21:39:09.8492964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8493356Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8493862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8494298Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8494739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8495178Z self_outputs = self.self( 2025-08-14T21:39:09.8495599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8496058Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8496580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8497196Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8497449Z 2025-08-14T21:39:09.8497548Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8497772Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8497996Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8498224Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8498466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8499007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8499514Z layer_outputs = layer_module( 2025-08-14T21:39:09.8499883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8500257Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8500698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8501133Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8501567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8501998Z self_outputs = self.self( 2025-08-14T21:39:09.8502420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.8502889Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8503410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8503977Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.8504530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.8505093Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.8505310Z 2025-08-14T21:39:09.8505394Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8505688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8506216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8506707Z layer_outputs = layer_module( 2025-08-14T21:39:09.8507047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8507411Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8507847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8508277Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8508969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8509386Z self_outputs = self.self( 2025-08-14T21:39:09.8509789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.8510217Z attn_scores += diagonal_mask 2025-08-14T21:39:09.8510356Z 2025-08-14T21:39:09.8510465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8511002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8511504Z layer_outputs = layer_module( 2025-08-14T21:39:09.8511859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8512244Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8512698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8513142Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8513595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8514053Z self_outputs = self.self( 2025-08-14T21:39:09.8514485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.8514930Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.8515080Z 2025-08-14T21:39:09.8515190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8515739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8516333Z layer_outputs = layer_module( 2025-08-14T21:39:09.8516710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8517107Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8517523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8517926Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8518338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8518742Z self_outputs = self.self( 2025-08-14T21:39:09.8519134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8519582Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8520109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8520691Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.8521179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8521535Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8521724Z 2025-08-14T21:39:09.8521829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8522336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8522801Z layer_outputs = layer_module( 2025-08-14T21:39:09.8523146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8523507Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8523961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8524373Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8524786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8525194Z self_outputs = self.self( 2025-08-14T21:39:09.8525591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8526043Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8526549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8527078Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.8527586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.8528049Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.8528391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8528737Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8528886Z 2025-08-14T21:39:09.8528990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8529496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8529976Z layer_outputs = layer_module( 2025-08-14T21:39:09.8530321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8530674Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8531089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8531517Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8531916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8532304Z self_outputs = self.self( 2025-08-14T21:39:09.8532688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8533126Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8533622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8534175Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8534390Z 2025-08-14T21:39:09.8534504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8534999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8535493Z layer_outputs = layer_module( 2025-08-14T21:39:09.8535827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8536180Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8536586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8536989Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8537406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8537869Z self_outputs = self.self( 2025-08-14T21:39:09.8538258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8538702Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8539223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8539785Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8539989Z 2025-08-14T21:39:09.8540098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8540601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8541082Z layer_outputs = layer_module( 2025-08-14T21:39:09.8541429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8541782Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8542210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8542654Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8543088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8543489Z self_outputs = self.self( 2025-08-14T21:39:09.8543883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.8544407Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.8544646Z 2025-08-14T21:39:09.8544760Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8545266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8545751Z layer_outputs = layer_module( 2025-08-14T21:39:09.8546099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8546463Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8546876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8547292Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8547706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.8548149Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.8548600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.8549067Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8549206Z 2025-08-14T21:39:09.8549318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8549817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8550273Z layer_outputs = layer_module( 2025-08-14T21:39:09.8550600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8550945Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8551340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8551784Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8552188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8552577Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8552989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8553463Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8553937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.8554373Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8554526Z 2025-08-14T21:39:09.8554636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8555175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8555681Z layer_outputs = layer_module( 2025-08-14T21:39:09.8556141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8556543Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8556998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8557440Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8557885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8558264Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8558660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8559135Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8559572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.8560018Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.8560393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.8560721Z return self.act(input) 2025-08-14T21:39:09.8560839Z 2025-08-14T21:39:09.8560950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8561432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8561887Z layer_outputs = layer_module( 2025-08-14T21:39:09.8562211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8562564Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8562974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8563432Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8563857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8564276Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8564689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.8565141Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.8565608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.8566023Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8566165Z 2025-08-14T21:39:09.8566307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8566794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8567278Z layer_outputs = layer_module( 2025-08-14T21:39:09.8567623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8567976Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8568389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8568805Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8569216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8569617Z self_outputs = self.self( 2025-08-14T21:39:09.8570016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.8570439Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.8570575Z 2025-08-14T21:39:09.8570683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8571177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8571650Z layer_outputs = layer_module( 2025-08-14T21:39:09.8571992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8572347Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8572763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8573182Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8573614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8574040Z self_outputs = self.self( 2025-08-14T21:39:09.8574463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8574932Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8575462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8576034Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8576284Z 2025-08-14T21:39:09.8576388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8576894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8577373Z layer_outputs = layer_module( 2025-08-14T21:39:09.8577751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8578110Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8578526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8578930Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8579343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8579751Z self_outputs = self.self( 2025-08-14T21:39:09.8580142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.8580586Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.8580729Z 2025-08-14T21:39:09.8580831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8581344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8581826Z layer_outputs = layer_module( 2025-08-14T21:39:09.8582169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8582520Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8582934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8583361Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8583806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8584244Z self_outputs = self.self( 2025-08-14T21:39:09.8584662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8585143Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8585638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8586217Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8586466Z 2025-08-14T21:39:09.8586574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8587064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8587592Z layer_outputs = layer_module( 2025-08-14T21:39:09.8587936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8588298Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8588706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8589123Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8589532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8589931Z self_outputs = self.self( 2025-08-14T21:39:09.8590328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8590764Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8591260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8591860Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8592163Z 2025-08-14T21:39:09.8592273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8592816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8593324Z layer_outputs = layer_module( 2025-08-14T21:39:09.8593681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8594062Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8594505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8595004Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8595433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8595866Z self_outputs = self.self( 2025-08-14T21:39:09.8596375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8596850Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8597400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8598006Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8598254Z 2025-08-14T21:39:09.8598343Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8598550Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8598776Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8599006Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8599255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8599815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8600355Z layer_outputs = layer_module( 2025-08-14T21:39:09.8600731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8601125Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8601581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8602036Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8602494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8602931Z self_outputs = self.self( 2025-08-14T21:39:09.8603367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.8603857Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8604411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8605006Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.8605580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.8606162Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.8606386Z 2025-08-14T21:39:09.8606482Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8606735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8607286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8607840Z layer_outputs = layer_module( 2025-08-14T21:39:09.8608178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8608539Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8609092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8609513Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8609926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8610342Z self_outputs = self.self( 2025-08-14T21:39:09.8610815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.8611227Z attn_scores += diagonal_mask 2025-08-14T21:39:09.8611354Z 2025-08-14T21:39:09.8611455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8611969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8612451Z layer_outputs = layer_module( 2025-08-14T21:39:09.8612791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8613159Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8613578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8614064Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8614475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8614914Z self_outputs = self.self( 2025-08-14T21:39:09.8615341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.8615792Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.8615942Z 2025-08-14T21:39:09.8616046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8616558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8617040Z layer_outputs = layer_module( 2025-08-14T21:39:09.8617376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8617746Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8618160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8618574Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8618981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8619385Z self_outputs = self.self( 2025-08-14T21:39:09.8619779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.8620201Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.8620340Z 2025-08-14T21:39:09.8620442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8620945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8621435Z layer_outputs = layer_module( 2025-08-14T21:39:09.8621758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8622168Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8622579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8622992Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8623400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8623828Z self_outputs = self.self( 2025-08-14T21:39:09.8624246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8624762Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8625324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8625919Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.8626349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8626699Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8626859Z 2025-08-14T21:39:09.8626963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8627481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8627950Z layer_outputs = layer_module( 2025-08-14T21:39:09.8628283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8628648Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8629065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8629483Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8629892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8630305Z self_outputs = self.self( 2025-08-14T21:39:09.8630700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8631174Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8631727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8632308Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.8632850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.8633345Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.8633714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8634096Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8634260Z 2025-08-14T21:39:09.8634378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8634929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8635456Z layer_outputs = layer_module( 2025-08-14T21:39:09.8635837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8636306Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8636814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8637276Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8637739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8638196Z self_outputs = self.self( 2025-08-14T21:39:09.8638613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8639072Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8639657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8640265Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8640501Z 2025-08-14T21:39:09.8640614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8641166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8641689Z layer_outputs = layer_module( 2025-08-14T21:39:09.8642063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8642465Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8642928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8643440Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8643897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8644338Z self_outputs = self.self( 2025-08-14T21:39:09.8644774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8645270Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8645833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8646437Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8646658Z 2025-08-14T21:39:09.8646779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8647337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8647870Z layer_outputs = layer_module( 2025-08-14T21:39:09.8648251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8648648Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8649094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8649498Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8649902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8650303Z self_outputs = self.self( 2025-08-14T21:39:09.8650683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.8651194Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.8651424Z 2025-08-14T21:39:09.8651530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8652069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8652529Z layer_outputs = layer_module( 2025-08-14T21:39:09.8652876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8653240Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8653651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8654069Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8654524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.8654960Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.8655387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.8655810Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8655947Z 2025-08-14T21:39:09.8656054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8656545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8657003Z layer_outputs = layer_module( 2025-08-14T21:39:09.8657349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8657713Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8658127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8658554Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8658973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8659356Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8659752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8660197Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8660633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.8661047Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8661182Z 2025-08-14T21:39:09.8661283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8661780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8662264Z layer_outputs = layer_module( 2025-08-14T21:39:09.8662608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8662961Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8663388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8663835Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8664244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8664639Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8665059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8665511Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8666011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.8666493Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.8666897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.8667256Z return self.act(input) 2025-08-14T21:39:09.8667370Z 2025-08-14T21:39:09.8667473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8667984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8668462Z layer_outputs = layer_module( 2025-08-14T21:39:09.8668841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8669207Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8669627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8670055Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8670451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8670843Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8671260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.8671749Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.8672232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.8672681Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8672825Z 2025-08-14T21:39:09.8672943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8673479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8673987Z layer_outputs = layer_module( 2025-08-14T21:39:09.8674354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8674737Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8675178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8675628Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8676163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8676621Z self_outputs = self.self( 2025-08-14T21:39:09.8677057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.8677483Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.8677619Z 2025-08-14T21:39:09.8677736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8678244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8678723Z layer_outputs = layer_module( 2025-08-14T21:39:09.8679069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8679435Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8679844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8680258Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8680717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8681132Z self_outputs = self.self( 2025-08-14T21:39:09.8681518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8681963Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8682452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8683028Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8683265Z 2025-08-14T21:39:09.8683401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8683907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8684387Z layer_outputs = layer_module( 2025-08-14T21:39:09.8684729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8685085Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8685502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8685907Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8686300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8686704Z self_outputs = self.self( 2025-08-14T21:39:09.8687090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.8687491Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.8687622Z 2025-08-14T21:39:09.8687719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8688210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8688675Z layer_outputs = layer_module( 2025-08-14T21:39:09.8689008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8689351Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8689753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8690153Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8690558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8690946Z self_outputs = self.self( 2025-08-14T21:39:09.8691323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8691740Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8692202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8692766Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8693010Z 2025-08-14T21:39:09.8693572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8694082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8694550Z layer_outputs = layer_module( 2025-08-14T21:39:09.8694935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8695289Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8695705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8696118Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8696518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8696926Z self_outputs = self.self( 2025-08-14T21:39:09.8697324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8697789Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8698270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8698866Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8699125Z 2025-08-14T21:39:09.8699228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8699737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8700213Z layer_outputs = layer_module( 2025-08-14T21:39:09.8700562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8700906Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8701348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8701785Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8702228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8702662Z self_outputs = self.self( 2025-08-14T21:39:09.8703087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8703563Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8704085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8704704Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8704965Z 2025-08-14T21:39:09.8705056Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8705286Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8705501Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8705724Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8705972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8706507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8707015Z layer_outputs = layer_module( 2025-08-14T21:39:09.8707386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8707777Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8708213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8708650Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8709235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8709773Z self_outputs = self.self( 2025-08-14T21:39:09.8710191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.8710673Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8711202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8711769Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.8712323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.8712934Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.8713153Z 2025-08-14T21:39:09.8713243Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8713487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8714031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8714541Z layer_outputs = layer_module( 2025-08-14T21:39:09.8714908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8715301Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8715754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8716269Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8716728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8717161Z self_outputs = self.self( 2025-08-14T21:39:09.8717595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.8718019Z attn_scores += diagonal_mask 2025-08-14T21:39:09.8718141Z 2025-08-14T21:39:09.8718245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8718748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8719224Z layer_outputs = layer_module( 2025-08-14T21:39:09.8719566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8719922Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8720341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8720756Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8721170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8721567Z self_outputs = self.self( 2025-08-14T21:39:09.8721962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.8722373Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.8722501Z 2025-08-14T21:39:09.8722611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8723106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8723582Z layer_outputs = layer_module( 2025-08-14T21:39:09.8723928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8724281Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8724750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8725150Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8725548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8725937Z self_outputs = self.self( 2025-08-14T21:39:09.8726320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.8726729Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.8726864Z 2025-08-14T21:39:09.8726971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8727553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8728026Z layer_outputs = layer_module( 2025-08-14T21:39:09.8728360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8728709Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8729116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8729519Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8729924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8730320Z self_outputs = self.self( 2025-08-14T21:39:09.8730711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8731161Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8731680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8732243Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.8732657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8733003Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8733152Z 2025-08-14T21:39:09.8733262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8733761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8734248Z layer_outputs = layer_module( 2025-08-14T21:39:09.8734597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8734958Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8735377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8735798Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8736206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8736603Z self_outputs = self.self( 2025-08-14T21:39:09.8736992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8737437Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8737949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8738517Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.8739012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.8739474Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.8739808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8740156Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8740317Z 2025-08-14T21:39:09.8740428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8740950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8741470Z layer_outputs = layer_module( 2025-08-14T21:39:09.8741822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8742200Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8742606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8743003Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8743413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8743825Z self_outputs = self.self( 2025-08-14T21:39:09.8744214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8744667Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8745188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8745756Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8745960Z 2025-08-14T21:39:09.8746062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8746569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8747051Z layer_outputs = layer_module( 2025-08-14T21:39:09.8747397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8747752Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8748172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8748586Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8748997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8749400Z self_outputs = self.self( 2025-08-14T21:39:09.8749798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8750252Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8750760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8751316Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8751525Z 2025-08-14T21:39:09.8751631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8752136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8752666Z layer_outputs = layer_module( 2025-08-14T21:39:09.8753004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8753375Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8753809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8754247Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8754683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8755122Z self_outputs = self.self( 2025-08-14T21:39:09.8755591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.8756224Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.8756499Z 2025-08-14T21:39:09.8756614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8757174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8757657Z layer_outputs = layer_module( 2025-08-14T21:39:09.8758000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8758368Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8758794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8759207Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8759640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.8760101Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.8760553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.8760974Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8761124Z 2025-08-14T21:39:09.8761227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8761738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8762224Z layer_outputs = layer_module( 2025-08-14T21:39:09.8762567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8762936Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8763359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8763791Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8764189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8764595Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8765006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8765447Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8765897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.8766322Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8766462Z 2025-08-14T21:39:09.8766571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8767075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8767600Z layer_outputs = layer_module( 2025-08-14T21:39:09.8767957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8768337Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8768765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8769208Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8769628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8770069Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8770507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8770957Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8771397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.8771824Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.8772190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.8772522Z return self.act(input) 2025-08-14T21:39:09.8772632Z 2025-08-14T21:39:09.8772741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8773221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8773699Z layer_outputs = layer_module( 2025-08-14T21:39:09.8774043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8774396Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8774811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8775233Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8775631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8776020Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8776429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.8776889Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.8777340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.8777746Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8777885Z 2025-08-14T21:39:09.8777987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8778492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8778972Z layer_outputs = layer_module( 2025-08-14T21:39:09.8779307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8779669Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8780082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8780494Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8780906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8781362Z self_outputs = self.self( 2025-08-14T21:39:09.8781760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.8782174Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.8782320Z 2025-08-14T21:39:09.8782425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8782934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8783419Z layer_outputs = layer_module( 2025-08-14T21:39:09.8783760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8784159Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8784577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8784987Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8785401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8785812Z self_outputs = self.self( 2025-08-14T21:39:09.8786205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8786638Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8787156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8787763Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8788002Z 2025-08-14T21:39:09.8788110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8788615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8789091Z layer_outputs = layer_module( 2025-08-14T21:39:09.8789434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8789795Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8790225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8790669Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8791111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8791545Z self_outputs = self.self( 2025-08-14T21:39:09.8791967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.8792444Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.8792591Z 2025-08-14T21:39:09.8792713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8793269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8793801Z layer_outputs = layer_module( 2025-08-14T21:39:09.8794185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8794588Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8795037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8795488Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8796056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8796513Z self_outputs = self.self( 2025-08-14T21:39:09.8796942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8797420Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8797959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8798582Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8798854Z 2025-08-14T21:39:09.8799017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8799576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8800102Z layer_outputs = layer_module( 2025-08-14T21:39:09.8800469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8800860Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8801307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8801757Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8802197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8802640Z self_outputs = self.self( 2025-08-14T21:39:09.8803072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8803541Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8804081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8804706Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8804966Z 2025-08-14T21:39:09.8805085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8805627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8806151Z layer_outputs = layer_module( 2025-08-14T21:39:09.8806509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8806888Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8807317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8807752Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8808184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8808608Z self_outputs = self.self( 2025-08-14T21:39:09.8809168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8809640Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8810163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8810773Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8811033Z 2025-08-14T21:39:09.8811120Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8811436Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8811658Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8811870Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8812117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8812653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8813155Z layer_outputs = layer_module( 2025-08-14T21:39:09.8813521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8813905Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8814411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8814849Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8815292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8815742Z self_outputs = self.self( 2025-08-14T21:39:09.8816165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.8816634Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8817170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8817744Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.8818285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.8818822Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.8819038Z 2025-08-14T21:39:09.8819116Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8819354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8819878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8820361Z layer_outputs = layer_module( 2025-08-14T21:39:09.8820709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8821074Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8821488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8821914Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8822194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8822279Z self_outputs = self.self( 2025-08-14T21:39:09.8822577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.8822664Z attn_scores += diagonal_mask 2025-08-14T21:39:09.8822667Z 2025-08-14T21:39:09.8822778Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8823150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8823234Z layer_outputs = layer_module( 2025-08-14T21:39:09.8823469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8823564Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8823856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8823975Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8824276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8824348Z self_outputs = self.self( 2025-08-14T21:39:09.8824648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.8824739Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.8824742Z 2025-08-14T21:39:09.8824861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8825251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8825325Z layer_outputs = layer_module( 2025-08-14T21:39:09.8825541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8825630Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8825905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8825985Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8826262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8826329Z self_outputs = self.self( 2025-08-14T21:39:09.8826614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.8826703Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.8826706Z 2025-08-14T21:39:09.8826813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8827160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8827233Z layer_outputs = layer_module( 2025-08-14T21:39:09.8827454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8827531Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8827806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8827884Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8828157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8828233Z self_outputs = self.self( 2025-08-14T21:39:09.8828505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8828626Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8828978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8829149Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.8829345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8829444Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8829448Z 2025-08-14T21:39:09.8829548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8829904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8829973Z layer_outputs = layer_module( 2025-08-14T21:39:09.8830237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8830316Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8830591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8830672Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8830947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8831016Z self_outputs = self.self( 2025-08-14T21:39:09.8831296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8831454Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8831828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8831973Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.8832310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.8832414Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.8832618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8832728Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8832732Z 2025-08-14T21:39:09.8832841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8833214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8833300Z layer_outputs = layer_module( 2025-08-14T21:39:09.8833533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8833624Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8833920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8833999Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8834300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8834372Z self_outputs = self.self( 2025-08-14T21:39:09.8834667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8834796Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8835168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8835339Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8835343Z 2025-08-14T21:39:09.8835453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8835832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8835917Z layer_outputs = layer_module( 2025-08-14T21:39:09.8836216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8836311Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8836620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8836746Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8837060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8837134Z self_outputs = self.self( 2025-08-14T21:39:09.8837450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8837577Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8837957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8838126Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8838165Z 2025-08-14T21:39:09.8838274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8838646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8838724Z layer_outputs = layer_module( 2025-08-14T21:39:09.8838953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8839044Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8839338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8839417Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8839716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8839790Z self_outputs = self.self( 2025-08-14T21:39:09.8840092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.8840293Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.8840297Z 2025-08-14T21:39:09.8840401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8840777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8840853Z layer_outputs = layer_module( 2025-08-14T21:39:09.8841094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8841178Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8841490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8841574Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8841870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.8841997Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.8842290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.8842378Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8842382Z 2025-08-14T21:39:09.8842496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8842870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8842952Z layer_outputs = layer_module( 2025-08-14T21:39:09.8843183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8843264Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8843611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8843693Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8843941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8844025Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8844301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8844419Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8844726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.8844809Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8844813Z 2025-08-14T21:39:09.8844923Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8845269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8845345Z layer_outputs = layer_module( 2025-08-14T21:39:09.8845562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8845640Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8845926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8846006Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8846273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8862683Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8863218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8863350Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8863673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.8863809Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.8864047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.8864139Z return self.act(input) 2025-08-14T21:39:09.8864146Z 2025-08-14T21:39:09.8864271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8864671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8864764Z layer_outputs = layer_module( 2025-08-14T21:39:09.8864992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8865086Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8865369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8865459Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8865727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8865809Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8866103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.8866231Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.8866519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.8866727Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8866731Z 2025-08-14T21:39:09.8866836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8867183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8867265Z layer_outputs = layer_module( 2025-08-14T21:39:09.8867484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8867572Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8867905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8867987Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8868273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8868347Z self_outputs = self.self( 2025-08-14T21:39:09.8868637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.8868720Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.8868725Z 2025-08-14T21:39:09.8868833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8869194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8869268Z layer_outputs = layer_module( 2025-08-14T21:39:09.8869499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8869578Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8869861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8869947Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8870226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8870295Z self_outputs = self.self( 2025-08-14T21:39:09.8870581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8870689Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8871041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8871231Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8871237Z 2025-08-14T21:39:09.8871344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8871702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8871774Z layer_outputs = layer_module( 2025-08-14T21:39:09.8872004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8872087Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8872390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8872478Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8872783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8872903Z self_outputs = self.self( 2025-08-14T21:39:09.8873203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.8873286Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.8873291Z 2025-08-14T21:39:09.8873407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8873791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8873865Z layer_outputs = layer_module( 2025-08-14T21:39:09.8874106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8874237Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8874547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8874631Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8874927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8875007Z self_outputs = self.self( 2025-08-14T21:39:09.8875319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8875440Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8875814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8876121Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8876129Z 2025-08-14T21:39:09.8876252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8876632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8876715Z layer_outputs = layer_module( 2025-08-14T21:39:09.8876951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8877034Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8877340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8877416Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8877690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8877770Z self_outputs = self.self( 2025-08-14T21:39:09.8878043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8878154Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8878491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8878674Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8878678Z 2025-08-14T21:39:09.8878788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8879192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8879270Z layer_outputs = layer_module( 2025-08-14T21:39:09.8879484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8879559Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8879872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8879948Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8880232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8880302Z self_outputs = self.self( 2025-08-14T21:39:09.8880575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8880681Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8881042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8881224Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8881238Z 2025-08-14T21:39:09.8881324Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8881404Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8881488Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8881564Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8881665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8882021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8882092Z layer_outputs = layer_module( 2025-08-14T21:39:09.8882316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8882396Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8882673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8882759Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8883032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8883103Z self_outputs = self.self( 2025-08-14T21:39:09.8883383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.8883497Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8883867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8884023Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.8884363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.8884533Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.8884537Z 2025-08-14T21:39:09.8884620Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8884745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8885092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8885165Z layer_outputs = layer_module( 2025-08-14T21:39:09.8885389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8885466Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8885755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8885831Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8886148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8886231Z self_outputs = self.self( 2025-08-14T21:39:09.8886522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.8886601Z attn_scores += diagonal_mask 2025-08-14T21:39:09.8886605Z 2025-08-14T21:39:09.8886719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8887085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8887168Z layer_outputs = layer_module( 2025-08-14T21:39:09.8887429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8887517Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8887820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8887897Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8888195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8888266Z self_outputs = self.self( 2025-08-14T21:39:09.8888543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.8888632Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.8888635Z 2025-08-14T21:39:09.8888742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8889109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8889194Z layer_outputs = layer_module( 2025-08-14T21:39:09.8889423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8889510Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8889803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8889887Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8890171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8890238Z self_outputs = self.self( 2025-08-14T21:39:09.8890520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.8890603Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.8890609Z 2025-08-14T21:39:09.8890708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8891060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8891130Z layer_outputs = layer_module( 2025-08-14T21:39:09.8891352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8891431Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8891707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8891787Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8892065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8892132Z self_outputs = self.self( 2025-08-14T21:39:09.8892449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8892568Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8892937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8893124Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.8893332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8893447Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8893451Z 2025-08-14T21:39:09.8893589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8893969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8894049Z layer_outputs = layer_module( 2025-08-14T21:39:09.8894280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8894372Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8894664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8894744Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8895047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8895120Z self_outputs = self.self( 2025-08-14T21:39:09.8895423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8895551Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8895919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8896074Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.8896409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.8896515Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.8896720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8896825Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8896828Z 2025-08-14T21:39:09.8896948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8897314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8897400Z layer_outputs = layer_module( 2025-08-14T21:39:09.8897627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8897710Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8898011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8898089Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8898380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8898462Z self_outputs = self.self( 2025-08-14T21:39:09.8898753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8898918Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8899292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8899456Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8899469Z 2025-08-14T21:39:09.8899576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8899951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8900032Z layer_outputs = layer_module( 2025-08-14T21:39:09.8901152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8901248Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8901556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8901637Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8901949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8902022Z self_outputs = self.self( 2025-08-14T21:39:09.8902312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8902442Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8902812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8902979Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8902986Z 2025-08-14T21:39:09.8903096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8903478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8903564Z layer_outputs = layer_module( 2025-08-14T21:39:09.8903796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8903878Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8904184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8904262Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8904574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8904647Z self_outputs = self.self( 2025-08-14T21:39:09.8904943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.8905152Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.8905155Z 2025-08-14T21:39:09.8905261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8905636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8905713Z layer_outputs = layer_module( 2025-08-14T21:39:09.8905945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8906039Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8906337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8906462Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8906758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.8906878Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.8907185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.8907273Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8907276Z 2025-08-14T21:39:09.8907381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8907800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8907877Z layer_outputs = layer_module( 2025-08-14T21:39:09.8908117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8908200Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8908489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8908584Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8909036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8909125Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8909426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8909558Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8909862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.8909961Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8909965Z 2025-08-14T21:39:09.8910072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8910451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8910534Z layer_outputs = layer_module( 2025-08-14T21:39:09.8910768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8910849Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8911157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8911246Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8911534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8911618Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8911920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8912045Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8912345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.8912470Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.8912699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.8912774Z return self.act(input) 2025-08-14T21:39:09.8912778Z 2025-08-14T21:39:09.8912894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8913271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8913444Z layer_outputs = layer_module( 2025-08-14T21:39:09.8913671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8913753Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8914049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8914133Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8914404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8914491Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8914840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.8914981Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.8915273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.8915359Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8915363Z 2025-08-14T21:39:09.8915478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8915840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8915920Z layer_outputs = layer_module( 2025-08-14T21:39:09.8916217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8916302Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8916601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8916687Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8916987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8917069Z self_outputs = self.self( 2025-08-14T21:39:09.8917368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.8917463Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.8917467Z 2025-08-14T21:39:09.8917576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8917962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8918045Z layer_outputs = layer_module( 2025-08-14T21:39:09.8918271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8918363Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8918654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8918734Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8919037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8919111Z self_outputs = self.self( 2025-08-14T21:39:09.8919422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8919536Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8919899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8920153Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8920157Z 2025-08-14T21:39:09.8920264Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8920648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8920724Z layer_outputs = layer_module( 2025-08-14T21:39:09.8920959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8921050Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8921391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8921474Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8921783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8921858Z self_outputs = self.self( 2025-08-14T21:39:09.8922164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.8922250Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.8922254Z 2025-08-14T21:39:09.8922363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8922743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8922819Z layer_outputs = layer_module( 2025-08-14T21:39:09.8923062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8923145Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8923447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8923533Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8923841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8923916Z self_outputs = self.self( 2025-08-14T21:39:09.8924223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8924331Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8924703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8924901Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8924908Z 2025-08-14T21:39:09.8925017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8925409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8925484Z layer_outputs = layer_module( 2025-08-14T21:39:09.8925723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8925805Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8926103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8926192Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8926503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8926629Z self_outputs = self.self( 2025-08-14T21:39:09.8926929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8927037Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8927414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8927608Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8927612Z 2025-08-14T21:39:09.8927730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8928143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8928222Z layer_outputs = layer_module( 2025-08-14T21:39:09.8928464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8928551Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8928849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8928936Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8929225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8929302Z self_outputs = self.self( 2025-08-14T21:39:09.8929573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8929676Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8930013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8930195Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8930198Z 2025-08-14T21:39:09.8930288Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8930366Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8930443Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8930526Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8930627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8930973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8931051Z layer_outputs = layer_module( 2025-08-14T21:39:09.8931269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8931352Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8931632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8931711Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8932010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8932082Z self_outputs = self.self( 2025-08-14T21:39:09.8932386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.8932501Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8932862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8933018Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.8933395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.8933562Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.8933573Z 2025-08-14T21:39:09.8933651Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8933753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8934175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8934247Z layer_outputs = layer_module( 2025-08-14T21:39:09.8934499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8934585Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8934859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8934943Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8935219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8935288Z self_outputs = self.self( 2025-08-14T21:39:09.8935565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.8935639Z attn_scores += diagonal_mask 2025-08-14T21:39:09.8935643Z 2025-08-14T21:39:09.8935750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8936099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8936171Z layer_outputs = layer_module( 2025-08-14T21:39:09.8936395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8936473Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8936749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8936830Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8937104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8937179Z self_outputs = self.self( 2025-08-14T21:39:09.8937451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.8937533Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.8937536Z 2025-08-14T21:39:09.8937645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8938002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8938079Z layer_outputs = layer_module( 2025-08-14T21:39:09.8938294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8938371Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8938653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8938726Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8939002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8939080Z self_outputs = self.self( 2025-08-14T21:39:09.8939352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.8939483Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.8939487Z 2025-08-14T21:39:09.8939586Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8939928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8940006Z layer_outputs = layer_module( 2025-08-14T21:39:09.8940221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8940306Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8940619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8940694Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8940976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8941047Z self_outputs = self.self( 2025-08-14T21:39:09.8941328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8941443Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8941791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8941973Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.8942163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8942264Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8942275Z 2025-08-14T21:39:09.8942376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8942722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8942798Z layer_outputs = layer_module( 2025-08-14T21:39:09.8943015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8943092Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8943377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8943450Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8943740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8943809Z self_outputs = self.self( 2025-08-14T21:39:09.8944091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8944220Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8944584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8944735Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.8945071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.8945167Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.8945381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8945477Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8945481Z 2025-08-14T21:39:09.8945612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8945962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8946034Z layer_outputs = layer_module( 2025-08-14T21:39:09.8946256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8946334Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8946611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8946693Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8947007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8947087Z self_outputs = self.self( 2025-08-14T21:39:09.8947371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8947482Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8947826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8947974Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8947978Z 2025-08-14T21:39:09.8948082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8948414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8948485Z layer_outputs = layer_module( 2025-08-14T21:39:09.8948703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8948779Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8949048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8949127Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8949395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8949471Z self_outputs = self.self( 2025-08-14T21:39:09.8949745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8949856Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8950212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8950363Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8950367Z 2025-08-14T21:39:09.8950472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8950821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8950890Z layer_outputs = layer_module( 2025-08-14T21:39:09.8951106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8951183Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8951459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8951537Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8951811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8951932Z self_outputs = self.self( 2025-08-14T21:39:09.8952208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.8952395Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.8952407Z 2025-08-14T21:39:09.8952508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8952852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8952930Z layer_outputs = layer_module( 2025-08-14T21:39:09.8953182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8953262Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8953549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8953623Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8953909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.8954026Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.8954328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.8954423Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8954427Z 2025-08-14T21:39:09.8954532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8954908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8954986Z layer_outputs = layer_module( 2025-08-14T21:39:09.8955225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8955313Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8955624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8955720Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8956256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8956347Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8956672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8956793Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8957108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.8957204Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8957208Z 2025-08-14T21:39:09.8957321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8957690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8957761Z layer_outputs = layer_module( 2025-08-14T21:39:09.8957980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8958069Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8958349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8958442Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8958750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8958839Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8959123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.8959228Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.8959498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.8959614Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.8959855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.8959935Z return self.act(input) 2025-08-14T21:39:09.8959938Z 2025-08-14T21:39:09.8960041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8960392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8960471Z layer_outputs = layer_module( 2025-08-14T21:39:09.8960693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8960777Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8961059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.8961140Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.8961407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.8961482Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.8961774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.8961895Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.8962173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.8962263Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.8962267Z 2025-08-14T21:39:09.8962368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8962715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8962794Z layer_outputs = layer_module( 2025-08-14T21:39:09.8963014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8963100Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8963375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8963451Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8963738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8963807Z self_outputs = self.self( 2025-08-14T21:39:09.8964092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.8964172Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.8964175Z 2025-08-14T21:39:09.8964278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8964632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8964743Z layer_outputs = layer_module( 2025-08-14T21:39:09.8964967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8965043Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8965321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8965404Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8965684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8965751Z self_outputs = self.self( 2025-08-14T21:39:09.8966066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8966168Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8966513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8966696Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8966699Z 2025-08-14T21:39:09.8966800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8967154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8967223Z layer_outputs = layer_module( 2025-08-14T21:39:09.8967447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8967526Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8967799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8967883Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8968160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8968235Z self_outputs = self.self( 2025-08-14T21:39:09.8968509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.8968587Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.8968591Z 2025-08-14T21:39:09.8968698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8969045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8969116Z layer_outputs = layer_module( 2025-08-14T21:39:09.8969336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8969414Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8969694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8969768Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8970045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8970122Z self_outputs = self.self( 2025-08-14T21:39:09.8970400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8970507Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8970845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8971065Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8971068Z 2025-08-14T21:39:09.8971179Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8971523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8971601Z layer_outputs = layer_module( 2025-08-14T21:39:09.8971820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8971896Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8972222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8972300Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8972576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8972653Z self_outputs = self.self( 2025-08-14T21:39:09.8972924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8973030Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8973365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8973545Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8973548Z 2025-08-14T21:39:09.8973657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8974006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8974087Z layer_outputs = layer_module( 2025-08-14T21:39:09.8974308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8974385Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8974672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8974747Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8975035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8975103Z self_outputs = self.self( 2025-08-14T21:39:09.8975380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.8975487Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8975826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8976003Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.8976015Z 2025-08-14T21:39:09.8976097Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8976176Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8976271Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8976345Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8976444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8976792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8976861Z layer_outputs = layer_module( 2025-08-14T21:39:09.8977079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8977194Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8977462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8977541Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8977811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8977890Z self_outputs = self.self( 2025-08-14T21:39:09.8978163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.8978269Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.8978637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.8978781Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.8979095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.8979251Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.8979254Z 2025-08-14T21:39:09.8979334Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.8979444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8979785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8979860Z layer_outputs = layer_module( 2025-08-14T21:39:09.8980084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8980166Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8980443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8980518Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8980791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8980869Z self_outputs = self.self( 2025-08-14T21:39:09.8981138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.8981213Z attn_scores += diagonal_mask 2025-08-14T21:39:09.8981223Z 2025-08-14T21:39:09.8981324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8981668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8981751Z layer_outputs = layer_module( 2025-08-14T21:39:09.8981969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8982049Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8982347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8982421Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8982700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8982770Z self_outputs = self.self( 2025-08-14T21:39:09.8983076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.8983171Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.8983207Z 2025-08-14T21:39:09.8983316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8983690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8983764Z layer_outputs = layer_module( 2025-08-14T21:39:09.8983995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8984082Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8984386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8984463Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8984808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8984881Z self_outputs = self.self( 2025-08-14T21:39:09.8985177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.8985261Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.8985265Z 2025-08-14T21:39:09.8985365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8985717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8985788Z layer_outputs = layer_module( 2025-08-14T21:39:09.8986011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8986086Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8986365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8986451Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8986729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8986804Z self_outputs = self.self( 2025-08-14T21:39:09.8987077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8987194Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8987547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8987720Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.8987914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8988028Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8988032Z 2025-08-14T21:39:09.8988136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8988509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8988583Z layer_outputs = layer_module( 2025-08-14T21:39:09.8988813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8988900Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8989200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8989286Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8989576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8989712Z self_outputs = self.self( 2025-08-14T21:39:09.8990017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8990139Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8990528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8990672Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.8991018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.8991153Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.8991358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.8991465Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.8991476Z 2025-08-14T21:39:09.8991582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8991960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8992042Z layer_outputs = layer_module( 2025-08-14T21:39:09.8992274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8992356Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8992667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8992749Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8993050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8993125Z self_outputs = self.self( 2025-08-14T21:39:09.8993426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8993558Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8993937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8994110Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8994114Z 2025-08-14T21:39:09.8994223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8994613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8994698Z layer_outputs = layer_module( 2025-08-14T21:39:09.8994940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8995026Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8995337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8995420Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8995729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8995804Z self_outputs = self.self( 2025-08-14T21:39:09.8996196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.8996340Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.8996719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.8996938Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.8996943Z 2025-08-14T21:39:09.8997053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8997445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8997530Z layer_outputs = layer_module( 2025-08-14T21:39:09.8997758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.8997850Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.8998184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.8998269Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.8998574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.8998646Z self_outputs = self.self( 2025-08-14T21:39:09.8998937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.8999142Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.8999145Z 2025-08-14T21:39:09.8999251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.8999627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.8999701Z layer_outputs = layer_module( 2025-08-14T21:39:09.8999931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9000025Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9000314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9000398Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9000687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.9000803Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.9001101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.9001188Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9001194Z 2025-08-14T21:39:09.9001306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9001672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9001749Z layer_outputs = layer_module( 2025-08-14T21:39:09.9001986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9002066Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9002364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9002450Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9002722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9002813Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9003110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9003260Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9003562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.9003647Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9003650Z 2025-08-14T21:39:09.9003762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9004126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9004207Z layer_outputs = layer_module( 2025-08-14T21:39:09.9004467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9004551Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9004845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9004934Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9005202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9005290Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9005583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9005702Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9006002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.9006121Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.9006353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.9006432Z return self.act(input) 2025-08-14T21:39:09.9006435Z 2025-08-14T21:39:09.9006548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9006921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9007007Z layer_outputs = layer_module( 2025-08-14T21:39:09.9007229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9007306Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9007584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9007674Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9007926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9008012Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9008288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.9008416Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.9008881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.9008974Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9008978Z 2025-08-14T21:39:09.9009091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9009471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9009547Z layer_outputs = layer_module( 2025-08-14T21:39:09.9009862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9009939Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9010224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9010298Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9010574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9010650Z self_outputs = self.self( 2025-08-14T21:39:09.9010925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.9011053Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.9011057Z 2025-08-14T21:39:09.9011168Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9011512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9011589Z layer_outputs = layer_module( 2025-08-14T21:39:09.9011807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9011884Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9012167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9012241Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9012525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9012598Z self_outputs = self.self( 2025-08-14T21:39:09.9012871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9012982Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9013321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9013503Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9013513Z 2025-08-14T21:39:09.9013614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9013957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9014034Z layer_outputs = layer_module( 2025-08-14T21:39:09.9014253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9014330Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9014615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9014688Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9014974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9015045Z self_outputs = self.self( 2025-08-14T21:39:09.9015319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.9015406Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.9015410Z 2025-08-14T21:39:09.9015509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9015864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9015973Z layer_outputs = layer_module( 2025-08-14T21:39:09.9016189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9016275Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9016549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9016622Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9016901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9016969Z self_outputs = self.self( 2025-08-14T21:39:09.9017277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9017380Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9017717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9017907Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9017911Z 2025-08-14T21:39:09.9018012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9018368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9018437Z layer_outputs = layer_module( 2025-08-14T21:39:09.9018647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9018731Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9019003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9019088Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9019360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9019428Z self_outputs = self.self( 2025-08-14T21:39:09.9019708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9019806Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9020137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9020325Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9020328Z 2025-08-14T21:39:09.9020427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9020782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9020853Z layer_outputs = layer_module( 2025-08-14T21:39:09.9021065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9021149Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9021420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9021500Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9021772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9021844Z self_outputs = self.self( 2025-08-14T21:39:09.9022135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9022285Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9022644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9022833Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9022837Z 2025-08-14T21:39:09.9022923Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9023015Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9023096Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9023177Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9023293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9023694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9023781Z layer_outputs = layer_module( 2025-08-14T21:39:09.9024015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9024090Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9024365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9024438Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9024712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9024780Z self_outputs = self.self( 2025-08-14T21:39:09.9025047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.9025162Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9025486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9025624Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.9025939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.9026085Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.9026089Z 2025-08-14T21:39:09.9026173Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9026269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9026607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9026682Z layer_outputs = layer_module( 2025-08-14T21:39:09.9026895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9026976Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9027248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9027319Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9027589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9027653Z self_outputs = self.self( 2025-08-14T21:39:09.9027929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.9028002Z attn_scores += diagonal_mask 2025-08-14T21:39:09.9028005Z 2025-08-14T21:39:09.9028103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9028481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9028549Z layer_outputs = layer_module( 2025-08-14T21:39:09.9028760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9028845Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9029120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9029200Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9029474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9029572Z self_outputs = self.self( 2025-08-14T21:39:09.9029852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.9029933Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.9029937Z 2025-08-14T21:39:09.9030045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9030385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9030454Z layer_outputs = layer_module( 2025-08-14T21:39:09.9030674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9030750Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9031031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9031104Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9031384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9031466Z self_outputs = self.self( 2025-08-14T21:39:09.9031751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.9031836Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.9031841Z 2025-08-14T21:39:09.9031953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9032313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9032394Z layer_outputs = layer_module( 2025-08-14T21:39:09.9032627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9032711Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9033016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9033098Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9033402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9033475Z self_outputs = self.self( 2025-08-14T21:39:09.9033770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9033904Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9034287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9034467Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.9034720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9034830Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9034834Z 2025-08-14T21:39:09.9034952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9035335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9035411Z layer_outputs = layer_module( 2025-08-14T21:39:09.9035657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9035740Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9036154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9036242Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9036599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9036683Z self_outputs = self.self( 2025-08-14T21:39:09.9036988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9037121Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9037511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9037657Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.9038011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.9038111Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.9038322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9038436Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9038441Z 2025-08-14T21:39:09.9038552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9038939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9039016Z layer_outputs = layer_module( 2025-08-14T21:39:09.9039253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9039344Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9039654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9039742Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9040052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9040126Z self_outputs = self.self( 2025-08-14T21:39:09.9040433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9040558Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9040952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9041118Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9041122Z 2025-08-14T21:39:09.9041234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9041701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9041823Z layer_outputs = layer_module( 2025-08-14T21:39:09.9042072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9042155Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9042466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9042552Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9042871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9042944Z self_outputs = self.self( 2025-08-14T21:39:09.9043288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9043418Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9043810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9043971Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9043975Z 2025-08-14T21:39:09.9044085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9044480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9044556Z layer_outputs = layer_module( 2025-08-14T21:39:09.9044811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9044888Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9045163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9045247Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9045522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9045595Z self_outputs = self.self( 2025-08-14T21:39:09.9045870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.9046052Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.9046055Z 2025-08-14T21:39:09.9046162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9046511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9046585Z layer_outputs = layer_module( 2025-08-14T21:39:09.9046807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9046881Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9047164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9047237Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9047512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.9047628Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.9047907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.9047996Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9048044Z 2025-08-14T21:39:09.9048146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9048492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9048571Z layer_outputs = layer_module( 2025-08-14T21:39:09.9048788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9048871Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9049149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9049232Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9049535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9049612Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9049892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9050008Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9050287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.9050373Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9050377Z 2025-08-14T21:39:09.9050475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9050818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9050900Z layer_outputs = layer_module( 2025-08-14T21:39:09.9051117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9051203Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9051483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9051563Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9051827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9051903Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9052189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9052297Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9052578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.9052696Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.9052909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.9052980Z return self.act(input) 2025-08-14T21:39:09.9052984Z 2025-08-14T21:39:09.9053089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9053435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9053514Z layer_outputs = layer_module( 2025-08-14T21:39:09.9053730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9053806Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9054090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9054172Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9054472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9054547Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9054827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.9054955Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.9055237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.9055317Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9055328Z 2025-08-14T21:39:09.9055426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9055801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9055884Z layer_outputs = layer_module( 2025-08-14T21:39:09.9056101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9056177Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9056460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9056535Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9056818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9056888Z self_outputs = self.self( 2025-08-14T21:39:09.9057165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.9057255Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.9057261Z 2025-08-14T21:39:09.9057362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9057712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9057784Z layer_outputs = layer_module( 2025-08-14T21:39:09.9058003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9058085Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9058345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9058417Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9058693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9058763Z self_outputs = self.self( 2025-08-14T21:39:09.9059039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9059137Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9059463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9059647Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9059650Z 2025-08-14T21:39:09.9059747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9060091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9060161Z layer_outputs = layer_module( 2025-08-14T21:39:09.9060371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9060487Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9060759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9060839Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9061113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9061181Z self_outputs = self.self( 2025-08-14T21:39:09.9061458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.9061538Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.9061574Z 2025-08-14T21:39:09.9061676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9062028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9062099Z layer_outputs = layer_module( 2025-08-14T21:39:09.9062321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9062396Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9062670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9062749Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9063028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9063103Z self_outputs = self.self( 2025-08-14T21:39:09.9063372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9063472Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9063802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9063977Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9063981Z 2025-08-14T21:39:09.9064084Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9064418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9064485Z layer_outputs = layer_module( 2025-08-14T21:39:09.9064705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9064778Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9065051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9065128Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9065394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9065469Z self_outputs = self.self( 2025-08-14T21:39:09.9065743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9065841Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9066186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9066364Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9066403Z 2025-08-14T21:39:09.9066512Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9066861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9066931Z layer_outputs = layer_module( 2025-08-14T21:39:09.9067153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9067227Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9067507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9067577Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9067876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9067951Z self_outputs = self.self( 2025-08-14T21:39:09.9068213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9068306Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9068627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9068794Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9068798Z 2025-08-14T21:39:09.9068883Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9068960Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9069034Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9069120Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9069218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9069560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9069631Z layer_outputs = layer_module( 2025-08-14T21:39:09.9069840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9069922Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9070192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9070265Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9070554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9070625Z self_outputs = self.self( 2025-08-14T21:39:09.9070904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.9071015Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9071347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9071505Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.9071820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.9071975Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.9071979Z 2025-08-14T21:39:09.9072059Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9072166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9072541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9072649Z layer_outputs = layer_module( 2025-08-14T21:39:09.9072883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9072965Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9073266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9073349Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9073636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9073709Z self_outputs = self.self( 2025-08-14T21:39:09.9074036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.9074116Z attn_scores += diagonal_mask 2025-08-14T21:39:09.9074123Z 2025-08-14T21:39:09.9074236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9074609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9074684Z layer_outputs = layer_module( 2025-08-14T21:39:09.9074924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9075005Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9075322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9075402Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9075710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9075795Z self_outputs = self.self( 2025-08-14T21:39:09.9076176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.9076270Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.9076283Z 2025-08-14T21:39:09.9076392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9076768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9076856Z layer_outputs = layer_module( 2025-08-14T21:39:09.9077091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9077176Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9077495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9077576Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9077891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9077964Z self_outputs = self.self( 2025-08-14T21:39:09.9078265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.9078360Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.9078364Z 2025-08-14T21:39:09.9078469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9078849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9078927Z layer_outputs = layer_module( 2025-08-14T21:39:09.9079157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9079284Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9079586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9079661Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9080031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9080105Z self_outputs = self.self( 2025-08-14T21:39:09.9080412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9080534Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9080947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9081140Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.9081341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9081451Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9081454Z 2025-08-14T21:39:09.9081559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9081936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9082016Z layer_outputs = layer_module( 2025-08-14T21:39:09.9082254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9082345Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9082648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9082728Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9083032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9083103Z self_outputs = self.self( 2025-08-14T21:39:09.9083393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9083521Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9083905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9084057Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.9084388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.9084487Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.9084697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9084798Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9084801Z 2025-08-14T21:39:09.9084915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9085287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9085361Z layer_outputs = layer_module( 2025-08-14T21:39:09.9085600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9085682Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9085990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9086115Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9086408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9086489Z self_outputs = self.self( 2025-08-14T21:39:09.9086780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9086900Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9087289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9087481Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9087485Z 2025-08-14T21:39:09.9087601Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9087978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9088052Z layer_outputs = layer_module( 2025-08-14T21:39:09.9088290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9088372Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9088680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9088754Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9089035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9089110Z self_outputs = self.self( 2025-08-14T21:39:09.9089397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9089526Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9089890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9090047Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9090050Z 2025-08-14T21:39:09.9090164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9090539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9090624Z layer_outputs = layer_module( 2025-08-14T21:39:09.9090857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9090942Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9091251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9091326Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9091603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9091679Z self_outputs = self.self( 2025-08-14T21:39:09.9091953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.9092144Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.9092148Z 2025-08-14T21:39:09.9092251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9092598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9092718Z layer_outputs = layer_module( 2025-08-14T21:39:09.9092935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9093018Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9093308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9093385Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9093685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.9093842Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.9094135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.9094233Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9094237Z 2025-08-14T21:39:09.9094343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9094726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9094801Z layer_outputs = layer_module( 2025-08-14T21:39:09.9095037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9095125Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9095435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9095529Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9095815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9095893Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9096186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9096294Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9096582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.9096662Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9096666Z 2025-08-14T21:39:09.9096766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9097151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9097225Z layer_outputs = layer_module( 2025-08-14T21:39:09.9097466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9097545Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9097848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9097939Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9098217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9098296Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9098619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9098734Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9099039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.9099199Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.9099419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.9099502Z return self.act(input) 2025-08-14T21:39:09.9099506Z 2025-08-14T21:39:09.9099612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9099979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9100052Z layer_outputs = layer_module( 2025-08-14T21:39:09.9100319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9100408Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9100701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9100790Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9101066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9101140Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9101429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.9101551Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.9101825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.9101915Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9101918Z 2025-08-14T21:39:09.9102019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9102373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9102444Z layer_outputs = layer_module( 2025-08-14T21:39:09.9102659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9102743Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9103021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9103105Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9103380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9103452Z self_outputs = self.self( 2025-08-14T21:39:09.9103734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.9103818Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.9103821Z 2025-08-14T21:39:09.9103921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9104273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9104342Z layer_outputs = layer_module( 2025-08-14T21:39:09.9104569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9104651Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9104946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9105035Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9105325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9105442Z self_outputs = self.self( 2025-08-14T21:39:09.9105741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9105849Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9106219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9106411Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9106416Z 2025-08-14T21:39:09.9106529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9106934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9107014Z layer_outputs = layer_module( 2025-08-14T21:39:09.9107253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9107335Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9107625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9107712Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9108002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9108080Z self_outputs = self.self( 2025-08-14T21:39:09.9108376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.9108463Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.9108469Z 2025-08-14T21:39:09.9108578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9109100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9109190Z layer_outputs = layer_module( 2025-08-14T21:39:09.9109420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9109500Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9109805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9109883Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9110190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9110270Z self_outputs = self.self( 2025-08-14T21:39:09.9110560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9110673Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9111029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9111219Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9111223Z 2025-08-14T21:39:09.9111340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9111722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9111804Z layer_outputs = layer_module( 2025-08-14T21:39:09.9112035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9112185Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9112498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9112579Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9112886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9112959Z self_outputs = self.self( 2025-08-14T21:39:09.9113259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9113377Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9113792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9113999Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9114003Z 2025-08-14T21:39:09.9114113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9114491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9114576Z layer_outputs = layer_module( 2025-08-14T21:39:09.9114815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9114900Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9115211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9115293Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9115604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9115678Z self_outputs = self.self( 2025-08-14T21:39:09.9116025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9116150Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9116519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9116719Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9116723Z 2025-08-14T21:39:09.9116812Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9116903Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9116993Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9117076Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9117188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9117577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9117652Z layer_outputs = layer_module( 2025-08-14T21:39:09.9117891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9117973Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9118266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9118355Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9118652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9118732Z self_outputs = self.self( 2025-08-14T21:39:09.9119076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.9119193Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9119550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9119699Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.9120035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.9120199Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.9120235Z 2025-08-14T21:39:09.9120320Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9120436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9120804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9120878Z layer_outputs = layer_module( 2025-08-14T21:39:09.9121117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9121202Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9121504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9121584Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9121887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9121970Z self_outputs = self.self( 2025-08-14T21:39:09.9122268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.9122355Z attn_scores += diagonal_mask 2025-08-14T21:39:09.9122358Z 2025-08-14T21:39:09.9122459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9122800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9122879Z layer_outputs = layer_module( 2025-08-14T21:39:09.9123095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9123174Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9123460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9123536Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9123817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9123889Z self_outputs = self.self( 2025-08-14T21:39:09.9124164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.9124253Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.9124257Z 2025-08-14T21:39:09.9124359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9124707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9124777Z layer_outputs = layer_module( 2025-08-14T21:39:09.9124995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9125079Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9125396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9125470Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9125754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9125822Z self_outputs = self.self( 2025-08-14T21:39:09.9126102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.9126184Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.9126188Z 2025-08-14T21:39:09.9126287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9126671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9126743Z layer_outputs = layer_module( 2025-08-14T21:39:09.9126970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9127046Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9127322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9127403Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9127686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9127761Z self_outputs = self.self( 2025-08-14T21:39:09.9128037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9128157Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9128511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9128686Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.9128883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9128991Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9128994Z 2025-08-14T21:39:09.9129095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9129454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9129523Z layer_outputs = layer_module( 2025-08-14T21:39:09.9129744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9129828Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9130107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9130188Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9130466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9130533Z self_outputs = self.self( 2025-08-14T21:39:09.9130813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9130927Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9131280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9131418Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.9131765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.9131863Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.9132053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9132151Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9132161Z 2025-08-14T21:39:09.9132260Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9132603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9132680Z layer_outputs = layer_module( 2025-08-14T21:39:09.9132927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9133007Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9133290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9133363Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9133664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9133736Z self_outputs = self.self( 2025-08-14T21:39:09.9134024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9134157Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9134500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9134658Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9134664Z 2025-08-14T21:39:09.9134763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9135106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9135186Z layer_outputs = layer_module( 2025-08-14T21:39:09.9135401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9135482Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9135758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9135834Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9136114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9136184Z self_outputs = self.self( 2025-08-14T21:39:09.9136456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9136576Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9136917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9137073Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9137077Z 2025-08-14T21:39:09.9137176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9137519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9137597Z layer_outputs = layer_module( 2025-08-14T21:39:09.9137854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9137937Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9138216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9138289Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9138575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9138643Z self_outputs = self.self( 2025-08-14T21:39:09.9138921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.9139146Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.9139150Z 2025-08-14T21:39:09.9139255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9139607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9139677Z layer_outputs = layer_module( 2025-08-14T21:39:09.9139893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9139978Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9140253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9140334Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9140611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.9140723Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.9141009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.9141092Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9141095Z 2025-08-14T21:39:09.9141202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9141547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9141627Z layer_outputs = layer_module( 2025-08-14T21:39:09.9141852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9141926Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9142208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9142294Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9142554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9142638Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9142931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9143037Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9143310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.9143386Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9143389Z 2025-08-14T21:39:09.9143493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9143829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9143939Z layer_outputs = layer_module( 2025-08-14T21:39:09.9144158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9144231Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9144508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9144587Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9144837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9144919Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9145223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9145342Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9145625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.9145736Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.9145952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.9146023Z return self.act(input) 2025-08-14T21:39:09.9146027Z 2025-08-14T21:39:09.9146126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9146497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9146567Z layer_outputs = layer_module( 2025-08-14T21:39:09.9146784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9146864Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9147135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9147223Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9147471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9147550Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9147826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.9147945Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.9148225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.9148303Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9148309Z 2025-08-14T21:39:09.9148408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9148753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9148823Z layer_outputs = layer_module( 2025-08-14T21:39:09.9149041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9149117Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9149388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9149471Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9149745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9149865Z self_outputs = self.self( 2025-08-14T21:39:09.9150140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.9150220Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.9150223Z 2025-08-14T21:39:09.9150330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9150674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9150753Z layer_outputs = layer_module( 2025-08-14T21:39:09.9150965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9151041Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9151356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9151435Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9151709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9151787Z self_outputs = self.self( 2025-08-14T21:39:09.9152077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9152191Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9152544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9152739Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9152745Z 2025-08-14T21:39:09.9152857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9153233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9153320Z layer_outputs = layer_module( 2025-08-14T21:39:09.9153550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9153631Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9153931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9154010Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9154320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9154395Z self_outputs = self.self( 2025-08-14T21:39:09.9154683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.9154779Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.9154783Z 2025-08-14T21:39:09.9154888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9155251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9155332Z layer_outputs = layer_module( 2025-08-14T21:39:09.9155564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9155652Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9156017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9156109Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9156415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9156530Z self_outputs = self.self( 2025-08-14T21:39:09.9156837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9156950Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9157319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9157527Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9157532Z 2025-08-14T21:39:09.9157640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9158035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9158110Z layer_outputs = layer_module( 2025-08-14T21:39:09.9158337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9158431Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9158742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9158823Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9159143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9159218Z self_outputs = self.self( 2025-08-14T21:39:09.9159534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9159645Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9160010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9160221Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9160224Z 2025-08-14T21:39:09.9160335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9160721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9160798Z layer_outputs = layer_module( 2025-08-14T21:39:09.9161031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9161123Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9161431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9161520Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9161821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9161895Z self_outputs = self.self( 2025-08-14T21:39:09.9162199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9162305Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9162669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9162867Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9162874Z 2025-08-14T21:39:09.9162963Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9163058Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9163258Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9163341Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9163459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9163839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9163923Z layer_outputs = layer_module( 2025-08-14T21:39:09.9164165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9164249Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9164559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9164684Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9164975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9165056Z self_outputs = self.self( 2025-08-14T21:39:09.9165336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.9165458Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9165823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9165984Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.9166308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.9166456Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.9166460Z 2025-08-14T21:39:09.9166545Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9166646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9166995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9167071Z layer_outputs = layer_module( 2025-08-14T21:39:09.9167289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9167381Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9167649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9167721Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9167998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9168066Z self_outputs = self.self( 2025-08-14T21:39:09.9168332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.9168410Z attn_scores += diagonal_mask 2025-08-14T21:39:09.9168413Z 2025-08-14T21:39:09.9168509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9168850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9168918Z layer_outputs = layer_module( 2025-08-14T21:39:09.9169127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9169211Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9169479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9169593Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9169856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9169924Z self_outputs = self.self( 2025-08-14T21:39:09.9170195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.9170272Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.9170275Z 2025-08-14T21:39:09.9170373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9170713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9170813Z layer_outputs = layer_module( 2025-08-14T21:39:09.9171032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9171110Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9171376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9171456Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9171721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9171795Z self_outputs = self.self( 2025-08-14T21:39:09.9172060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.9172140Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.9172143Z 2025-08-14T21:39:09.9172252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9172589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9172668Z layer_outputs = layer_module( 2025-08-14T21:39:09.9172883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9172962Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9173245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9173323Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9173612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9173691Z self_outputs = self.self( 2025-08-14T21:39:09.9173983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9174117Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9174483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9174666Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.9174877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9174975Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9174979Z 2025-08-14T21:39:09.9175087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9175431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9175500Z layer_outputs = layer_module( 2025-08-14T21:39:09.9175727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9175837Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9176128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9176200Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9176466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9176540Z self_outputs = self.self( 2025-08-14T21:39:09.9176806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9176947Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9177294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9177428Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.9177740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.9177832Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.9178020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9178125Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9178128Z 2025-08-14T21:39:09.9178228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9178582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9178655Z layer_outputs = layer_module( 2025-08-14T21:39:09.9178874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9178958Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9179234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9179316Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9179600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9179667Z self_outputs = self.self( 2025-08-14T21:39:09.9179941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9180056Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9180393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9180553Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9180557Z 2025-08-14T21:39:09.9180656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9181009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9181081Z layer_outputs = layer_module( 2025-08-14T21:39:09.9181295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9181380Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9181660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9181741Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9182054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9182123Z self_outputs = self.self( 2025-08-14T21:39:09.9182403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9182517Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9182871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9183018Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9183022Z 2025-08-14T21:39:09.9183159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9183523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9183598Z layer_outputs = layer_module( 2025-08-14T21:39:09.9183835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9183917Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9184204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9184283Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9184560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9184627Z self_outputs = self.self( 2025-08-14T21:39:09.9184911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.9185095Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.9185102Z 2025-08-14T21:39:09.9185210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9185555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9185624Z layer_outputs = layer_module( 2025-08-14T21:39:09.9185848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9185925Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9186210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9186287Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9186564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.9186684Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.9186958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.9187040Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9187050Z 2025-08-14T21:39:09.9187149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9187490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9187569Z layer_outputs = layer_module( 2025-08-14T21:39:09.9187788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9187865Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9188147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9188270Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9188536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9188616Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9188900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9189019Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9189298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.9189436Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9189440Z 2025-08-14T21:39:09.9189544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9189895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9189973Z layer_outputs = layer_module( 2025-08-14T21:39:09.9190190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9190265Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9190558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9190638Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9190911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9190987Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9191274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9191393Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9191676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.9191796Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.9192022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.9192098Z return self.act(input) 2025-08-14T21:39:09.9192101Z 2025-08-14T21:39:09.9192215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9192596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9192678Z layer_outputs = layer_module( 2025-08-14T21:39:09.9192916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9192998Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9193307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9193392Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9193669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9193757Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9194069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.9194209Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.9194510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.9194634Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9194638Z 2025-08-14T21:39:09.9194753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9195119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9195200Z layer_outputs = layer_module( 2025-08-14T21:39:09.9195427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9195509Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9195844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9195996Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9196298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9196383Z self_outputs = self.self( 2025-08-14T21:39:09.9196672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:39:09.9196768Z query_vectors = self.query(hidden_states) 2025-08-14T21:39:09.9196772Z 2025-08-14T21:39:09.9196878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9197252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9197330Z layer_outputs = layer_module( 2025-08-14T21:39:09.9197551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9197636Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9197922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9197999Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9198297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9198365Z self_outputs = self.self( 2025-08-14T21:39:09.9198646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9198745Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9199084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9199277Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9199283Z 2025-08-14T21:39:09.9199383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9199747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9199816Z layer_outputs = layer_module( 2025-08-14T21:39:09.9200029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9200112Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9200394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9200468Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9200758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9200827Z self_outputs = self.self( 2025-08-14T21:39:09.9201149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:39:09.9201227Z key_vectors = self.key(hidden_states) 2025-08-14T21:39:09.9201231Z 2025-08-14T21:39:09.9201331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9201680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9201749Z layer_outputs = layer_module( 2025-08-14T21:39:09.9201970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9202043Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9202351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9202437Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9202713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9202782Z self_outputs = self.self( 2025-08-14T21:39:09.9203064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9203163Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9203508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9203689Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9203693Z 2025-08-14T21:39:09.9203796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9204168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9204246Z layer_outputs = layer_module( 2025-08-14T21:39:09.9204491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9204567Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9204843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9204927Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9205204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9205280Z self_outputs = self.self( 2025-08-14T21:39:09.9205559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9205663Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9206005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9206184Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9206187Z 2025-08-14T21:39:09.9206293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9206639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9206710Z layer_outputs = layer_module( 2025-08-14T21:39:09.9206937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9207015Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9207337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9207420Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9207695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9207771Z self_outputs = self.self( 2025-08-14T21:39:09.9208044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:39:09.9208142Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9208537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9208851Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:39:09.9208861Z 2025-08-14T21:39:09.9208955Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9209037Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9209116Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9209201Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9209302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9209648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9209733Z layer_outputs = layer_module( 2025-08-14T21:39:09.9209950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9210038Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9210320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9210398Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9210684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9210756Z self_outputs = self.self( 2025-08-14T21:39:09.9211044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:39:09.9211151Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:39:09.9211486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:39:09.9211635Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:39:09.9211956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:39:09.9212113Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:39:09.9212117Z 2025-08-14T21:39:09.9212194Z cudagraph partition due to non gpu ops 2025-08-14T21:39:09.9212295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9212652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9212721Z layer_outputs = layer_module( 2025-08-14T21:39:09.9212932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9213012Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9213283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9213365Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9213708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9213774Z self_outputs = self.self( 2025-08-14T21:39:09.9214046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:39:09.9214118Z attn_scores += diagonal_mask 2025-08-14T21:39:09.9214121Z 2025-08-14T21:39:09.9214228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9214570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9214639Z layer_outputs = layer_module( 2025-08-14T21:39:09.9214911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9214989Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9215259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9215339Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9215616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9215692Z self_outputs = self.self( 2025-08-14T21:39:09.9215971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:39:09.9216050Z attn_probs = nn.functional.softmax( 2025-08-14T21:39:09.9216053Z 2025-08-14T21:39:09.9216160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9216502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9216582Z layer_outputs = layer_module( 2025-08-14T21:39:09.9216799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9216875Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9217152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9217225Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9217498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9217572Z self_outputs = self.self( 2025-08-14T21:39:09.9217844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:39:09.9217936Z value_vectors = self.value(hidden_states) 2025-08-14T21:39:09.9217940Z 2025-08-14T21:39:09.9218039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9218385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9218461Z layer_outputs = layer_module( 2025-08-14T21:39:09.9218679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9218762Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9219040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9219111Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9219396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9219464Z self_outputs = self.self( 2025-08-14T21:39:09.9219744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9219897Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9220239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9220415Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:39:09.9220604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9220700Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9220711Z 2025-08-14T21:39:09.9220809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9221181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9221263Z layer_outputs = layer_module( 2025-08-14T21:39:09.9221481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9221556Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9221841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9221913Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9222199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9222267Z self_outputs = self.self( 2025-08-14T21:39:09.9222542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9222666Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9223016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9223160Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:39:09.9223477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:39:09.9223567Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:39:09.9223766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:39:09.9223864Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:39:09.9223868Z 2025-08-14T21:39:09.9223971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9224324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9224398Z layer_outputs = layer_module( 2025-08-14T21:39:09.9224623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9224700Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9224982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9225062Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9225343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9225420Z self_outputs = self.self( 2025-08-14T21:39:09.9225699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9225817Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9226230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9226401Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9226405Z 2025-08-14T21:39:09.9226513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9226860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9226931Z layer_outputs = layer_module( 2025-08-14T21:39:09.9227155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9227260Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9227548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9227625Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9227900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9227976Z self_outputs = self.self( 2025-08-14T21:39:09.9228250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:39:09.9228361Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:39:09.9228714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:39:09.9228864Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:39:09.9228868Z 2025-08-14T21:39:09.9228977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9229336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9229407Z layer_outputs = layer_module( 2025-08-14T21:39:09.9229633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9229709Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9229991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9230066Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9230369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:39:09.9230451Z self_outputs = self.self( 2025-08-14T21:39:09.9230749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:39:09.9230954Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:39:09.9230958Z 2025-08-14T21:39:09.9231065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9231437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9231520Z layer_outputs = layer_module( 2025-08-14T21:39:09.9231748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9231829Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9232130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:39:09.9232206Z self_attn_outputs = self.attention( 2025-08-14T21:39:09.9232524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:39:09.9232639Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:39:09.9232944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:39:09.9233039Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9233043Z 2025-08-14T21:39:09.9233148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9233526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9233632Z layer_outputs = layer_module( 2025-08-14T21:39:09.9233863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9233952Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9234256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9234348Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9234621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9234700Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9235005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9235119Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9235410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:39:09.9235502Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9235509Z 2025-08-14T21:39:09.9235614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9236052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9236137Z layer_outputs = layer_module( 2025-08-14T21:39:09.9236377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9236465Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9236781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9236874Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9237171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9237254Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9237562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:39:09.9237676Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:39:09.9237979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:39:09.9238097Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:39:09.9238325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:09.9238408Z return self.act(input) 2025-08-14T21:39:09.9238412Z 2025-08-14T21:39:09.9238521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:09.9238892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:39:09.9239023Z layer_outputs = layer_module( 2025-08-14T21:39:09.9239250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:09.9239339Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:09.9239642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:39:09.9239729Z layer_output = apply_chunking_to_forward( 2025-08-14T21:39:09.9240005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:39:09.9240082Z return forward_fn(*input_tensors) 2025-08-14T21:39:09.9240419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:39:09.9240551Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:39:09.9240843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:39:09.9240938Z hidden_states = self.dense(hidden_states) 2025-08-14T21:39:09.9240942Z 2025-08-14T21:40:20.2396701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:20.2397569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-08-14T21:40:20.2403993Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:40:20.2404671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1333, in forward 2025-08-14T21:40:20.2405193Z x = self.dense(features) 2025-08-14T21:40:20.2405347Z 2025-08-14T21:40:20.2405472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:20.2406077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-08-14T21:40:20.2406658Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:40:20.2407158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1338, in forward 2025-08-14T21:40:20.2407615Z x = self.decoder(x) 2025-08-14T21:40:20.2407735Z 2025-08-14T21:40:20.2407858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:20.2408415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1723, in torch_dynamo_resume_in_forward_at_1703 2025-08-14T21:40:20.2409430Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:40:20.2409710Z 2025-08-14T21:40:21.7717577Z Compilation time (from dynamo_timed): 102.212985845 2025-08-14T21:40:21.7946823Z pass 2025-08-14T21:40:21.7947176Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:21.7948012Z TIMING: gc:0.00929 entire_frame_compile:102.21299 _recursive_pre_grad_passes:0.02142 _recursive_joint_graph_passes:0.99463 _recursive_post_grad_passes:1.87997 async_compile.wait:2.83623 code_gen:79.76851 inductor_compile:87.14611 backend_compile:96.96318 total_wall_time:102.21299 2025-08-14T21:40:21.7948989Z STATS: call_* op count: 1787 | FakeTensorMode.__torch_dispatch__:56377 | FakeTensor.__torch_dispatch__:16842 | ProxyTorchDispatchMode.__torch_dispatch__:17446 2025-08-14T21:40:21.7949531Z Dynamo produced 4 graphs covering 1787 ops with 4 graph breaks (1 unique) 2025-08-14T21:40:28.0024199Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:40:28.0025527Z from pkg_resources import resource_filename 2025-08-14T21:40:28.5912331Z 2025-08-14T21:40:31.5191761Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:40:31.5192031Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:40:31.5207663Z cpu eval BartForCausalLM 2025-08-14T21:40:33.1667390Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:33.8411012Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:34.5117005Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:42.0310630Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0311613Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0312460Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0315540Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0316003Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0316348Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0316586Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0316874Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0317138Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0317979Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0321847Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0322254Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0325616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0328538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0329058Z return mod(**inputs) 2025-08-14T21:40:42.0334644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0340146Z outputs = self.model.decoder( 2025-08-14T21:40:42.0345539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0346042Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0346432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0346833Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0347260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0347700Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0348138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0348655Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0348878Z 2025-08-14T21:40:42.0349013Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0349405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0349771Z return mod(**inputs) 2025-08-14T21:40:42.0350170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0350597Z outputs = self.model.decoder( 2025-08-14T21:40:42.0351017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0351436Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0351827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0352236Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0352663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0353385Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0353816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0354248Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0354415Z 2025-08-14T21:40:42.0354537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0354950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0355324Z return mod(**inputs) 2025-08-14T21:40:42.0355732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0356381Z outputs = self.model.decoder( 2025-08-14T21:40:42.0356915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0357346Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0357742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0358139Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0358547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0358985Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0359422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0359904Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0360057Z 2025-08-14T21:40:42.0360158Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0360392Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0360628Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0360853Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0361107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0361506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0361865Z return mod(**inputs) 2025-08-14T21:40:42.0362262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0362694Z outputs = self.model.decoder( 2025-08-14T21:40:42.0363389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0363814Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0364195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0364598Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0365015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0365455Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0365896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0366301Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0366747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0367227Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0367418Z 2025-08-14T21:40:42.0367523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0367886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0368209Z return mod(**inputs) 2025-08-14T21:40:42.0368562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0368991Z outputs = self.model.decoder( 2025-08-14T21:40:42.0369367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0369745Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0370084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0370442Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0370821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0371215Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0371640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0372042Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0372482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0372928Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0373097Z 2025-08-14T21:40:42.0373201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0373559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0373876Z return mod(**inputs) 2025-08-14T21:40:42.0374233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0374616Z outputs = self.model.decoder( 2025-08-14T21:40:42.0374994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0375364Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0375712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0376074Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0376453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0376847Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0377243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0377629Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0377766Z 2025-08-14T21:40:42.0377870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0378232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0378554Z return mod(**inputs) 2025-08-14T21:40:42.0378909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0379291Z outputs = self.model.decoder( 2025-08-14T21:40:42.0379668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0380048Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0380383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0380747Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0381117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0381536Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0381704Z 2025-08-14T21:40:42.0381808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0382166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0382523Z return mod(**inputs) 2025-08-14T21:40:42.0382882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0383260Z outputs = self.model.decoder( 2025-08-14T21:40:42.0383632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0384009Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0384345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0384702Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0385114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0385539Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0385917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0386277Z return self.act(input) 2025-08-14T21:40:42.0386385Z 2025-08-14T21:40:42.0386493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0386838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0387161Z return mod(**inputs) 2025-08-14T21:40:42.0387524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0387893Z outputs = self.model.decoder( 2025-08-14T21:40:42.0388249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0388622Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0388968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0389326Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0389699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0390086Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0390221Z 2025-08-14T21:40:42.0390332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0390699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0391039Z return mod(**inputs) 2025-08-14T21:40:42.0391415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0391824Z outputs = self.model.decoder( 2025-08-14T21:40:42.0392214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0392616Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0392986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0393362Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0393762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0394192Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0394618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0395095Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0395321Z 2025-08-14T21:40:42.0395431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0395812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0396468Z return mod(**inputs) 2025-08-14T21:40:42.0396932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0397356Z outputs = self.model.decoder( 2025-08-14T21:40:42.0397827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0398266Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0398664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0399068Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0399471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0399973Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0400401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0400825Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0400968Z 2025-08-14T21:40:42.0401077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0401456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0401801Z return mod(**inputs) 2025-08-14T21:40:42.0402196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0402618Z outputs = self.model.decoder( 2025-08-14T21:40:42.0403045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0403474Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0403867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0404259Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0404665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0405093Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0405505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0405918Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0406072Z 2025-08-14T21:40:42.0406159Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0406387Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0406600Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0406819Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0407069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0407444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0407793Z return mod(**inputs) 2025-08-14T21:40:42.0408156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0408543Z outputs = self.model.decoder( 2025-08-14T21:40:42.0409109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0409494Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0409843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0410196Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0410583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0410996Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0411399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0411868Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0412312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0412798Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0412992Z 2025-08-14T21:40:42.0413110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0413479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0413828Z return mod(**inputs) 2025-08-14T21:40:42.0414207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0414657Z outputs = self.model.decoder( 2025-08-14T21:40:42.0415037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0415423Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0415774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0416127Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0416511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0416915Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0417305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0417705Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0418154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0418610Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0418776Z 2025-08-14T21:40:42.0418880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0419239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0419566Z return mod(**inputs) 2025-08-14T21:40:42.0419923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0420300Z outputs = self.model.decoder( 2025-08-14T21:40:42.0420675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0421054Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0421401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0421764Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0422150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0422560Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0422976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0423391Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0423536Z 2025-08-14T21:40:42.0423653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0424024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0424367Z return mod(**inputs) 2025-08-14T21:40:42.0424742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0425151Z outputs = self.model.decoder( 2025-08-14T21:40:42.0425544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0425981Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0426346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0426726Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0427122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0427577Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0427758Z 2025-08-14T21:40:42.0427876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0428246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0428590Z return mod(**inputs) 2025-08-14T21:40:42.0429987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0430447Z outputs = self.model.decoder( 2025-08-14T21:40:42.0430837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0431241Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0431609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0432011Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0432420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0432869Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0433284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0433657Z return self.act(input) 2025-08-14T21:40:42.0433785Z 2025-08-14T21:40:42.0433896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0434283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0434626Z return mod(**inputs) 2025-08-14T21:40:42.0434996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0435403Z outputs = self.model.decoder( 2025-08-14T21:40:42.0435803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0436303Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0436685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0437084Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0437499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0437881Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0438029Z 2025-08-14T21:40:42.0438135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0438496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0438816Z return mod(**inputs) 2025-08-14T21:40:42.0439175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0439563Z outputs = self.model.decoder( 2025-08-14T21:40:42.0439944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0440318Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0440667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0441029Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0441402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0441852Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0442253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0442711Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0442913Z 2025-08-14T21:40:42.0443018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0443374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0443700Z return mod(**inputs) 2025-08-14T21:40:42.0444093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0444472Z outputs = self.model.decoder( 2025-08-14T21:40:42.0444899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0445289Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0445627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0445988Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0446369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0446773Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0447167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0447559Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0447696Z 2025-08-14T21:40:42.0447808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0448167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0448488Z return mod(**inputs) 2025-08-14T21:40:42.0448847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0449230Z outputs = self.model.decoder( 2025-08-14T21:40:42.0449599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0449980Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0450335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0450686Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0451054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0451452Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0451845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0452222Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0452364Z 2025-08-14T21:40:42.0452442Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0452653Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0452859Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0453055Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0453282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0453629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0453938Z return mod(**inputs) 2025-08-14T21:40:42.0454286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0454662Z outputs = self.model.decoder( 2025-08-14T21:40:42.0455054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0455425Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0455771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0456130Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0456505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0456910Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0457306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0457752Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0458179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0458652Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0458830Z 2025-08-14T21:40:42.0458938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0459287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0459593Z return mod(**inputs) 2025-08-14T21:40:42.0459946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0460323Z outputs = self.model.decoder( 2025-08-14T21:40:42.0460683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0461053Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0461390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0461744Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0462110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0462502Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0462901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0463299Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0463739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0464197Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0464356Z 2025-08-14T21:40:42.0464470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0464820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0465147Z return mod(**inputs) 2025-08-14T21:40:42.0465514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0465900Z outputs = self.model.decoder( 2025-08-14T21:40:42.0466267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0466652Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0466999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0467357Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0467761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0468167Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0468567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0469002Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0469146Z 2025-08-14T21:40:42.0469250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0469611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0469932Z return mod(**inputs) 2025-08-14T21:40:42.0470283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0470665Z outputs = self.model.decoder( 2025-08-14T21:40:42.0471042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0471415Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0471832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0472221Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0472629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0473075Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0473265Z 2025-08-14T21:40:42.0473375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0473754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0474090Z return mod(**inputs) 2025-08-14T21:40:42.0474469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0474883Z outputs = self.model.decoder( 2025-08-14T21:40:42.0475293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0475702Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0476162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0476570Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0476981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0477425Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0477816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0478162Z return self.act(input) 2025-08-14T21:40:42.0478275Z 2025-08-14T21:40:42.0478381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0478746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0479074Z return mod(**inputs) 2025-08-14T21:40:42.0479436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0479821Z outputs = self.model.decoder( 2025-08-14T21:40:42.0480205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0480590Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0480934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0481305Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0481688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0482083Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0482222Z 2025-08-14T21:40:42.0482341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0482694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0483052Z return mod(**inputs) 2025-08-14T21:40:42.0483402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0483818Z outputs = self.model.decoder( 2025-08-14T21:40:42.0484229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0484646Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0485020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0485413Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0485845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0486254Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0486650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0487108Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0487314Z 2025-08-14T21:40:42.0487425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0487778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0488106Z return mod(**inputs) 2025-08-14T21:40:42.0488460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0488844Z outputs = self.model.decoder( 2025-08-14T21:40:42.0489217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0489602Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0489946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0490304Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0490688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0491093Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0491490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0491893Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0492043Z 2025-08-14T21:40:42.0492153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0492534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0492876Z return mod(**inputs) 2025-08-14T21:40:42.0493246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0493653Z outputs = self.model.decoder( 2025-08-14T21:40:42.0494049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0494455Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0494820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0495209Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0495608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0496030Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0496454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0496868Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0497017Z 2025-08-14T21:40:42.0497141Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0497361Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0497586Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0497808Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0498048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0498427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0498874Z return mod(**inputs) 2025-08-14T21:40:42.0499251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0499665Z outputs = self.model.decoder( 2025-08-14T21:40:42.0500100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0500515Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0500886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0501274Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0501679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0502109Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0502536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0502981Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0503473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0504002Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0504202Z 2025-08-14T21:40:42.0504312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0504699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0505048Z return mod(**inputs) 2025-08-14T21:40:42.0505428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0505841Z outputs = self.model.decoder( 2025-08-14T21:40:42.0506242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0506646Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0507020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0507408Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0507816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0508249Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0508862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0509320Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0509803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0510289Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0510478Z 2025-08-14T21:40:42.0510592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0510980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0511324Z return mod(**inputs) 2025-08-14T21:40:42.0511718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0512136Z outputs = self.model.decoder( 2025-08-14T21:40:42.0512613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0513027Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0513403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0513802Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0514215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0514655Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0515093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0515563Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0515714Z 2025-08-14T21:40:42.0515829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0516281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0516648Z return mod(**inputs) 2025-08-14T21:40:42.0517040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0517523Z outputs = self.model.decoder( 2025-08-14T21:40:42.0517939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0518366Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0518740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0519148Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0519576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0520048Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0520236Z 2025-08-14T21:40:42.0520350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0520739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0521094Z return mod(**inputs) 2025-08-14T21:40:42.0521477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0521891Z outputs = self.model.decoder( 2025-08-14T21:40:42.0522303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0522718Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0523090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0523477Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0523862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0524291Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0524682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0525014Z return self.act(input) 2025-08-14T21:40:42.0525120Z 2025-08-14T21:40:42.0525229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0525572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0525885Z return mod(**inputs) 2025-08-14T21:40:42.0526234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0526617Z outputs = self.model.decoder( 2025-08-14T21:40:42.0526985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0527420Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0527774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0528115Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0528484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0528860Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0528995Z 2025-08-14T21:40:42.0529105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0529445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0529762Z return mod(**inputs) 2025-08-14T21:40:42.0530140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0530503Z outputs = self.model.decoder( 2025-08-14T21:40:42.0530866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0531235Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0531569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0531918Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0532289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0532680Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0533070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0533529Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0533739Z 2025-08-14T21:40:42.0533850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0534211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0534520Z return mod(**inputs) 2025-08-14T21:40:42.0534871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0535248Z outputs = self.model.decoder( 2025-08-14T21:40:42.0535610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0535977Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0536314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0536678Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0537049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0537442Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0537834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0538218Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0538351Z 2025-08-14T21:40:42.0538456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0538818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0539155Z return mod(**inputs) 2025-08-14T21:40:42.0539500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0539868Z outputs = self.model.decoder( 2025-08-14T21:40:42.0540247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0540644Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0541010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0541365Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0541740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0542143Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0542527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0542914Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0543052Z 2025-08-14T21:40:42.0543141Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0543381Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0543582Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0543792Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0544040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0544410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0544755Z return mod(**inputs) 2025-08-14T21:40:42.0545143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0545528Z outputs = self.model.decoder( 2025-08-14T21:40:42.0545904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0546286Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0546633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0546994Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0547364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0547766Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0548160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0548558Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0548995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0549473Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0549654Z 2025-08-14T21:40:42.0549759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0550117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0550444Z return mod(**inputs) 2025-08-14T21:40:42.0550808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0551185Z outputs = self.model.decoder( 2025-08-14T21:40:42.0551565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0551947Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0552283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0552650Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0553031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0553436Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0553829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0554232Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0554718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0555198Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0555369Z 2025-08-14T21:40:42.0555479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0555927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0556287Z return mod(**inputs) 2025-08-14T21:40:42.0556660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0557090Z outputs = self.model.decoder( 2025-08-14T21:40:42.0557513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0557904Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0558249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0558617Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0559005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0559409Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0559812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0560205Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0560344Z 2025-08-14T21:40:42.0560455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0560806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0561135Z return mod(**inputs) 2025-08-14T21:40:42.0561492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0561880Z outputs = self.model.decoder( 2025-08-14T21:40:42.0562262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0562652Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0563002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0563356Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0563739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0564173Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0564344Z 2025-08-14T21:40:42.0564460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0564810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0565138Z return mod(**inputs) 2025-08-14T21:40:42.0565499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0565882Z outputs = self.model.decoder( 2025-08-14T21:40:42.0566254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0566636Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0566981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0567335Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0567726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0568156Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0568543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0568919Z return self.act(input) 2025-08-14T21:40:42.0569039Z 2025-08-14T21:40:42.0569146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0569516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0569841Z return mod(**inputs) 2025-08-14T21:40:42.0570213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0570605Z outputs = self.model.decoder( 2025-08-14T21:40:42.0570992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0571389Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0571805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0572194Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0572589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0573001Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0573153Z 2025-08-14T21:40:42.0573262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0573644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0573981Z return mod(**inputs) 2025-08-14T21:40:42.0574355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0574761Z outputs = self.model.decoder( 2025-08-14T21:40:42.0575159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0575558Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0575929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0576321Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0576718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0577152Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0577581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0578058Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0578276Z 2025-08-14T21:40:42.0578388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0578769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0579115Z return mod(**inputs) 2025-08-14T21:40:42.0579488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0579909Z outputs = self.model.decoder( 2025-08-14T21:40:42.0580310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0580731Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0581088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0581477Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0581879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0582301Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0582717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0583183Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0583327Z 2025-08-14T21:40:42.0583446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0583816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0584159Z return mod(**inputs) 2025-08-14T21:40:42.0584544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0584958Z outputs = self.model.decoder( 2025-08-14T21:40:42.0585360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0585774Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0586175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0586564Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0586974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0587407Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0587846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0588286Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0588434Z 2025-08-14T21:40:42.0588518Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0588735Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0588947Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0589161Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0589412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0589795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0590136Z return mod(**inputs) 2025-08-14T21:40:42.0590534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0590956Z outputs = self.model.decoder( 2025-08-14T21:40:42.0591362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0591782Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0592155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0592546Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0592944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0593383Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0593814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0594247Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0594717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0595246Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0595445Z 2025-08-14T21:40:42.0595567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0596045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0596415Z return mod(**inputs) 2025-08-14T21:40:42.0596811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0597251Z outputs = self.model.decoder( 2025-08-14T21:40:42.0597628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0598056Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0598404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0598765Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0599139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0599543Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0599940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0600337Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0600809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0601270Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0601436Z 2025-08-14T21:40:42.0601551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0601902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0602244Z return mod(**inputs) 2025-08-14T21:40:42.0602640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0603024Z outputs = self.model.decoder( 2025-08-14T21:40:42.0603401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0603788Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0604140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0604500Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0604891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0605304Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0605705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0606096Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0606239Z 2025-08-14T21:40:42.0606350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0606736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0607082Z return mod(**inputs) 2025-08-14T21:40:42.0607459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0607869Z outputs = self.model.decoder( 2025-08-14T21:40:42.0608272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0608775Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0609168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0609572Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0609975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0610428Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0610619Z 2025-08-14T21:40:42.0610728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0611109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0611442Z return mod(**inputs) 2025-08-14T21:40:42.0611826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0612232Z outputs = self.model.decoder( 2025-08-14T21:40:42.0612705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0613108Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0613485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0613891Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0614301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0614759Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0615171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0615597Z return self.act(input) 2025-08-14T21:40:42.0615720Z 2025-08-14T21:40:42.0615828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0616203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0616521Z return mod(**inputs) 2025-08-14T21:40:42.0616856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0617228Z outputs = self.model.decoder( 2025-08-14T21:40:42.0617591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0617959Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0618283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0618635Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0619009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0619391Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0619528Z 2025-08-14T21:40:42.0619626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0619979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0620295Z return mod(**inputs) 2025-08-14T21:40:42.0620635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0621006Z outputs = self.model.decoder( 2025-08-14T21:40:42.0621369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0621741Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0622072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0622424Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0622798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0623186Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0623574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0624015Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0624211Z 2025-08-14T21:40:42.0624320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0624660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0624974Z return mod(**inputs) 2025-08-14T21:40:42.0625325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0625702Z outputs = self.model.decoder( 2025-08-14T21:40:42.0626057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0626460Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0626794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0627133Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0627507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0627898Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0628282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0628651Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0628791Z 2025-08-14T21:40:42.0628933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0629284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0629595Z return mod(**inputs) 2025-08-14T21:40:42.0629947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0630351Z outputs = self.model.decoder( 2025-08-14T21:40:42.0630745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0631139Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0631504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0631891Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0632271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0632679Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0633095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0633507Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0633654Z 2025-08-14T21:40:42.0633738Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0633963Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0634178Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0634393Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0634630Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0635003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0635347Z return mod(**inputs) 2025-08-14T21:40:42.0635722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0636198Z outputs = self.model.decoder( 2025-08-14T21:40:42.0636600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0637007Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0637370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0637778Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0638184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0638621Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0639048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0639501Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0639977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0640535Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0640743Z 2025-08-14T21:40:42.0640858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0641247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0641596Z return mod(**inputs) 2025-08-14T21:40:42.0641974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0642387Z outputs = self.model.decoder( 2025-08-14T21:40:42.0642790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0643193Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0643606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0643995Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0644405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0644803Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0645206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0645631Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0646058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0646508Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0646672Z 2025-08-14T21:40:42.0646772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0647133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0647446Z return mod(**inputs) 2025-08-14T21:40:42.0647805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0648193Z outputs = self.model.decoder( 2025-08-14T21:40:42.0648570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0648950Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0649306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0649663Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0650034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0650434Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0650829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0651215Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0651349Z 2025-08-14T21:40:42.0651452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0651800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0652113Z return mod(**inputs) 2025-08-14T21:40:42.0652458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0652839Z outputs = self.model.decoder( 2025-08-14T21:40:42.0653205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0653580Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0653918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0654281Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0654711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0655140Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0655313Z 2025-08-14T21:40:42.0655420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0655783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0656100Z return mod(**inputs) 2025-08-14T21:40:42.0656440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0656814Z outputs = self.model.decoder( 2025-08-14T21:40:42.0657224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0657608Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0657958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0658325Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0658717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0659143Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0659524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0659865Z return self.act(input) 2025-08-14T21:40:42.0659978Z 2025-08-14T21:40:42.0660091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0660443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0660763Z return mod(**inputs) 2025-08-14T21:40:42.0661115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0661502Z outputs = self.model.decoder( 2025-08-14T21:40:42.0661868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0662245Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0662587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0662934Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0663312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0663700Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0663849Z 2025-08-14T21:40:42.0663970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0664366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0664704Z return mod(**inputs) 2025-08-14T21:40:42.0665066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0665449Z outputs = self.model.decoder( 2025-08-14T21:40:42.0665840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0666217Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0666561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0666919Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0667296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0667699Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0668097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0668584Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0668789Z 2025-08-14T21:40:42.0668892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0669259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0669594Z return mod(**inputs) 2025-08-14T21:40:42.0669989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0670415Z outputs = self.model.decoder( 2025-08-14T21:40:42.0670828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0671279Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0671656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0672044Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0672439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0672873Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0673292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0673710Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0673853Z 2025-08-14T21:40:42.0673962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0674345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0674701Z return mod(**inputs) 2025-08-14T21:40:42.0675126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0675544Z outputs = self.model.decoder( 2025-08-14T21:40:42.0676039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0676472Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0676839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0677222Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0677635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0678065Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0678495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0678919Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0679072Z 2025-08-14T21:40:42.0679177Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0679404Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0679628Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0679846Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0680093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0680464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0680806Z return mod(**inputs) 2025-08-14T21:40:42.0681186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0681585Z outputs = self.model.decoder( 2025-08-14T21:40:42.0681986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0682407Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0682774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0683252Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0683659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0684091Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0684507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0684928Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0685375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0685859Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0686074Z 2025-08-14T21:40:42.0686180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0686543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0686865Z return mod(**inputs) 2025-08-14T21:40:42.0687219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0687597Z outputs = self.model.decoder( 2025-08-14T21:40:42.0687973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0688351Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0688686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0689046Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0689429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0689833Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0690229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0690634Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0691078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0691563Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0691734Z 2025-08-14T21:40:42.0691846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0692225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0692571Z return mod(**inputs) 2025-08-14T21:40:42.0692941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0693348Z outputs = self.model.decoder( 2025-08-14T21:40:42.0693733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0694116Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0694458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0694817Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0695203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0695600Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0695999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0696386Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0696521Z 2025-08-14T21:40:42.0696633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0696983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0697358Z return mod(**inputs) 2025-08-14T21:40:42.0697716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0698102Z outputs = self.model.decoder( 2025-08-14T21:40:42.0698474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0698880Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0699246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0699632Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0700094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0700542Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0700732Z 2025-08-14T21:40:42.0700842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0701193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0701523Z return mod(**inputs) 2025-08-14T21:40:42.0701878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0702255Z outputs = self.model.decoder( 2025-08-14T21:40:42.0702644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0703051Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0703415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0703790Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0704194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0704646Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0705048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0705397Z return self.act(input) 2025-08-14T21:40:42.0705522Z 2025-08-14T21:40:42.0705633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0706015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0706347Z return mod(**inputs) 2025-08-14T21:40:42.0706720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0707141Z outputs = self.model.decoder( 2025-08-14T21:40:42.0707545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0707952Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0708324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0708901Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0709311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0709730Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0709883Z 2025-08-14T21:40:42.0709994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0710377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0710732Z return mod(**inputs) 2025-08-14T21:40:42.0711131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0711552Z outputs = self.model.decoder( 2025-08-14T21:40:42.0712036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0712456Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0712837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0713232Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0713638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0714075Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0714508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0715056Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0715273Z 2025-08-14T21:40:42.0715383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0715769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0716212Z return mod(**inputs) 2025-08-14T21:40:42.0716611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0717043Z outputs = self.model.decoder( 2025-08-14T21:40:42.0717452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0717880Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0718252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0718644Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0719068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0719507Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0719947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0720372Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0720522Z 2025-08-14T21:40:42.0720648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0721030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0721385Z return mod(**inputs) 2025-08-14T21:40:42.0721774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0722198Z outputs = self.model.decoder( 2025-08-14T21:40:42.0722603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0723022Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0723406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0723789Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0724202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0724644Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0725076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0725496Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0725655Z 2025-08-14T21:40:42.0725743Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0725981Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0726207Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0726444Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0726740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0727121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0727191Z return mod(**inputs) 2025-08-14T21:40:42.0727456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0727542Z outputs = self.model.decoder( 2025-08-14T21:40:42.0727805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0727880Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0728117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0728232Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0728498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0728605Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0728862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0728974Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0729275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0729424Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0729428Z 2025-08-14T21:40:42.0729537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0729745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0729825Z return mod(**inputs) 2025-08-14T21:40:42.0730084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0730165Z outputs = self.model.decoder( 2025-08-14T21:40:42.0730429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0730504Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0730742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0730825Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0731081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0731190Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0731447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0731561Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0731864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0731979Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0731982Z 2025-08-14T21:40:42.0732095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0732303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0732373Z return mod(**inputs) 2025-08-14T21:40:42.0732640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0732721Z outputs = self.model.decoder( 2025-08-14T21:40:42.0732996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0733072Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0733308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0733453Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0733720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0733824Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0734102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0734191Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0734194Z 2025-08-14T21:40:42.0734316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0734530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0734631Z return mod(**inputs) 2025-08-14T21:40:42.0734917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0734997Z outputs = self.model.decoder( 2025-08-14T21:40:42.0735263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0735337Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0735566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0735656Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0735911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0736036Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0736048Z 2025-08-14T21:40:42.0736159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0736366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0736446Z return mod(**inputs) 2025-08-14T21:40:42.0736707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0736784Z outputs = self.model.decoder( 2025-08-14T21:40:42.0737049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0737123Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0737355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0737437Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0737690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0737823Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0738047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0738124Z return self.act(input) 2025-08-14T21:40:42.0738128Z 2025-08-14T21:40:42.0738247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0738459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0738534Z return mod(**inputs) 2025-08-14T21:40:42.0738800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0738882Z outputs = self.model.decoder( 2025-08-14T21:40:42.0739152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0739239Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0739468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0739556Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0739857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0739952Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0739956Z 2025-08-14T21:40:42.0740062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0740267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0740343Z return mod(**inputs) 2025-08-14T21:40:42.0740603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0740685Z outputs = self.model.decoder( 2025-08-14T21:40:42.0740976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0741052Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0741293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0741375Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0741633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0741743Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0742001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0742171Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0742175Z 2025-08-14T21:40:42.0742281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0742491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0742567Z return mod(**inputs) 2025-08-14T21:40:42.0742829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0742917Z outputs = self.model.decoder( 2025-08-14T21:40:42.0743178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0743254Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0743493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0743576Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0743836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0743947Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0744212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0744305Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0744309Z 2025-08-14T21:40:42.0744413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0744622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0744698Z return mod(**inputs) 2025-08-14T21:40:42.0744960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0745042Z outputs = self.model.decoder( 2025-08-14T21:40:42.0745302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0745378Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0745620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0745700Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0745991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0746101Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0746364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0746464Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0746468Z 2025-08-14T21:40:42.0746555Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0746640Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0746733Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0746814Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0746925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0747192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0747266Z return mod(**inputs) 2025-08-14T21:40:42.0747553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0747630Z outputs = self.model.decoder( 2025-08-14T21:40:42.0747884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0747966Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0748201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0748279Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0748528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0748625Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0748874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0748974Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0749258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0749398Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0749402Z 2025-08-14T21:40:42.0749502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0749713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0749781Z return mod(**inputs) 2025-08-14T21:40:42.0750038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0750120Z outputs = self.model.decoder( 2025-08-14T21:40:42.0750378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0750456Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0750690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0750772Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0751033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0751135Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0751388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0751498Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0751801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0751923Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0751961Z 2025-08-14T21:40:42.0752069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0752272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0752346Z return mod(**inputs) 2025-08-14T21:40:42.0752604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0752681Z outputs = self.model.decoder( 2025-08-14T21:40:42.0752947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0753022Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0753258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0753384Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0753656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0753772Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0754041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0754130Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0754140Z 2025-08-14T21:40:42.0754249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0754463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0754539Z return mod(**inputs) 2025-08-14T21:40:42.0754809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0754886Z outputs = self.model.decoder( 2025-08-14T21:40:42.0755175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0755252Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0755489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0755572Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0755833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0756051Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0756058Z 2025-08-14T21:40:42.0756168Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0756386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0756463Z return mod(**inputs) 2025-08-14T21:40:42.0756735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0756822Z outputs = self.model.decoder( 2025-08-14T21:40:42.0757097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0757174Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0757414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0757492Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0757742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0757860Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0758070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0758149Z return self.act(input) 2025-08-14T21:40:42.0758152Z 2025-08-14T21:40:42.0758254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0758504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0758584Z return mod(**inputs) 2025-08-14T21:40:42.0758852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0758940Z outputs = self.model.decoder( 2025-08-14T21:40:42.0759210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0759285Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0759529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0759613Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0759912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0760009Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0760016Z 2025-08-14T21:40:42.0760125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0760342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0760413Z return mod(**inputs) 2025-08-14T21:40:42.0760676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0760762Z outputs = self.model.decoder( 2025-08-14T21:40:42.0761024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0761110Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0761347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0761432Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0761708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0761818Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0762085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0762257Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0762261Z 2025-08-14T21:40:42.0762370Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0762592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0762662Z return mod(**inputs) 2025-08-14T21:40:42.0762937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0763023Z outputs = self.model.decoder( 2025-08-14T21:40:42.0763295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0763385Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0763622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0763706Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0763976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0764069Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0764307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0764393Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0764396Z 2025-08-14T21:40:42.0764495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0764695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0764799Z return mod(**inputs) 2025-08-14T21:40:42.0765037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0765115Z outputs = self.model.decoder( 2025-08-14T21:40:42.0765349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0765417Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0765633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0765708Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0765979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0766076Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0766314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0766410Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0766414Z 2025-08-14T21:40:42.0766491Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0766576Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0766651Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0766725Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0766830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0767024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0767088Z return mod(**inputs) 2025-08-14T21:40:42.0767354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0767427Z outputs = self.model.decoder( 2025-08-14T21:40:42.0767683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0767759Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0767978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0768064Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0768311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0768407Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0768657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0768764Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0769055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0769188Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0769191Z 2025-08-14T21:40:42.0769288Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0769493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0769556Z return mod(**inputs) 2025-08-14T21:40:42.0769810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0769884Z outputs = self.model.decoder( 2025-08-14T21:40:42.0770141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0770218Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0770433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0770509Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0770781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0770874Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0771118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0771210Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0771496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0771607Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0771610Z 2025-08-14T21:40:42.0771704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0771923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0771988Z return mod(**inputs) 2025-08-14T21:40:42.0772222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0772300Z outputs = self.model.decoder( 2025-08-14T21:40:42.0772570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0772640Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0772857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0772934Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0773201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0773299Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0773539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0773631Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0773635Z 2025-08-14T21:40:42.0773737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0773939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0774004Z return mod(**inputs) 2025-08-14T21:40:42.0774252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0774333Z outputs = self.model.decoder( 2025-08-14T21:40:42.0774585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0774656Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0774880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0774969Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0775215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0775329Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0775333Z 2025-08-14T21:40:42.0775430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0775627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0775690Z return mod(**inputs) 2025-08-14T21:40:42.0775929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0776008Z outputs = self.model.decoder( 2025-08-14T21:40:42.0776252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0776330Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0776548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0776663Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0776910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0777023Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0777238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0777307Z return self.act(input) 2025-08-14T21:40:42.0777312Z 2025-08-14T21:40:42.0777411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0777617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0777716Z return mod(**inputs) 2025-08-14T21:40:42.0777956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0778039Z outputs = self.model.decoder( 2025-08-14T21:40:42.0778276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0778353Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0778564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0778638Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0778879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0778958Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0778962Z 2025-08-14T21:40:42.0779061Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0779257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0779323Z return mod(**inputs) 2025-08-14T21:40:42.0779568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0779638Z outputs = self.model.decoder( 2025-08-14T21:40:42.0779875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0779952Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0780163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0780246Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0780480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0780576Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0780817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:42.0780964Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:42.0780968Z 2025-08-14T21:40:42.0781066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0781263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0781328Z return mod(**inputs) 2025-08-14T21:40:42.0781571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0781641Z outputs = self.model.decoder( 2025-08-14T21:40:42.0781877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0781957Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0782166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0782284Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0782520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0782615Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0782857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:42.0782936Z key_states = self.k_proj(current_states) 2025-08-14T21:40:42.0782940Z 2025-08-14T21:40:42.0783041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0783241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0783306Z return mod(**inputs) 2025-08-14T21:40:42.0783578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0783653Z outputs = self.model.decoder( 2025-08-14T21:40:42.0783891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0783967Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0784183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0784259Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0784513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0784608Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0784856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:42.0784944Z value_states = self.v_proj(current_states) 2025-08-14T21:40:42.0784947Z 2025-08-14T21:40:42.0785024Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0785112Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0785187Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0785270Z cudagraph partition due to non gpu ops 2025-08-14T21:40:42.0785370Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0785562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0785635Z return mod(**inputs) 2025-08-14T21:40:42.0785895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0785965Z outputs = self.model.decoder( 2025-08-14T21:40:42.0786208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0786278Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0786498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0786576Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0786812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0786914Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0787149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0787243Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0787532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:42.0787663Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:42.0787670Z 2025-08-14T21:40:42.0787774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0787966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0788067Z return mod(**inputs) 2025-08-14T21:40:42.0788313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0788386Z outputs = self.model.decoder( 2025-08-14T21:40:42.0788628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0788697Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0788906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0788990Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0789250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0789344Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0789590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:42.0789684Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:42.0789969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:42.0790077Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:42.0790080Z 2025-08-14T21:40:42.0790178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0790375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0790438Z return mod(**inputs) 2025-08-14T21:40:42.0790687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0790759Z outputs = self.model.decoder( 2025-08-14T21:40:42.0790997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0791076Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0791285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0791360Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0791602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:42.0791693Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:42.0791933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:42.0792012Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:42.0792018Z 2025-08-14T21:40:42.0792116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0792316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0792384Z return mod(**inputs) 2025-08-14T21:40:42.0792630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0792710Z outputs = self.model.decoder( 2025-08-14T21:40:42.0792956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0793036Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0793272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0793353Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0793620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0793743Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0793775Z 2025-08-14T21:40:42.0793888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0794093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0794162Z return mod(**inputs) 2025-08-14T21:40:42.0794432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0794509Z outputs = self.model.decoder( 2025-08-14T21:40:42.0794774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0794856Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0795123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0795215Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0795469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:42.0795595Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:42.0795823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:42.0795978Z return self.act(input) 2025-08-14T21:40:42.0795984Z 2025-08-14T21:40:42.0796106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0796340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0796410Z return mod(**inputs) 2025-08-14T21:40:42.0796689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:40:42.0796773Z outputs = self.model.decoder( 2025-08-14T21:40:42.0797059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:42.0797148Z layer_outputs = decoder_layer( 2025-08-14T21:40:42.0797391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:42.0797482Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:42.0797755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:42.0797842Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:42.0797846Z 2025-08-14T21:40:42.0797960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0798204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0798272Z return mod(**inputs) 2025-08-14T21:40:42.0798550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1917, in forward 2025-08-14T21:40:42.0798634Z logits = self.lm_head(outputs[0]) 2025-08-14T21:40:42.0798640Z 2025-08-14T21:40:42.0798753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:42.0798975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:42.0799045Z return mod(**inputs) 2025-08-14T21:40:42.0799333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1923, in forward 2025-08-14T21:40:42.0799491Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:40:42.0799495Z 2025-08-14T21:40:51.7240376Z Compilation time (from dynamo_timed): 14.927714014 2025-08-14T21:40:51.7528482Z pass 2025-08-14T21:40:51.7529122Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:51.7530064Z TIMING: _recursive_pre_grad_passes:0.00747 _recursive_joint_graph_passes:0.64579 _recursive_post_grad_passes:0.08347 async_compile.wait:0.77368 code_gen:8.1792 inductor_compile:9.41364 backend_compile:12.67118 gc:0.00101 entire_frame_compile:14.92771 total_wall_time:14.92771 2025-08-14T21:40:51.7533352Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:13198 | FakeTensor.__torch_dispatch__:4868 | ProxyTorchDispatchMode.__torch_dispatch__:4813 2025-08-14T21:40:51.7533967Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2025-08-14T21:40:56.9833567Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:40:56.9834524Z from pkg_resources import resource_filename 2025-08-14T21:40:57.5528563Z 2025-08-14T21:41:02.8132599Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:41:02.8132940Z loading model: 0it [00:05, ?it/s] 2025-08-14T21:41:02.8155085Z cpu eval BartForConditionalGeneration 2025-08-14T21:41:06.2596245Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:07.5202651Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:08.8173141Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:25.6543153Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6547898Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6550142Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6550387Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6550609Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6550834Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6553659Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6553951Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6564787Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6565101Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6565337Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6565576Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6565947Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6566369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6566738Z return mod(**inputs) 2025-08-14T21:41:25.6567177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6567607Z outputs = self.model( 2025-08-14T21:41:25.6568004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6568441Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6568862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6569280Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6569647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6570022Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6570416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6570820Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6571224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6571691Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6571904Z 2025-08-14T21:41:25.6572024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6572393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6573043Z return mod(**inputs) 2025-08-14T21:41:25.6573431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6573883Z outputs = self.model( 2025-08-14T21:41:25.6574248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6574665Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6575089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6575461Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6575811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6576292Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6576709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6577122Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6577531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6577958Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6578103Z 2025-08-14T21:41:25.6578225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6578604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6578972Z return mod(**inputs) 2025-08-14T21:41:25.6579358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6579761Z outputs = self.model( 2025-08-14T21:41:25.6580143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6580558Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6580954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6581360Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6581731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6582119Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6582513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6582947Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6583369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6583785Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6583933Z 2025-08-14T21:41:25.6584024Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6584254Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6584476Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6584689Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6584941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6585328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6585680Z return mod(**inputs) 2025-08-14T21:41:25.6586054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6586455Z outputs = self.model( 2025-08-14T21:41:25.6586835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6587239Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6587638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6588088Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6588461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6588837Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6589242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6589663Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6590074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6590517Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6591051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6591581Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6591788Z 2025-08-14T21:41:25.6591902Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6592303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6592657Z return mod(**inputs) 2025-08-14T21:41:25.6593048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6593460Z outputs = self.model( 2025-08-14T21:41:25.6593852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6594283Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6594690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6595109Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6595492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6596106Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6596529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6596967Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6597405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6597852Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6598339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6598844Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6599020Z 2025-08-14T21:41:25.6599143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6599535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6599894Z return mod(**inputs) 2025-08-14T21:41:25.6600285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6600692Z outputs = self.model( 2025-08-14T21:41:25.6601087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6601501Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6601906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6602314Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6602810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6603205Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6603707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6604218Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6604644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.6605065Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.6605219Z 2025-08-14T21:41:25.6605332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6605725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6606075Z return mod(**inputs) 2025-08-14T21:41:25.6606502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6606914Z outputs = self.model( 2025-08-14T21:41:25.6607320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6607721Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6608119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6608519Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6609074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6609462Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6609866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6610321Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6610508Z 2025-08-14T21:41:25.6610621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6611006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6611357Z return mod(**inputs) 2025-08-14T21:41:25.6611740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6612131Z outputs = self.model( 2025-08-14T21:41:25.6612510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6612916Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6613307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6613709Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6614079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6614458Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6614850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6615298Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6615709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.6616062Z return self.act(input) 2025-08-14T21:41:25.6616188Z 2025-08-14T21:41:25.6616298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6616677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6617019Z return mod(**inputs) 2025-08-14T21:41:25.6617388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6617793Z outputs = self.model( 2025-08-14T21:41:25.6618169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6618655Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6619059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6619467Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6619837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6620226Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6620629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.6621050Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.6621196Z 2025-08-14T21:41:25.6621313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6621743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6622092Z return mod(**inputs) 2025-08-14T21:41:25.6622463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6622868Z outputs = self.model( 2025-08-14T21:41:25.6623248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6623665Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6624062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6624451Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6624814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6625203Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6625594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6626015Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6626429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6626922Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6627140Z 2025-08-14T21:41:25.6627249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6627626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6627967Z return mod(**inputs) 2025-08-14T21:41:25.6628346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6628749Z outputs = self.model( 2025-08-14T21:41:25.6629132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6629545Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6629945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6630355Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6630729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6631113Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6631505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6631926Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6632351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6632757Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6632898Z 2025-08-14T21:41:25.6633157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6633608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6633960Z return mod(**inputs) 2025-08-14T21:41:25.6634339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6634753Z outputs = self.model( 2025-08-14T21:41:25.6635142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6635563Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6636028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6636451Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6636877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6637266Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6637684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6638114Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6638547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6638970Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6639133Z 2025-08-14T21:41:25.6639224Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6639455Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6639678Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6639904Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6640168Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6640578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6640942Z return mod(**inputs) 2025-08-14T21:41:25.6641342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6641783Z outputs = self.model( 2025-08-14T21:41:25.6642178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6642635Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6643056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6643483Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6643862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6644269Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6644695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6645135Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6645583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6646057Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6646558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6647104Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6647313Z 2025-08-14T21:41:25.6647426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6647820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6648164Z return mod(**inputs) 2025-08-14T21:41:25.6648532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6648979Z outputs = self.model( 2025-08-14T21:41:25.6649357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6649758Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6650163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6650565Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6650936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6651317Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6651773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6652204Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6652630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6653058Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6653524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6654004Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6654174Z 2025-08-14T21:41:25.6654281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6654661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6655012Z return mod(**inputs) 2025-08-14T21:41:25.6655394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6655782Z outputs = self.model( 2025-08-14T21:41:25.6656163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6656568Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6656969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6657384Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6657771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6658183Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6658586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6659016Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6659450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.6659861Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.6660012Z 2025-08-14T21:41:25.6660122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6660505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6660853Z return mod(**inputs) 2025-08-14T21:41:25.6661223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6661625Z outputs = self.model( 2025-08-14T21:41:25.6662012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6662418Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6662804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6663201Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6663568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6663985Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6664393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6664844Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6665028Z 2025-08-14T21:41:25.6665147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6665630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6666000Z return mod(**inputs) 2025-08-14T21:41:25.6666380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6666822Z outputs = self.model( 2025-08-14T21:41:25.6667198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6667610Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6668010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6668401Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6668768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6669155Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6669559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6670010Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6670527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.6670914Z return self.act(input) 2025-08-14T21:41:25.6671034Z 2025-08-14T21:41:25.6671163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6671553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6671912Z return mod(**inputs) 2025-08-14T21:41:25.6672309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6672726Z outputs = self.model( 2025-08-14T21:41:25.6673122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6673542Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6673962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6674371Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6674754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6675148Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6675558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.6676047Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.6676211Z 2025-08-14T21:41:25.6676331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6676736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6677094Z return mod(**inputs) 2025-08-14T21:41:25.6677493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6677922Z outputs = self.model( 2025-08-14T21:41:25.6678321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6678760Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6679240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6679662Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6680042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6680440Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6680864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6681310Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6681748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6682310Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6682548Z 2025-08-14T21:41:25.6682672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6683064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6683412Z return mod(**inputs) 2025-08-14T21:41:25.6683791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6684191Z outputs = self.model( 2025-08-14T21:41:25.6684562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6684967Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6685363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6685759Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6686133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6686516Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6686916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6687342Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6687764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6688180Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6688321Z 2025-08-14T21:41:25.6688440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6688817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6689159Z return mod(**inputs) 2025-08-14T21:41:25.6689538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6689932Z outputs = self.model( 2025-08-14T21:41:25.6690316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6690721Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6691113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6691534Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6691911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6692304Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6692690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6693085Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6693511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6694861Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6695011Z 2025-08-14T21:41:25.6695100Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6695330Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6695556Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6695765Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6695997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6696385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6696737Z return mod(**inputs) 2025-08-14T21:41:25.6697092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6697472Z outputs = self.model( 2025-08-14T21:41:25.6697869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6698256Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6698631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6699016Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6699367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6699722Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6700107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6700510Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6700908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6701315Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6701784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6702304Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6702496Z 2025-08-14T21:41:25.6702610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6702975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6703307Z return mod(**inputs) 2025-08-14T21:41:25.6703666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6704043Z outputs = self.model( 2025-08-14T21:41:25.6704408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6704811Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6705214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6705618Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6705971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6706339Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6706722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6707125Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6707522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6707932Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6708375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6709039Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6709284Z 2025-08-14T21:41:25.6709398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6709762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6710084Z return mod(**inputs) 2025-08-14T21:41:25.6710457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6710847Z outputs = self.model( 2025-08-14T21:41:25.6711210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6711623Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6712100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6712502Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6712865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6713251Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6713656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6714064Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6714481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.6714890Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.6715036Z 2025-08-14T21:41:25.6715154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6715527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6715917Z return mod(**inputs) 2025-08-14T21:41:25.6716323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6716747Z outputs = self.model( 2025-08-14T21:41:25.6717129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6717546Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6717909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6718268Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6718607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6718958Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6719333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6719738Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6719912Z 2025-08-14T21:41:25.6720016Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6720369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6720677Z return mod(**inputs) 2025-08-14T21:41:25.6721023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6721393Z outputs = self.model( 2025-08-14T21:41:25.6721752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6722125Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6722496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6722881Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6723237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6723625Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6723994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6724415Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6724797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.6725143Z return self.act(input) 2025-08-14T21:41:25.6725253Z 2025-08-14T21:41:25.6725364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6725719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6726038Z return mod(**inputs) 2025-08-14T21:41:25.6726439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6726808Z outputs = self.model( 2025-08-14T21:41:25.6727151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6727521Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6727884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6728253Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6728584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6728930Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6729306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.6729676Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.6729821Z 2025-08-14T21:41:25.6729924Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6730281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6730596Z return mod(**inputs) 2025-08-14T21:41:25.6730932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6731296Z outputs = self.model( 2025-08-14T21:41:25.6731648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6732012Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6732375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6732741Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6733079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6733419Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6733795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6734183Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6734568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6735014Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6735220Z 2025-08-14T21:41:25.6735333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6735680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6735988Z return mod(**inputs) 2025-08-14T21:41:25.6736338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6736703Z outputs = self.model( 2025-08-14T21:41:25.6737049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6737443Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6737804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6738170Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6738506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6738867Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6739249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6739702Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6740112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6740503Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6740644Z 2025-08-14T21:41:25.6740750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6741107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6741423Z return mod(**inputs) 2025-08-14T21:41:25.6741780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6742163Z outputs = self.model( 2025-08-14T21:41:25.6742520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6742906Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6743289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6743675Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6744028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6744384Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6744769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6745171Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6745575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6745977Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6746123Z 2025-08-14T21:41:25.6746214Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6746428Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6746647Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6746862Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6747099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6747474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6747808Z return mod(**inputs) 2025-08-14T21:41:25.6748170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6748542Z outputs = self.model( 2025-08-14T21:41:25.6748905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6749295Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6749667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6750048Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6750399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6750764Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6751178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6751573Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6751980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6752413Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6752877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6753380Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6753572Z 2025-08-14T21:41:25.6753690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6754095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6754457Z return mod(**inputs) 2025-08-14T21:41:25.6754843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6755253Z outputs = self.model( 2025-08-14T21:41:25.6755624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6756181Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6756599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6757007Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6757393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6757760Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6758148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6758549Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6758951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6759361Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6759810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6760261Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6760432Z 2025-08-14T21:41:25.6761259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6761623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6761948Z return mod(**inputs) 2025-08-14T21:41:25.6762313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6762697Z outputs = self.model( 2025-08-14T21:41:25.6763052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6763441Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6763818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6764196Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6764539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6764902Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6765285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6765681Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6766087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.6766534Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.6766669Z 2025-08-14T21:41:25.6766780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6767134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6767465Z return mod(**inputs) 2025-08-14T21:41:25.6767809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6768172Z outputs = self.model( 2025-08-14T21:41:25.6768517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6768927Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6769297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6769664Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6770000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6770353Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6770735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6771157Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6771337Z 2025-08-14T21:41:25.6771442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6771801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6772131Z return mod(**inputs) 2025-08-14T21:41:25.6772481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6772846Z outputs = self.model( 2025-08-14T21:41:25.6773198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6773562Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6773925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6774297Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6774627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6774986Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6775369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6775798Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6776180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.6776529Z return self.act(input) 2025-08-14T21:41:25.6776640Z 2025-08-14T21:41:25.6776750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6777117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6777427Z return mod(**inputs) 2025-08-14T21:41:25.6777773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6778137Z outputs = self.model( 2025-08-14T21:41:25.6778476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6778844Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6779217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6779583Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6779957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6780324Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6780709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.6781090Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.6781233Z 2025-08-14T21:41:25.6781339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6781700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6782029Z return mod(**inputs) 2025-08-14T21:41:25.6782431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6782798Z outputs = self.model( 2025-08-14T21:41:25.6783153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6783523Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6783881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6784253Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6784602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6784956Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6785340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6785739Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6786136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6786589Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6786803Z 2025-08-14T21:41:25.6786908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6787264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6787583Z return mod(**inputs) 2025-08-14T21:41:25.6787940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6788321Z outputs = self.model( 2025-08-14T21:41:25.6788678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6789054Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6789434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6789816Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6790162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6790520Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6790906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6791309Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6791698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6792094Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6792235Z 2025-08-14T21:41:25.6792342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6792702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6793027Z return mod(**inputs) 2025-08-14T21:41:25.6793386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6793815Z outputs = self.model( 2025-08-14T21:41:25.6794166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6794546Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6794928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6795340Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6795707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6796199Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6796677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6797114Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6797540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6797940Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6798084Z 2025-08-14T21:41:25.6798175Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6798384Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6798596Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6798804Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6799034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6799395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6799722Z return mod(**inputs) 2025-08-14T21:41:25.6800087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6800462Z outputs = self.model( 2025-08-14T21:41:25.6800823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6801210Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6801577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6801955Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6802303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6802663Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6803040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6803443Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6803841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6804249Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6804691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6805179Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6805364Z 2025-08-14T21:41:25.6805474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6805824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6806152Z return mod(**inputs) 2025-08-14T21:41:25.6806511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6806894Z outputs = self.model( 2025-08-14T21:41:25.6807250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6807652Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6808091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6808494Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6809013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6809394Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6809769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6810154Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6810540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6811011Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6811447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6811883Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6812049Z 2025-08-14T21:41:25.6812149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6812500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6812812Z return mod(**inputs) 2025-08-14T21:41:25.6813161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6813530Z outputs = self.model( 2025-08-14T21:41:25.6813892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6814269Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6814647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6815031Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6815381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6815738Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6816108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6816493Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6816867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.6817241Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.6817380Z 2025-08-14T21:41:25.6817481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6817833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6818147Z return mod(**inputs) 2025-08-14T21:41:25.6818508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6818882Z outputs = self.model( 2025-08-14T21:41:25.6819236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6819617Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6819992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6820366Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6820705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6821060Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6821443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6821947Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6822121Z 2025-08-14T21:41:25.6822226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6822587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6822910Z return mod(**inputs) 2025-08-14T21:41:25.6823257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6823635Z outputs = self.model( 2025-08-14T21:41:25.6823996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6824379Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6824790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6825170Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6825520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6825872Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6826255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6826681Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6827065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.6827399Z return self.act(input) 2025-08-14T21:41:25.6827517Z 2025-08-14T21:41:25.6827622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6827982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6828303Z return mod(**inputs) 2025-08-14T21:41:25.6828693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6829099Z outputs = self.model( 2025-08-14T21:41:25.6829481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6829868Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6830241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6830618Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6830958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6831318Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6831701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.6832087Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.6832222Z 2025-08-14T21:41:25.6832328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6832688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6833011Z return mod(**inputs) 2025-08-14T21:41:25.6833366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6833732Z outputs = self.model( 2025-08-14T21:41:25.6834105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6834507Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6834895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6835297Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6835661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6836149Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6836556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6836981Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6837402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6837880Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6838105Z 2025-08-14T21:41:25.6838217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6838604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6839062Z return mod(**inputs) 2025-08-14T21:41:25.6839436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6839836Z outputs = self.model( 2025-08-14T21:41:25.6840216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6840621Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6841010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6841415Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6841793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6842176Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6842599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6843021Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6843448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6843866Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6844017Z 2025-08-14T21:41:25.6844129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6844512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6844860Z return mod(**inputs) 2025-08-14T21:41:25.6845238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6845639Z outputs = self.model( 2025-08-14T21:41:25.6846022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6846420Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6846818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6847220Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6847594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6847971Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6848380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6848766Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6849139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6849520Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6849664Z 2025-08-14T21:41:25.6849744Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6849953Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6850153Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6850404Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6850631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6850975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6851289Z return mod(**inputs) 2025-08-14T21:41:25.6851635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6852000Z outputs = self.model( 2025-08-14T21:41:25.6852339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6852706Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6853108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6853475Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6853814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6854164Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6854532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6854915Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6855328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6855766Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6856230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6856740Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6856947Z 2025-08-14T21:41:25.6857050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6857417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6857724Z return mod(**inputs) 2025-08-14T21:41:25.6858068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6858433Z outputs = self.model( 2025-08-14T21:41:25.6858778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6859142Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6859504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6859875Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6860209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6860561Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6860934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6861311Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6861679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6862067Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6862503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6862955Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6863115Z 2025-08-14T21:41:25.6863220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6863582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6863905Z return mod(**inputs) 2025-08-14T21:41:25.6864301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6864683Z outputs = self.model( 2025-08-14T21:41:25.6865047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6865427Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6865794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6866172Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6866519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6866869Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6867311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6867707Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6868099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.6868476Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.6868619Z 2025-08-14T21:41:25.6868722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6869078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6869393Z return mod(**inputs) 2025-08-14T21:41:25.6869751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6870125Z outputs = self.model( 2025-08-14T21:41:25.6870483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6870857Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6871235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6871611Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6871953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6872303Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6872678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6873103Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6873275Z 2025-08-14T21:41:25.6873377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6873734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6874054Z return mod(**inputs) 2025-08-14T21:41:25.6874409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6874794Z outputs = self.model( 2025-08-14T21:41:25.6875172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6875562Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6876012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6876405Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6876779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6877171Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6877556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6877986Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6878420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.6878758Z return self.act(input) 2025-08-14T21:41:25.6878869Z 2025-08-14T21:41:25.6878972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6879329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6879649Z return mod(**inputs) 2025-08-14T21:41:25.6879997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6880371Z outputs = self.model( 2025-08-14T21:41:25.6880736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6881132Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6881482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6881852Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6882195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6882542Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6882921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.6883305Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.6883440Z 2025-08-14T21:41:25.6883551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6883900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6884219Z return mod(**inputs) 2025-08-14T21:41:25.6884574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6884942Z outputs = self.model( 2025-08-14T21:41:25.6885302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6885697Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6886066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6886437Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6886786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6887157Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6887532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6887919Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6888308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6888761Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6888961Z 2025-08-14T21:41:25.6889064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6889421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6889742Z return mod(**inputs) 2025-08-14T21:41:25.6890102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6890461Z outputs = self.model( 2025-08-14T21:41:25.6890807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6891182Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6891545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6891951Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6892291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6892642Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6893010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6893440Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6893859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6894277Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6894420Z 2025-08-14T21:41:25.6894530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6894948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6895300Z return mod(**inputs) 2025-08-14T21:41:25.6895676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6896043Z outputs = self.model( 2025-08-14T21:41:25.6896391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6896759Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6897122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6897496Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6897837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6898190Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6898562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6898961Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6899354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6899738Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6899885Z 2025-08-14T21:41:25.6899967Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6900184Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6900391Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6900600Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6900839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6901206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6901519Z return mod(**inputs) 2025-08-14T21:41:25.6901867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6902238Z outputs = self.model( 2025-08-14T21:41:25.6902581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6902958Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6903331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6903712Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6904049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6904406Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6904796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6905224Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6905626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6906069Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6906509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6906976Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6907166Z 2025-08-14T21:41:25.6907282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6907630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6907944Z return mod(**inputs) 2025-08-14T21:41:25.6908313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6908802Z outputs = self.model( 2025-08-14T21:41:25.6909179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6909561Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6909940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6910325Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6910679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6911035Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6911422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6911853Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6912277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6912702Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6913177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6913667Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6913843Z 2025-08-14T21:41:25.6913957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6914355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6914719Z return mod(**inputs) 2025-08-14T21:41:25.6915094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6915501Z outputs = self.model( 2025-08-14T21:41:25.6915928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6916349Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6916746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6917160Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6917537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6917935Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6918333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6918761Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6919174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.6919591Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.6919759Z 2025-08-14T21:41:25.6919870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6920246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6920677Z return mod(**inputs) 2025-08-14T21:41:25.6921054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6921466Z outputs = self.model( 2025-08-14T21:41:25.6921856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6922268Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6922668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6923083Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6923522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6923904Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6924313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6924767Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6924948Z 2025-08-14T21:41:25.6925065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6925456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6925795Z return mod(**inputs) 2025-08-14T21:41:25.6926143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6926511Z outputs = self.model( 2025-08-14T21:41:25.6926857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6927236Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6927601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6927973Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6928310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6928661Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6929032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6929442Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6929819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.6930153Z return self.act(input) 2025-08-14T21:41:25.6930261Z 2025-08-14T21:41:25.6930364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6930712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6931030Z return mod(**inputs) 2025-08-14T21:41:25.6931371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6931730Z outputs = self.model( 2025-08-14T21:41:25.6932079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6932451Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6932804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6933172Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6933508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6933873Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6934246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.6934672Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.6934810Z 2025-08-14T21:41:25.6934921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6935282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6935609Z return mod(**inputs) 2025-08-14T21:41:25.6935956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6936331Z outputs = self.model( 2025-08-14T21:41:25.6936676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6937049Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6937449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6937818Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6938161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6938511Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6938884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6939266Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6939651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6940097Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6940293Z 2025-08-14T21:41:25.6940398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6940743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6941056Z return mod(**inputs) 2025-08-14T21:41:25.6941407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6941778Z outputs = self.model( 2025-08-14T21:41:25.6942124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6942505Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6942877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6943248Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6943592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6943960Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6944366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6944797Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6945224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6945646Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6945790Z 2025-08-14T21:41:25.6945907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6946271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6946597Z return mod(**inputs) 2025-08-14T21:41:25.6946957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6947331Z outputs = self.model( 2025-08-14T21:41:25.6947699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6948084Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6948502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6948870Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6949218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6949296Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6949546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6949635Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6949877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6950006Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6950010Z 2025-08-14T21:41:25.6950093Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6950183Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6950259Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6950334Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6950443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6950639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6950705Z return mod(**inputs) 2025-08-14T21:41:25.6950959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6951028Z outputs = self.model( 2025-08-14T21:41:25.6951281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6951355Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6951623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6951710Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6951947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6952037Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6952300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6952397Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6952660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6952763Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6953071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6953216Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6953223Z 2025-08-14T21:41:25.6953332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6953546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6953617Z return mod(**inputs) 2025-08-14T21:41:25.6953873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6953953Z outputs = self.model( 2025-08-14T21:41:25.6954211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6954288Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6954551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6954630Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6954866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6954982Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6955239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6955340Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6955594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6955703Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6956076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6956198Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6956244Z 2025-08-14T21:41:25.6956365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6956573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6956646Z return mod(**inputs) 2025-08-14T21:41:25.6956912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6956995Z outputs = self.model( 2025-08-14T21:41:25.6957249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6957322Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6957564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6957645Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6957864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6957949Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6958188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6958281Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6958529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.6958610Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.6958613Z 2025-08-14T21:41:25.6958714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6958917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6958981Z return mod(**inputs) 2025-08-14T21:41:25.6959231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6959302Z outputs = self.model( 2025-08-14T21:41:25.6959547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6959658Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6959899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6959971Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6960196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6960273Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6960520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6960640Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6960644Z 2025-08-14T21:41:25.6960749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6960951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6961069Z return mod(**inputs) 2025-08-14T21:41:25.6961326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6961394Z outputs = self.model( 2025-08-14T21:41:25.6961640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6961720Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6961973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6962044Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6962263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6962368Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6962611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6962730Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6962933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.6963009Z return self.act(input) 2025-08-14T21:41:25.6963012Z 2025-08-14T21:41:25.6963112Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6963307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6963371Z return mod(**inputs) 2025-08-14T21:41:25.6963607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6963679Z outputs = self.model( 2025-08-14T21:41:25.6963920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6963995Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6964239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6964307Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6964523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6964598Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6964836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.6964922Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.6964926Z 2025-08-14T21:41:25.6965025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6965214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6965285Z return mod(**inputs) 2025-08-14T21:41:25.6965525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6965600Z outputs = self.model( 2025-08-14T21:41:25.6965840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6965909Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6966149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6966218Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6966433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6966508Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6966750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6966846Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6967113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6967262Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6967272Z 2025-08-14T21:41:25.6967371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6967560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6967631Z return mod(**inputs) 2025-08-14T21:41:25.6967873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6967938Z outputs = self.model( 2025-08-14T21:41:25.6968221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6968292Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6968533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6968602Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6968810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6968890Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6969126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6969213Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6969455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6969533Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6969536Z 2025-08-14T21:41:25.6969640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6969832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6969896Z return mod(**inputs) 2025-08-14T21:41:25.6970143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6970210Z outputs = self.model( 2025-08-14T21:41:25.6970452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6970533Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6970783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6970858Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6971070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6971145Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6971386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6971472Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6971719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6971803Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6971807Z 2025-08-14T21:41:25.6971884Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6971966Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6972042Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6972115Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6972223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6972414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6972486Z return mod(**inputs) 2025-08-14T21:41:25.6972760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6972827Z outputs = self.model( 2025-08-14T21:41:25.6973075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6973145Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6973381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6973457Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6973669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6973752Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6974032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6974122Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6974369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6974468Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6974757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6974896Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6974899Z 2025-08-14T21:41:25.6975002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6975202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6975270Z return mod(**inputs) 2025-08-14T21:41:25.6975522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6975600Z outputs = self.model( 2025-08-14T21:41:25.6975847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6975929Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6976177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6976246Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6976464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6976539Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6976773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6976870Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6977105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6977211Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6977488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6977595Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6977599Z 2025-08-14T21:41:25.6977707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6977898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6977969Z return mod(**inputs) 2025-08-14T21:41:25.6978209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6978276Z outputs = self.model( 2025-08-14T21:41:25.6978523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6978638Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6978878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6978954Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6979170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6979255Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6979503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6979593Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6979877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.6979961Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.6979964Z 2025-08-14T21:41:25.6980075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6980271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6980348Z return mod(**inputs) 2025-08-14T21:41:25.6980593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6980658Z outputs = self.model( 2025-08-14T21:41:25.6980893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6980970Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6981201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6981279Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6981490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6981571Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6981817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6981935Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6981940Z 2025-08-14T21:41:25.6982041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6982243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6982308Z return mod(**inputs) 2025-08-14T21:41:25.6982560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6982627Z outputs = self.model( 2025-08-14T21:41:25.6982872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6982956Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6983197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6983273Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6983485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6983562Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6983809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.6983925Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.6984132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.6984215Z return self.act(input) 2025-08-14T21:41:25.6984219Z 2025-08-14T21:41:25.6984320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6984562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6984626Z return mod(**inputs) 2025-08-14T21:41:25.6984873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6984945Z outputs = self.model( 2025-08-14T21:41:25.6985196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6985267Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6985522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6985594Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6985853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6985931Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6986173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.6986259Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.6986265Z 2025-08-14T21:41:25.6986364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6986564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6986627Z return mod(**inputs) 2025-08-14T21:41:25.6986870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6986942Z outputs = self.model( 2025-08-14T21:41:25.6987192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6987264Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6987515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6987589Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6987815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6987891Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6988133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6988231Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6988474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.6988624Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.6988638Z 2025-08-14T21:41:25.6988740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6988935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6989008Z return mod(**inputs) 2025-08-14T21:41:25.6989257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6989324Z outputs = self.model( 2025-08-14T21:41:25.6989578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6989651Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6989902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6989973Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6990212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6990300Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6990593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6990688Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6990959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.6991043Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.6991047Z 2025-08-14T21:41:25.6991161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6991365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6991433Z return mod(**inputs) 2025-08-14T21:41:25.6991730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6991804Z outputs = self.model( 2025-08-14T21:41:25.6992086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6992164Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6992430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6992514Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6992754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6992834Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6993106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6993198Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6993472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.6993564Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.6993570Z 2025-08-14T21:41:25.6993656Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6993747Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6993827Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6993907Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.6994024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6994229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6994305Z return mod(**inputs) 2025-08-14T21:41:25.6994579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6994652Z outputs = self.model( 2025-08-14T21:41:25.6994922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6994999Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6995256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6995340Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6995570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6995661Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6995999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6996099Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6996371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6996482Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6996804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.6996994Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.6996998Z 2025-08-14T21:41:25.6997111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.6997335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.6997409Z return mod(**inputs) 2025-08-14T21:41:25.6997681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.6997765Z outputs = self.model( 2025-08-14T21:41:25.6998037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.6998136Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.6998416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.6998490Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.6998725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.6998805Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.6999050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.6999146Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.6999409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.6999522Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.6999841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.6999961Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.6999965Z 2025-08-14T21:41:25.7000085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7000299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7000377Z return mod(**inputs) 2025-08-14T21:41:25.7000648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7000721Z outputs = self.model( 2025-08-14T21:41:25.7001000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7001078Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7001340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7001429Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7001667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7001763Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7002030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7002127Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7002398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7002485Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7002489Z 2025-08-14T21:41:25.7002605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7002820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7002889Z return mod(**inputs) 2025-08-14T21:41:25.7003168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7003241Z outputs = self.model( 2025-08-14T21:41:25.7003551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7003638Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7003902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7003989Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7004223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7004306Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7004575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.7004748Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7004752Z 2025-08-14T21:41:25.7004873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7005091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7005162Z return mod(**inputs) 2025-08-14T21:41:25.7005442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7005515Z outputs = self.model( 2025-08-14T21:41:25.7005793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7005880Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7006150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7006235Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7006479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7006563Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7006836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.7006964Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7007190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7007273Z return self.act(input) 2025-08-14T21:41:25.7007277Z 2025-08-14T21:41:25.7007387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7007618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7007696Z return mod(**inputs) 2025-08-14T21:41:25.7007942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7008018Z outputs = self.model( 2025-08-14T21:41:25.7008263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7008339Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7008595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7008801Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7009031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7009109Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7009345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.7009435Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7009439Z 2025-08-14T21:41:25.7009544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7009738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7009866Z return mod(**inputs) 2025-08-14T21:41:25.7010111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7010183Z outputs = self.model( 2025-08-14T21:41:25.7010429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7010499Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7010749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7010817Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7011088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7011166Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7011405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7011506Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7011744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7011901Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7011905Z 2025-08-14T21:41:25.7012004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7012196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7012268Z return mod(**inputs) 2025-08-14T21:41:25.7012507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7012575Z outputs = self.model( 2025-08-14T21:41:25.7012823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7012895Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7013138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7013210Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7013427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7013512Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7013758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7013846Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7014100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7014182Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7014188Z 2025-08-14T21:41:25.7014300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7014499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7014562Z return mod(**inputs) 2025-08-14T21:41:25.7014814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7014880Z outputs = self.model( 2025-08-14T21:41:25.7015134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7015205Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7015450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7015528Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7015758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7015869Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7016104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7016189Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7016431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7016513Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7016517Z 2025-08-14T21:41:25.7016595Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7016681Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7016755Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7016860Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7016970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7017159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7017232Z return mod(**inputs) 2025-08-14T21:41:25.7017471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7017534Z outputs = self.model( 2025-08-14T21:41:25.7017781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7017851Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7018087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7018161Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7018374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7018459Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7018699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7018789Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7019030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7019125Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7019410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7019540Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7019545Z 2025-08-14T21:41:25.7019642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7019843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7019906Z return mod(**inputs) 2025-08-14T21:41:25.7020148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7020224Z outputs = self.model( 2025-08-14T21:41:25.7020464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7020541Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7020775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7020844Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7021063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7021138Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7021383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7021471Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7021741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7021841Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7022124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7022232Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7022235Z 2025-08-14T21:41:25.7022343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7022536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7022605Z return mod(**inputs) 2025-08-14T21:41:25.7022876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7022943Z outputs = self.model( 2025-08-14T21:41:25.7023196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7023266Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7023506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7023577Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7023786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7023868Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7024106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7024196Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7024439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7024521Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7024525Z 2025-08-14T21:41:25.7024628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7024819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7024882Z return mod(**inputs) 2025-08-14T21:41:25.7025127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7025192Z outputs = self.model( 2025-08-14T21:41:25.7025429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7025506Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7025743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7025819Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7026031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7026106Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7026347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.7026462Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7026465Z 2025-08-14T21:41:25.7026569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7026758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7026820Z return mod(**inputs) 2025-08-14T21:41:25.7027069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7027134Z outputs = self.model( 2025-08-14T21:41:25.7027370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7027485Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7027716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7027802Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7028006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7028079Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7028314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.7028424Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7028656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7028729Z return self.act(input) 2025-08-14T21:41:25.7028735Z 2025-08-14T21:41:25.7028831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7029022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7029083Z return mod(**inputs) 2025-08-14T21:41:25.7029311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7029380Z outputs = self.model( 2025-08-14T21:41:25.7029608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7029681Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7029918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7029984Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7030195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7030271Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7030498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.7030584Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7030588Z 2025-08-14T21:41:25.7030683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7030875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7030937Z return mod(**inputs) 2025-08-14T21:41:25.7031177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7031254Z outputs = self.model( 2025-08-14T21:41:25.7031495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7031568Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7031811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7031880Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7032099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7032175Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7032414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7032512Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7032753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7032910Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7032947Z 2025-08-14T21:41:25.7033049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7033245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7033321Z return mod(**inputs) 2025-08-14T21:41:25.7033580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7033650Z outputs = self.model( 2025-08-14T21:41:25.7033918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7033995Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7034256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7034363Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7034596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7034690Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7034944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7035047Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7035300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7035383Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7035387Z 2025-08-14T21:41:25.7035498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7035703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7035771Z return mod(**inputs) 2025-08-14T21:41:25.7036118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7036199Z outputs = self.model( 2025-08-14T21:41:25.7036476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7036556Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7036821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7036911Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7037151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7037229Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7037480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7037573Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7037825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7037914Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7037918Z 2025-08-14T21:41:25.7038000Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7038086Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7038163Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7038250Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7038353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7038556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7038628Z return mod(**inputs) 2025-08-14T21:41:25.7038870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7038939Z outputs = self.model( 2025-08-14T21:41:25.7039186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7039293Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7039538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7039609Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7039824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7039909Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7040149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7040235Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7040512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7040611Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7040900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7041031Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7041034Z 2025-08-14T21:41:25.7041136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7041335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7041401Z return mod(**inputs) 2025-08-14T21:41:25.7041667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7041733Z outputs = self.model( 2025-08-14T21:41:25.7041971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7042051Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7042286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7042361Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7042585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7042665Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7042912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7043001Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7043241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7043344Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7043631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7043743Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7043755Z 2025-08-14T21:41:25.7043855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7044046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7044116Z return mod(**inputs) 2025-08-14T21:41:25.7044365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7044429Z outputs = self.model( 2025-08-14T21:41:25.7044676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7044746Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7044997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7045069Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7045321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7045405Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7045646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:41:25.7045735Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:41:25.7045982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7046064Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7046067Z 2025-08-14T21:41:25.7046174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7046401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7046469Z return mod(**inputs) 2025-08-14T21:41:25.7046723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7046792Z outputs = self.model( 2025-08-14T21:41:25.7047044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7047116Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7047356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7047434Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7047649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7047725Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7047973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.7048089Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7048096Z 2025-08-14T21:41:25.7048207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7048405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7048469Z return mod(**inputs) 2025-08-14T21:41:25.7048722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7048790Z outputs = self.model( 2025-08-14T21:41:25.7049033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7049116Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7049360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7049440Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7049654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7049734Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7049982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:41:25.7050096Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7050314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7050383Z return self.act(input) 2025-08-14T21:41:25.7050386Z 2025-08-14T21:41:25.7050487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7050688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7050755Z return mod(**inputs) 2025-08-14T21:41:25.7050996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7051118Z outputs = self.model( 2025-08-14T21:41:25.7051365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:41:25.7051444Z encoder_outputs = self.encoder( 2025-08-14T21:41:25.7051693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:41:25.7051764Z layer_outputs = encoder_layer( 2025-08-14T21:41:25.7051990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7052068Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7052347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:41:25.7052440Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7052444Z 2025-08-14T21:41:25.7052545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7052749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7052814Z return mod(**inputs) 2025-08-14T21:41:25.7053064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7053137Z outputs = self.model( 2025-08-14T21:41:25.7053382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7053463Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7053710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7053782Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7054009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7054091Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7054334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7054443Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7054689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7054850Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7054855Z 2025-08-14T21:41:25.7054962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7055181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7055258Z return mod(**inputs) 2025-08-14T21:41:25.7055529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7055604Z outputs = self.model( 2025-08-14T21:41:25.7055853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7055927Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7056180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7056255Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7056491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7056581Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7056842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7056952Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7057192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7057304Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7057308Z 2025-08-14T21:41:25.7057415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7057607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7057670Z return mod(**inputs) 2025-08-14T21:41:25.7057917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7057983Z outputs = self.model( 2025-08-14T21:41:25.7058233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7058307Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7058580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7058663Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7058878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7058963Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7059215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7059312Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7059552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7059635Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7059638Z 2025-08-14T21:41:25.7059717Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7059803Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7059878Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7059958Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7060060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7060248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7060316Z return mod(**inputs) 2025-08-14T21:41:25.7060553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7060618Z outputs = self.model( 2025-08-14T21:41:25.7060867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7060938Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7061182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7061256Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7061464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7061553Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7061787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7061883Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7062130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7062232Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7062528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7062658Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7062666Z 2025-08-14T21:41:25.7062768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7062971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7063073Z return mod(**inputs) 2025-08-14T21:41:25.7063335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7063402Z outputs = self.model( 2025-08-14T21:41:25.7063640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7063717Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7063954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7064024Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7064269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7064348Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7064590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7064687Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7064920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7065021Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7065298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7065412Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7065415Z 2025-08-14T21:41:25.7065513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7065704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7065774Z return mod(**inputs) 2025-08-14T21:41:25.7066016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7066084Z outputs = self.model( 2025-08-14T21:41:25.7066328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7066397Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7066641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7066712Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7066922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7067005Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7067246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7067339Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7067584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7067663Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7067666Z 2025-08-14T21:41:25.7067770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7067960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7068022Z return mod(**inputs) 2025-08-14T21:41:25.7068268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7068333Z outputs = self.model( 2025-08-14T21:41:25.7068584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7068654Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7068891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7069002Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7069211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7069288Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7069530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7069637Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7069877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7070074Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7070078Z 2025-08-14T21:41:25.7070181Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7070382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7070446Z return mod(**inputs) 2025-08-14T21:41:25.7070700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7070768Z outputs = self.model( 2025-08-14T21:41:25.7071012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7071095Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7071340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7071410Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7071636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7071714Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7071971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7072077Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7072327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7072416Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7072420Z 2025-08-14T21:41:25.7072522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7072727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7072794Z return mod(**inputs) 2025-08-14T21:41:25.7073058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7073136Z outputs = self.model( 2025-08-14T21:41:25.7073401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7073481Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7073755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7073829Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7074071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7074155Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7074415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7074530Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7074782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7074912Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7074923Z 2025-08-14T21:41:25.7075004Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7075084Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7075167Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7075244Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7075344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7075557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7075627Z return mod(**inputs) 2025-08-14T21:41:25.7075949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7076036Z outputs = self.model( 2025-08-14T21:41:25.7076335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7076422Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7076691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7076768Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7077016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7077102Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7077372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7077499Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7077764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7077884Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7078209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7078351Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7078355Z 2025-08-14T21:41:25.7078468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7078672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7078746Z return mod(**inputs) 2025-08-14T21:41:25.7079006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7079077Z outputs = self.model( 2025-08-14T21:41:25.7079345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7079424Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7079684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7079770Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7080000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7080090Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7080352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7080464Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7080733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7080835Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7081148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7081262Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7081360Z 2025-08-14T21:41:25.7081467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7081682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7081751Z return mod(**inputs) 2025-08-14T21:41:25.7082006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7082084Z outputs = self.model( 2025-08-14T21:41:25.7082341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7082424Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7082714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7082793Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7083033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7083118Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7083387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7083499Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7083760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7083853Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7083857Z 2025-08-14T21:41:25.7083965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7084172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7084252Z return mod(**inputs) 2025-08-14T21:41:25.7084513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7084592Z outputs = self.model( 2025-08-14T21:41:25.7084857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7084929Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7085185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7085257Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7085485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7085569Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7085810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7085931Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7085934Z 2025-08-14T21:41:25.7086036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7086230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7086302Z return mod(**inputs) 2025-08-14T21:41:25.7086548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7086620Z outputs = self.model( 2025-08-14T21:41:25.7086873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7086945Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7087197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7087270Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7087489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7087608Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7087847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7087970Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7088178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7088246Z return self.act(input) 2025-08-14T21:41:25.7088250Z 2025-08-14T21:41:25.7088358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7088551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7088620Z return mod(**inputs) 2025-08-14T21:41:25.7088894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7088963Z outputs = self.model( 2025-08-14T21:41:25.7089226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7089296Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7089531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7089607Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7089816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7089901Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7090139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7090220Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7090223Z 2025-08-14T21:41:25.7090329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7090520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7090583Z return mod(**inputs) 2025-08-14T21:41:25.7090833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7090897Z outputs = self.model( 2025-08-14T21:41:25.7091144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7091215Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7091452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7091529Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7091743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7091825Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7092063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7092158Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7092401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7092545Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7092548Z 2025-08-14T21:41:25.7092645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7092840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7092904Z return mod(**inputs) 2025-08-14T21:41:25.7093158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7093225Z outputs = self.model( 2025-08-14T21:41:25.7093509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7093588Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7093830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7093900Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7094122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7094199Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7094485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7094588Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7094892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7094989Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7094992Z 2025-08-14T21:41:25.7095106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7095312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7095378Z return mod(**inputs) 2025-08-14T21:41:25.7095621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7095696Z outputs = self.model( 2025-08-14T21:41:25.7095942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7096014Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7096270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7096340Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7096572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7096647Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7096887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7096990Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7097226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7097317Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7097320Z 2025-08-14T21:41:25.7097398Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7097475Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7097559Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7097634Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7097731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7097934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7097997Z return mod(**inputs) 2025-08-14T21:41:25.7098241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7098313Z outputs = self.model( 2025-08-14T21:41:25.7098553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7098632Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7098870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7098938Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7099160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7099267Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7099512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7099606Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7099847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7099950Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7100231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7100357Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7100368Z 2025-08-14T21:41:25.7100500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7100695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7100769Z return mod(**inputs) 2025-08-14T21:41:25.7101007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7101073Z outputs = self.model( 2025-08-14T21:41:25.7101319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7101390Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7101632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7101702Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7101920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7102008Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7102251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7102349Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7102595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7102690Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7102984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7103091Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7103095Z 2025-08-14T21:41:25.7103195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7103398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7103465Z return mod(**inputs) 2025-08-14T21:41:25.7103720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7103791Z outputs = self.model( 2025-08-14T21:41:25.7104033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7104114Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7107467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7107543Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7107771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7107851Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7108097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7108204Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7108478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7108570Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7108574Z 2025-08-14T21:41:25.7108819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7109026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7109135Z return mod(**inputs) 2025-08-14T21:41:25.7109388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7109457Z outputs = self.model( 2025-08-14T21:41:25.7109701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7109829Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7110091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7110170Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7110411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7110494Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7110762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7110876Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7111134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7111298Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7111303Z 2025-08-14T21:41:25.7111416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7111634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7111705Z return mod(**inputs) 2025-08-14T21:41:25.7111966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7112046Z outputs = self.model( 2025-08-14T21:41:25.7112305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7112383Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7112650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7112726Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7112969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7113052Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7113311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7113434Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7113692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7113862Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7113866Z 2025-08-14T21:41:25.7113973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7114180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7114257Z return mod(**inputs) 2025-08-14T21:41:25.7114514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7114586Z outputs = self.model( 2025-08-14T21:41:25.7114857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7114961Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7115229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7115304Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7115535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7115626Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7115932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7116052Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7116367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7116460Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7116466Z 2025-08-14T21:41:25.7116558Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7116643Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7116727Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7116819Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7116928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7117142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7117221Z return mod(**inputs) 2025-08-14T21:41:25.7117493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7117569Z outputs = self.model( 2025-08-14T21:41:25.7117815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7117886Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7118135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7118206Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7118427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7118503Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7118743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7118854Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7119095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7119191Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7119486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7119617Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7119620Z 2025-08-14T21:41:25.7119739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7119926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7120020Z return mod(**inputs) 2025-08-14T21:41:25.7120270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7120334Z outputs = self.model( 2025-08-14T21:41:25.7120577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7120647Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7120885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7120960Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7121191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7121266Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7121509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7121613Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7121852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7121949Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7122269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7122382Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7122385Z 2025-08-14T21:41:25.7122483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7122676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7122739Z return mod(**inputs) 2025-08-14T21:41:25.7122978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7123055Z outputs = self.model( 2025-08-14T21:41:25.7123294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7123366Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7123615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7123688Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7123911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7123990Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7124232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7124350Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7124606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7124691Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7124701Z 2025-08-14T21:41:25.7124806Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7125007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7125075Z return mod(**inputs) 2025-08-14T21:41:25.7125311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7125380Z outputs = self.model( 2025-08-14T21:41:25.7125622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7125692Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7125935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7126031Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7126244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7126326Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7126565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7126683Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7126686Z 2025-08-14T21:41:25.7126792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7127009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7127079Z return mod(**inputs) 2025-08-14T21:41:25.7127311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7127380Z outputs = self.model( 2025-08-14T21:41:25.7127625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7127697Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7127939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7128028Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7128266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7128349Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7128580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7128691Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7128894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7128960Z return self.act(input) 2025-08-14T21:41:25.7128964Z 2025-08-14T21:41:25.7129067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7129250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7129310Z return mod(**inputs) 2025-08-14T21:41:25.7129548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7129611Z outputs = self.model( 2025-08-14T21:41:25.7129842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7129920Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7130148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7130224Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7130427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7130500Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7130736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7130814Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7130817Z 2025-08-14T21:41:25.7130923Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7131103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7131167Z return mod(**inputs) 2025-08-14T21:41:25.7131402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7131465Z outputs = self.model( 2025-08-14T21:41:25.7131693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7131790Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7132020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7132094Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7132298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7132373Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7132609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7132718Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7132954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7133112Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7133116Z 2025-08-14T21:41:25.7133217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7133417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7133481Z return mod(**inputs) 2025-08-14T21:41:25.7133755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7133831Z outputs = self.model( 2025-08-14T21:41:25.7134075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7134156Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7134397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7134467Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7134693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7134780Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7135008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7135110Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7135343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7135425Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7135430Z 2025-08-14T21:41:25.7135526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7135710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7135778Z return mod(**inputs) 2025-08-14T21:41:25.7136010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7136074Z outputs = self.model( 2025-08-14T21:41:25.7136310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7136377Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7136619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7136689Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7136898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7136980Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7137211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7137313Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7137562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7137642Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7137646Z 2025-08-14T21:41:25.7137728Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7137804Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7137876Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7137957Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7138052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7138260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7138322Z return mod(**inputs) 2025-08-14T21:41:25.7138553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7138625Z outputs = self.model( 2025-08-14T21:41:25.7138857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7138926Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7139165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7139231Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7139475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7139551Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7139779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7139876Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7140106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7140200Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7140478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7140602Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7140605Z 2025-08-14T21:41:25.7140709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7140897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7140976Z return mod(**inputs) 2025-08-14T21:41:25.7141208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7141277Z outputs = self.model( 2025-08-14T21:41:25.7141519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7141591Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7141831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7141901Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7142117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7142191Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7142426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7142529Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7142764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7142859Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7143151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7143279Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7143283Z 2025-08-14T21:41:25.7143389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7143584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7143647Z return mod(**inputs) 2025-08-14T21:41:25.7143898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7143992Z outputs = self.model( 2025-08-14T21:41:25.7144245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7144318Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7144565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7144647Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7144872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7144947Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7145192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7145321Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7145567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7145647Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7145650Z 2025-08-14T21:41:25.7145749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7145948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7146013Z return mod(**inputs) 2025-08-14T21:41:25.7146263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7146330Z outputs = self.model( 2025-08-14T21:41:25.7146577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7146655Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7146901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7146972Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7147194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7147271Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7147516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7147624Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7147865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7148020Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7148024Z 2025-08-14T21:41:25.7148126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7148330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7148396Z return mod(**inputs) 2025-08-14T21:41:25.7148640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7148714Z outputs = self.model( 2025-08-14T21:41:25.7148955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7149046Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7149298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7149368Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7149591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7149670Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7149915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7150047Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7150291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7150370Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7150382Z 2025-08-14T21:41:25.7150482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7150677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7150748Z return mod(**inputs) 2025-08-14T21:41:25.7150993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7151059Z outputs = self.model( 2025-08-14T21:41:25.7151343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7151420Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7151670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7151740Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7151954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7152040Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7152279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7152383Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7152633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7152719Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7152723Z 2025-08-14T21:41:25.7152812Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7152892Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7152968Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7153051Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7153151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7153349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7153424Z return mod(**inputs) 2025-08-14T21:41:25.7153667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7153742Z outputs = self.model( 2025-08-14T21:41:25.7154010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7154091Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7154354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7154430Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7154664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7154754Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7155042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7155159Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7155423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7155523Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7156081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7156231Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7156261Z 2025-08-14T21:41:25.7156381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7156596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7156667Z return mod(**inputs) 2025-08-14T21:41:25.7157003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7157086Z outputs = self.model( 2025-08-14T21:41:25.7157332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7157413Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7157697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7157779Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7157998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7158077Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7158326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7158434Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7158682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7158778Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7159061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7159177Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7159181Z 2025-08-14T21:41:25.7159281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7159476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7159548Z return mod(**inputs) 2025-08-14T21:41:25.7159794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7159870Z outputs = self.model( 2025-08-14T21:41:25.7160113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7160185Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7160434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7160504Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7160733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7160819Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7161055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7161163Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7161407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7161504Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7161507Z 2025-08-14T21:41:25.7161610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7161793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7161861Z return mod(**inputs) 2025-08-14T21:41:25.7162098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7162161Z outputs = self.model( 2025-08-14T21:41:25.7162414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7162480Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7162715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7162791Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7162995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7163077Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7163318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7163464Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7163468Z 2025-08-14T21:41:25.7163577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7163783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7163858Z return mod(**inputs) 2025-08-14T21:41:25.7164133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7164204Z outputs = self.model( 2025-08-14T21:41:25.7164475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7164547Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7164788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7164865Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7165087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7165170Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7165405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7165521Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7165733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7165801Z return self.act(input) 2025-08-14T21:41:25.7165805Z 2025-08-14T21:41:25.7165903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7166102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7166167Z return mod(**inputs) 2025-08-14T21:41:25.7166413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7166479Z outputs = self.model( 2025-08-14T21:41:25.7166715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7166792Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7167034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7167109Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7167339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7167413Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7167650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7167728Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7167731Z 2025-08-14T21:41:25.7167830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7168019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7168100Z return mod(**inputs) 2025-08-14T21:41:25.7168339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7168403Z outputs = self.model( 2025-08-14T21:41:25.7168634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7168713Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7168946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7169014Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7169228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7169341Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7169581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7169677Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7169905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7170054Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7170059Z 2025-08-14T21:41:25.7170155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7170349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7170409Z return mod(**inputs) 2025-08-14T21:41:25.7170640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7170713Z outputs = self.model( 2025-08-14T21:41:25.7170946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7171016Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7171252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7171317Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7171529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7171604Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7171834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7171936Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7172166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7172248Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7172252Z 2025-08-14T21:41:25.7172348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7172530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7172596Z return mod(**inputs) 2025-08-14T21:41:25.7172826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7172915Z outputs = self.model( 2025-08-14T21:41:25.7173166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7173234Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7173486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7173559Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7173776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7173880Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7174119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7174221Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7174481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7174571Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7174575Z 2025-08-14T21:41:25.7174665Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7174757Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7174833Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7174914Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7175048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7175245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7175316Z return mod(**inputs) 2025-08-14T21:41:25.7175558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7175631Z outputs = self.model( 2025-08-14T21:41:25.7175885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7175957Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7176199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7176267Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7176488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7176573Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7176800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7176899Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7177127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7177220Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7177493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7177619Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7177622Z 2025-08-14T21:41:25.7177724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7177915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7177976Z return mod(**inputs) 2025-08-14T21:41:25.7178222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7178286Z outputs = self.model( 2025-08-14T21:41:25.7178528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7178601Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7178857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7178931Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7179136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7179208Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7179450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7179541Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7179792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7179882Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7180155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7180264Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7180267Z 2025-08-14T21:41:25.7180362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7180552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7180614Z return mod(**inputs) 2025-08-14T21:41:25.7180871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7180944Z outputs = self.model( 2025-08-14T21:41:25.7181175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7181245Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7181488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7181557Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7181770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7181844Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7182078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7182179Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7182413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7182493Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7182496Z 2025-08-14T21:41:25.7182599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7182787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7182857Z return mod(**inputs) 2025-08-14T21:41:25.7183097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7183161Z outputs = self.model( 2025-08-14T21:41:25.7183406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7183473Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7183712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7183789Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7184004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7184086Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7184335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7184458Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7184691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7184831Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7184834Z 2025-08-14T21:41:25.7184936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7185127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7185190Z return mod(**inputs) 2025-08-14T21:41:25.7185453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7185518Z outputs = self.model( 2025-08-14T21:41:25.7185756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7185834Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7186072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7186149Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7186364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7186439Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7186726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7186834Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7187078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7187157Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7187160Z 2025-08-14T21:41:25.7187259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7187466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7187527Z return mod(**inputs) 2025-08-14T21:41:25.7187756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7187829Z outputs = self.model( 2025-08-14T21:41:25.7188066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7188144Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7188381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7188451Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7188666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7188743Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7188976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7189088Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7189322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7189413Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7189416Z 2025-08-14T21:41:25.7189493Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7189569Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7189653Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7189725Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7189828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7190017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7190097Z return mod(**inputs) 2025-08-14T21:41:25.7190344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7190407Z outputs = self.model( 2025-08-14T21:41:25.7190643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7190724Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7190964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7191061Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7191278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7191355Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7191605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7191712Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7191954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7192057Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7192379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7192516Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7192522Z 2025-08-14T21:41:25.7192622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7192814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7192885Z return mod(**inputs) 2025-08-14T21:41:25.7193128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7193201Z outputs = self.model( 2025-08-14T21:41:25.7193442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7193514Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7193765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7193835Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7194058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7194151Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7194405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7194522Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7194779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7194880Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7195186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7195296Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7195303Z 2025-08-14T21:41:25.7195414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7195619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7195687Z return mod(**inputs) 2025-08-14T21:41:25.7196029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7196105Z outputs = self.model( 2025-08-14T21:41:25.7196408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7196493Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7196761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7196846Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7197086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7197169Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7197461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7197574Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7197828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7197920Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7197923Z 2025-08-14T21:41:25.7198025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7198227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7198294Z return mod(**inputs) 2025-08-14T21:41:25.7198578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7198664Z outputs = self.model( 2025-08-14T21:41:25.7198914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7198994Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7199241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7199311Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7199537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7199615Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7199863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7199991Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7199994Z 2025-08-14T21:41:25.7200099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7200302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7200369Z return mod(**inputs) 2025-08-14T21:41:25.7200616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7200693Z outputs = self.model( 2025-08-14T21:41:25.7200948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7201029Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7201275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7201344Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7201566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7201642Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7201881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7202005Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7202210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7202284Z return self.act(input) 2025-08-14T21:41:25.7202308Z 2025-08-14T21:41:25.7202408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7202600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7202671Z return mod(**inputs) 2025-08-14T21:41:25.7202919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7202987Z outputs = self.model( 2025-08-14T21:41:25.7203238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7203328Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7203578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7203648Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7203872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7203958Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7204195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7204279Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7204282Z 2025-08-14T21:41:25.7204382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7204606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7204680Z return mod(**inputs) 2025-08-14T21:41:25.7204929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7204995Z outputs = self.model( 2025-08-14T21:41:25.7205244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7205316Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7205617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7205686Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7205898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7205980Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7206220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7206315Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7206561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7206704Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7206709Z 2025-08-14T21:41:25.7206815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7207005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7207067Z return mod(**inputs) 2025-08-14T21:41:25.7207316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7207379Z outputs = self.model( 2025-08-14T21:41:25.7207633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7207706Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7207953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7208032Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7208249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7208367Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7208617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7208897Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7209157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7209243Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7209247Z 2025-08-14T21:41:25.7209351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7209596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7209659Z return mod(**inputs) 2025-08-14T21:41:25.7209914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7209983Z outputs = self.model( 2025-08-14T21:41:25.7210228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7210306Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7210551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7210621Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7210906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7210990Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7211239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7211338Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7211581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7211677Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7211680Z 2025-08-14T21:41:25.7211761Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7211839Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7211922Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7211997Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7212105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7212296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7212362Z return mod(**inputs) 2025-08-14T21:41:25.7212611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7212677Z outputs = self.model( 2025-08-14T21:41:25.7212920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7213001Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7213243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7213324Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7213541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7213621Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7213868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7213967Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7214204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7214309Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7214630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7214768Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7214771Z 2025-08-14T21:41:25.7214872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7215068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7215143Z return mod(**inputs) 2025-08-14T21:41:25.7215391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7215485Z outputs = self.model( 2025-08-14T21:41:25.7215727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7215799Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7216052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7216124Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7216337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7216421Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7216692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7216800Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7217046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7217141Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7217433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7217543Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7217547Z 2025-08-14T21:41:25.7217664Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7217849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7217909Z return mod(**inputs) 2025-08-14T21:41:25.7218147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7218211Z outputs = self.model( 2025-08-14T21:41:25.7218443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7218523Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7218759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7218835Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7219046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7219121Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7219362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7219456Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7219699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7219779Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7219784Z 2025-08-14T21:41:25.7219882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7220078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7220140Z return mod(**inputs) 2025-08-14T21:41:25.7220405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7220478Z outputs = self.model( 2025-08-14T21:41:25.7220715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7220791Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7221030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7221101Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7221318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7221412Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7221648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7221759Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7221994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7222143Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7222146Z 2025-08-14T21:41:25.7222244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7222464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7222535Z return mod(**inputs) 2025-08-14T21:41:25.7222773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7222847Z outputs = self.model( 2025-08-14T21:41:25.7223084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7223154Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7223398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7223467Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7223674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7223755Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7223992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7224105Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7224339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7224417Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7224420Z 2025-08-14T21:41:25.7224535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7224721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7224790Z return mod(**inputs) 2025-08-14T21:41:25.7225019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7225085Z outputs = self.model( 2025-08-14T21:41:25.7225327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7225396Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7225633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7225710Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7225925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7226009Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7226259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7226358Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7226595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7226676Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7226682Z 2025-08-14T21:41:25.7226765Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7226839Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7226931Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7227010Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7227105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7227291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7227361Z return mod(**inputs) 2025-08-14T21:41:25.7227590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7227654Z outputs = self.model( 2025-08-14T21:41:25.7227888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7227956Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7228224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7228294Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7228499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7228580Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7228811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7228921Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7229155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7229249Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7229541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7229670Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7229673Z 2025-08-14T21:41:25.7229773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7229968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7230030Z return mod(**inputs) 2025-08-14T21:41:25.7230275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7230343Z outputs = self.model( 2025-08-14T21:41:25.7230577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7230655Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7230893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7230970Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7231179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7231257Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7231504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7231607Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7231838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7231960Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7232244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7232358Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7232362Z 2025-08-14T21:41:25.7232467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7232670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7232774Z return mod(**inputs) 2025-08-14T21:41:25.7233038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7233113Z outputs = self.model( 2025-08-14T21:41:25.7233372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7233449Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7233718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7233793Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7234056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7234148Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7234400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7234518Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7234777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7234862Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7234867Z 2025-08-14T21:41:25.7234983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7235188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7235255Z return mod(**inputs) 2025-08-14T21:41:25.7235517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7235591Z outputs = self.model( 2025-08-14T21:41:25.7235910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7235997Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7236256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7236338Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7236570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7236660Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7236916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7237042Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7237046Z 2025-08-14T21:41:25.7237163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7237370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7237441Z return mod(**inputs) 2025-08-14T21:41:25.7237708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7237779Z outputs = self.model( 2025-08-14T21:41:25.7238046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7238151Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7238410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7238493Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7238721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7238806Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7239070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7239251Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7239477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7239552Z return self.act(input) 2025-08-14T21:41:25.7239555Z 2025-08-14T21:41:25.7239664Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7239889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7239956Z return mod(**inputs) 2025-08-14T21:41:25.7240222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7240292Z outputs = self.model( 2025-08-14T21:41:25.7240586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7240672Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7240931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7241005Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7241241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7241325Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7241595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7241680Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7241684Z 2025-08-14T21:41:25.7241791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7242007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7242076Z return mod(**inputs) 2025-08-14T21:41:25.7242334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7242412Z outputs = self.model( 2025-08-14T21:41:25.7242668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7242751Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7243009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7243082Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7243328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7243408Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7243684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7243789Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7244046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7244219Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7244223Z 2025-08-14T21:41:25.7244321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7244527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7244596Z return mod(**inputs) 2025-08-14T21:41:25.7244831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7244901Z outputs = self.model( 2025-08-14T21:41:25.7245143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7245213Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7245474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7245542Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7245760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7245840Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7246074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7246175Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7246412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7246517Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7246521Z 2025-08-14T21:41:25.7246628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7246817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7246887Z return mod(**inputs) 2025-08-14T21:41:25.7247124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7247187Z outputs = self.model( 2025-08-14T21:41:25.7247433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7247503Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7247738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7247813Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7248024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7248105Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7248340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7248433Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7248677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7248761Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7248764Z 2025-08-14T21:41:25.7248846Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7248925Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7248998Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7249076Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7249172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7249363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7249433Z return mod(**inputs) 2025-08-14T21:41:25.7249671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7249741Z outputs = self.model( 2025-08-14T21:41:25.7249979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7250073Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7250320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7250388Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7250600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7250686Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7250923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7251041Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7251274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7251367Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7251658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7251787Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7251790Z 2025-08-14T21:41:25.7251894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7252088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7252151Z return mod(**inputs) 2025-08-14T21:41:25.7252431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7252500Z outputs = self.model( 2025-08-14T21:41:25.7252732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7252811Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7253049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7253125Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7253334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7253410Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7253652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7253749Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7253985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7254087Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7254364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7254475Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7254480Z 2025-08-14T21:41:25.7254579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7254768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7254839Z return mod(**inputs) 2025-08-14T21:41:25.7255074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7255150Z outputs = self.model( 2025-08-14T21:41:25.7255386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7255457Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7255699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7255768Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7256010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7256093Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7256326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7256433Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7256677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7256756Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7256777Z 2025-08-14T21:41:25.7256882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7257072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7257144Z return mod(**inputs) 2025-08-14T21:41:25.7257378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7257445Z outputs = self.model( 2025-08-14T21:41:25.7257686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7257757Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7257990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7258102Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7258313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7258398Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7258631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7258734Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7258979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7259124Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7259128Z 2025-08-14T21:41:25.7259232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7259421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7259487Z return mod(**inputs) 2025-08-14T21:41:25.7259731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7259807Z outputs = self.model( 2025-08-14T21:41:25.7260037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7260111Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7260339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7260413Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7260615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7260687Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7260925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7261028Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7261266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7261350Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7261353Z 2025-08-14T21:41:25.7261450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7261647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7261729Z return mod(**inputs) 2025-08-14T21:41:25.7261967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7262041Z outputs = self.model( 2025-08-14T21:41:25.7262277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7262357Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7262589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7262683Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7262910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7262982Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7263214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7263321Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7263557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7263644Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7263648Z 2025-08-14T21:41:25.7263761Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7263839Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7263923Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7264000Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7264101Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7264307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7264370Z return mod(**inputs) 2025-08-14T21:41:25.7264623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7264690Z outputs = self.model( 2025-08-14T21:41:25.7264935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7265013Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7265262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7265331Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7265555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7265635Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7265893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7266011Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7266246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7266348Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7266627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7266769Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7266772Z 2025-08-14T21:41:25.7266875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7267071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7267143Z return mod(**inputs) 2025-08-14T21:41:25.7267394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7267485Z outputs = self.model( 2025-08-14T21:41:25.7267733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7267803Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7268047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7268116Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7268329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7268412Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7268666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7268777Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7269013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7269111Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7269406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7269512Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7269516Z 2025-08-14T21:41:25.7269649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7269851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7269916Z return mod(**inputs) 2025-08-14T21:41:25.7270167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7270234Z outputs = self.model( 2025-08-14T21:41:25.7270476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7270557Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7270855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7270930Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7271164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7271247Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7271504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7271616Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7271869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7271963Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7271969Z 2025-08-14T21:41:25.7272074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7272288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7272356Z return mod(**inputs) 2025-08-14T21:41:25.7272613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7272690Z outputs = self.model( 2025-08-14T21:41:25.7272951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7273029Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7273296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7273373Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7273615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7273722Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7273984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7274119Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7274123Z 2025-08-14T21:41:25.7274232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7274456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7274527Z return mod(**inputs) 2025-08-14T21:41:25.7274814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7274895Z outputs = self.model( 2025-08-14T21:41:25.7275163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7275242Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7275517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7275591Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7275902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7275999Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7276314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7276455Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7276683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7276758Z return self.act(input) 2025-08-14T21:41:25.7276770Z 2025-08-14T21:41:25.7276882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7277096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7277175Z return mod(**inputs) 2025-08-14T21:41:25.7277445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7277522Z outputs = self.model( 2025-08-14T21:41:25.7277800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7277878Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7278150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7278231Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7278467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7278560Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7278831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7278918Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7278922Z 2025-08-14T21:41:25.7279037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7279248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7279328Z return mod(**inputs) 2025-08-14T21:41:25.7279596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7279672Z outputs = self.model( 2025-08-14T21:41:25.7279943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7280020Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7280283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7280388Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7280624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7280715Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7280982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7281089Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7281384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7281546Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7281550Z 2025-08-14T21:41:25.7281666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7281886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7281955Z return mod(**inputs) 2025-08-14T21:41:25.7282229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7282301Z outputs = self.model( 2025-08-14T21:41:25.7282598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7282686Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7282951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7283038Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7283271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7283355Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7283629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7283734Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7284005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7284093Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7284096Z 2025-08-14T21:41:25.7284209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7284429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7284501Z return mod(**inputs) 2025-08-14T21:41:25.7284767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7284847Z outputs = self.model( 2025-08-14T21:41:25.7285113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7285199Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7285463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7285539Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7285786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7285869Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7286140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7286247Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7286487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7286603Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7286607Z 2025-08-14T21:41:25.7286687Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7286765Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7286848Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7286923Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7287023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7287227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7287291Z return mod(**inputs) 2025-08-14T21:41:25.7287541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7287627Z outputs = self.model( 2025-08-14T21:41:25.7287871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7287950Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7288196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7288278Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7288499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7288578Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7288896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7288998Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7289240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7289346Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7289629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7289769Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7289773Z 2025-08-14T21:41:25.7289873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7290065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7290140Z return mod(**inputs) 2025-08-14T21:41:25.7290386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7290458Z outputs = self.model( 2025-08-14T21:41:25.7290702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7290774Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7291026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7291098Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7291313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7291400Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7291639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7291742Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7291984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7292085Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7292379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7292485Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7292505Z 2025-08-14T21:41:25.7292613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7292805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7292867Z return mod(**inputs) 2025-08-14T21:41:25.7293115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7293181Z outputs = self.model( 2025-08-14T21:41:25.7293428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7293531Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7293774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7293850Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7294064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7294143Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7294387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7294483Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7294764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7294852Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7294856Z 2025-08-14T21:41:25.7294955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7295161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7295226Z return mod(**inputs) 2025-08-14T21:41:25.7295470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7295546Z outputs = self.model( 2025-08-14T21:41:25.7295790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7295868Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7296110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7296181Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7296406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7296485Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7296728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7296843Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7297093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7297258Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7297262Z 2025-08-14T21:41:25.7297369Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7297574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7297648Z return mod(**inputs) 2025-08-14T21:41:25.7297907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7297979Z outputs = self.model( 2025-08-14T21:41:25.7298246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7298321Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7298588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7298687Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7298918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7299007Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7299266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7299391Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7299656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7299763Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7299767Z 2025-08-14T21:41:25.7299880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7300087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7300157Z return mod(**inputs) 2025-08-14T21:41:25.7300423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7300493Z outputs = self.model( 2025-08-14T21:41:25.7300757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7300833Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7301131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7301228Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7301448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7301533Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7301776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7301888Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7302155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7302243Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7302247Z 2025-08-14T21:41:25.7302329Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7302422Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7302503Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7302589Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7302698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7302907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7302983Z return mod(**inputs) 2025-08-14T21:41:25.7303240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7303314Z outputs = self.model( 2025-08-14T21:41:25.7303578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7303655Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7303920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7303996Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7304225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7304317Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7304570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7304682Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7304973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7305074Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7305383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7305521Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7305528Z 2025-08-14T21:41:25.7305633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7305850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7305939Z return mod(**inputs) 2025-08-14T21:41:25.7306206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7306276Z outputs = self.model( 2025-08-14T21:41:25.7306536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7306619Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7306877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7306951Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7307609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7307704Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7307978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7308093Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7308355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7308469Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7308932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7309061Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7309065Z 2025-08-14T21:41:25.7309172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7309381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7309458Z return mod(**inputs) 2025-08-14T21:41:25.7309722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7309793Z outputs = self.model( 2025-08-14T21:41:25.7310056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7310137Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7310401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7310477Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7310707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7310798Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7311054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7311167Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7311428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7311514Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7311517Z 2025-08-14T21:41:25.7311633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7311890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7311959Z return mod(**inputs) 2025-08-14T21:41:25.7312229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7312302Z outputs = self.model( 2025-08-14T21:41:25.7312573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7312648Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7312927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7313008Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7313234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7313317Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7313578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7313704Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7313708Z 2025-08-14T21:41:25.7313822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7314080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7314150Z return mod(**inputs) 2025-08-14T21:41:25.7314414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7314487Z outputs = self.model( 2025-08-14T21:41:25.7314753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7314828Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7315085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7315169Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7315397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7315479Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7315751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7315926Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7316175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7316250Z return self.act(input) 2025-08-14T21:41:25.7316254Z 2025-08-14T21:41:25.7316363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7316580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7316652Z return mod(**inputs) 2025-08-14T21:41:25.7316918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7317005Z outputs = self.model( 2025-08-14T21:41:25.7317266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7317354Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7317627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7317704Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7317940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7318021Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7318317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7318404Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7318408Z 2025-08-14T21:41:25.7318515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7318733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7318806Z return mod(**inputs) 2025-08-14T21:41:25.7319092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7319199Z outputs = self.model( 2025-08-14T21:41:25.7319467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7319552Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7319826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7319905Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7320155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7320239Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7320510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7320664Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7320930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7321105Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7321109Z 2025-08-14T21:41:25.7321221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7321440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7321525Z return mod(**inputs) 2025-08-14T21:41:25.7321798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7321882Z outputs = self.model( 2025-08-14T21:41:25.7322155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7322238Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7322519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7322600Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7322841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7322934Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7323204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7323321Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7323589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7323678Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7323682Z 2025-08-14T21:41:25.7323804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7324022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7324102Z return mod(**inputs) 2025-08-14T21:41:25.7324401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7324476Z outputs = self.model( 2025-08-14T21:41:25.7324777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7324885Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7325186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7325273Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7325513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7325608Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7325911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7326050Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7326291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7326375Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7326380Z 2025-08-14T21:41:25.7326464Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7326540Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7326618Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7326697Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7326795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7326990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7327092Z return mod(**inputs) 2025-08-14T21:41:25.7327333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7327400Z outputs = self.model( 2025-08-14T21:41:25.7327649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7327720Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7327968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7328041Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7328259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7328342Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7328587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7328691Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7328932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7329026Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7329316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7329447Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7329450Z 2025-08-14T21:41:25.7329548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7329748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7329811Z return mod(**inputs) 2025-08-14T21:41:25.7330062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7330128Z outputs = self.model( 2025-08-14T21:41:25.7330367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7330447Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7330686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7330762Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7330997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7331073Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7331315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7331408Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7331644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7331744Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7332042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7332151Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7332154Z 2025-08-14T21:41:25.7332253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7332444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7332514Z return mod(**inputs) 2025-08-14T21:41:25.7332751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7332823Z outputs = self.model( 2025-08-14T21:41:25.7333092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7333165Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7333407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7333476Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7333689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7333777Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7334018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7334123Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7334376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7334456Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7334460Z 2025-08-14T21:41:25.7334563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7334753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7334816Z return mod(**inputs) 2025-08-14T21:41:25.7335061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7335125Z outputs = self.model( 2025-08-14T21:41:25.7335371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7335440Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7335678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7335754Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7335967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7336051Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7336286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7336390Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7336633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7336803Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7336807Z 2025-08-14T21:41:25.7336908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7337110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7337174Z return mod(**inputs) 2025-08-14T21:41:25.7337424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7337501Z outputs = self.model( 2025-08-14T21:41:25.7337752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7337830Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7338066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7338135Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7338350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7338423Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7338663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7338796Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7339031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7339115Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7339118Z 2025-08-14T21:41:25.7339214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7339409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7339473Z return mod(**inputs) 2025-08-14T21:41:25.7339709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7339781Z outputs = self.model( 2025-08-14T21:41:25.7340017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7340086Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7340333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7340401Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7340619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7340692Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7340926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7341037Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7341270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7341358Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7341361Z 2025-08-14T21:41:25.7341438Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7341513Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7341596Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7341670Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7341770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7341968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7342030Z return mod(**inputs) 2025-08-14T21:41:25.7342275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7342360Z outputs = self.model( 2025-08-14T21:41:25.7342596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7342673Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7342907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7342976Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7343194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7343286Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7343529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7343635Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7343874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7343978Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7344260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7344391Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7344401Z 2025-08-14T21:41:25.7344535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7344730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7344801Z return mod(**inputs) 2025-08-14T21:41:25.7345045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7345113Z outputs = self.model( 2025-08-14T21:41:25.7345367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7345440Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7345692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7345763Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7345978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7346067Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7346308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7346416Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7346666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7346761Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7347055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7347159Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7347163Z 2025-08-14T21:41:25.7347274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7347476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7347538Z return mod(**inputs) 2025-08-14T21:41:25.7347785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7347851Z outputs = self.model( 2025-08-14T21:41:25.7348087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7348165Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7348436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7348505Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7348723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7348798Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7349047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7349150Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7349414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7349499Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7349503Z 2025-08-14T21:41:25.7349602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7349803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7349867Z return mod(**inputs) 2025-08-14T21:41:25.7350105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7350179Z outputs = self.model( 2025-08-14T21:41:25.7350454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7350528Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7350778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7350849Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7351080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7351155Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7351401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7351526Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7351530Z 2025-08-14T21:41:25.7351628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7351829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7351904Z return mod(**inputs) 2025-08-14T21:41:25.7352151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7352225Z outputs = self.model( 2025-08-14T21:41:25.7352473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7352543Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7352797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7352871Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7353100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7353177Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7353433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7353564Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7353787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7353860Z return self.act(input) 2025-08-14T21:41:25.7353864Z 2025-08-14T21:41:25.7353979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7354197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7354293Z return mod(**inputs) 2025-08-14T21:41:25.7354567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7354636Z outputs = self.model( 2025-08-14T21:41:25.7354910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7354988Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7355254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7355359Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7355604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7355695Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7356051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7356149Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7356154Z 2025-08-14T21:41:25.7356273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7356486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7356564Z return mod(**inputs) 2025-08-14T21:41:25.7356869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7356944Z outputs = self.model( 2025-08-14T21:41:25.7357222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7357299Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7357571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7357657Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7357897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7357995Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7358225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7358319Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7358569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7358733Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7358737Z 2025-08-14T21:41:25.7358855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7359068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7359140Z return mod(**inputs) 2025-08-14T21:41:25.7359418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7359492Z outputs = self.model( 2025-08-14T21:41:25.7359761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7359847Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7360121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7360207Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7360444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7360528Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7360803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7360934Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7361197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7361292Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7361295Z 2025-08-14T21:41:25.7361402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7361624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7361695Z return mod(**inputs) 2025-08-14T21:41:25.7361983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7362064Z outputs = self.model( 2025-08-14T21:41:25.7362329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7362416Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7362679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7362755Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7362995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7363077Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7363390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7363507Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7363767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7363867Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7363871Z 2025-08-14T21:41:25.7363956Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7364041Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7364133Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7364215Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7364324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7364543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7364612Z return mod(**inputs) 2025-08-14T21:41:25.7364886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7364962Z outputs = self.model( 2025-08-14T21:41:25.7365225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7365311Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7365577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7365656Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7365899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7365974Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7366215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7366311Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7366545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7366647Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7366921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7367053Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7367078Z 2025-08-14T21:41:25.7367174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7367361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7367430Z return mod(**inputs) 2025-08-14T21:41:25.7367667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7367734Z outputs = self.model( 2025-08-14T21:41:25.7367978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7368068Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7368311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7368380Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7368589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7368670Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7368903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7368996Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7369273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7369366Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7369651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7369753Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7369757Z 2025-08-14T21:41:25.7369855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7370053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7370116Z return mod(**inputs) 2025-08-14T21:41:25.7370360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7370425Z outputs = self.model( 2025-08-14T21:41:25.7370665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7370744Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7370983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7371053Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7371268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7371343Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7371586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7371679Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7371915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7372003Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7372009Z 2025-08-14T21:41:25.7372108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7372306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7372370Z return mod(**inputs) 2025-08-14T21:41:25.7372610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7372684Z outputs = self.model( 2025-08-14T21:41:25.7372925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7373016Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7373304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7373373Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7373600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7373675Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7373919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7374050Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7374287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7374439Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7374442Z 2025-08-14T21:41:25.7374541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7374733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7374802Z return mod(**inputs) 2025-08-14T21:41:25.7375134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7375201Z outputs = self.model( 2025-08-14T21:41:25.7375449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7375522Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7375762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7375831Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7376040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7376133Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7376409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7376513Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7376765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7376844Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7376849Z 2025-08-14T21:41:25.7376957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7377161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7377223Z return mod(**inputs) 2025-08-14T21:41:25.7377467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7377530Z outputs = self.model( 2025-08-14T21:41:25.7377772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7377841Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7378078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7378154Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7378364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7378440Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7378679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7378781Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7379044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7379128Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7379131Z 2025-08-14T21:41:25.7379209Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7379293Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7379367Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7379442Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7379547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7379763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7379835Z return mod(**inputs) 2025-08-14T21:41:25.7380078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7380146Z outputs = self.model( 2025-08-14T21:41:25.7380402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7380472Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7380710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7380785Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7381034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7381121Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7381365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7381469Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7381715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7381811Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7382103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7382231Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7382234Z 2025-08-14T21:41:25.7382338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7382537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7382603Z return mod(**inputs) 2025-08-14T21:41:25.7382855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7382926Z outputs = self.model( 2025-08-14T21:41:25.7399635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7399876Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7400206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7400284Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7400524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7400637Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7400887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7401006Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7401247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7401350Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7401754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7401863Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7401869Z 2025-08-14T21:41:25.7401980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7402193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7402263Z return mod(**inputs) 2025-08-14T21:41:25.7402517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7402625Z outputs = self.model( 2025-08-14T21:41:25.7402863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7402944Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7403188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7403261Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7403485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7403569Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7403876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7403982Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7404225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7404311Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7404316Z 2025-08-14T21:41:25.7404425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7404642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7404711Z return mod(**inputs) 2025-08-14T21:41:25.7404965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7405054Z outputs = self.model( 2025-08-14T21:41:25.7405295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7405373Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7405623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7405696Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7405921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7406000Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7406239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7406368Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7406373Z 2025-08-14T21:41:25.7406479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7406685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7406752Z return mod(**inputs) 2025-08-14T21:41:25.7406993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7407068Z outputs = self.model( 2025-08-14T21:41:25.7407307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7407379Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7407624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7407717Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7407938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7408014Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7408251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7408377Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7408581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7408804Z return self.act(input) 2025-08-14T21:41:25.7408820Z 2025-08-14T21:41:25.7408927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7409121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7409199Z return mod(**inputs) 2025-08-14T21:41:25.7409435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7409500Z outputs = self.model( 2025-08-14T21:41:25.7409811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7409883Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7410221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7410296Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7410507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7410593Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7410828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7410912Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7410916Z 2025-08-14T21:41:25.7411021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7411207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7411272Z return mod(**inputs) 2025-08-14T21:41:25.7411507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7411573Z outputs = self.model( 2025-08-14T21:41:25.7411812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7411885Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7412121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7412196Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7412408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7412485Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7412717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7412814Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7413059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7413212Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7413217Z 2025-08-14T21:41:25.7413322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7413510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7413575Z return mod(**inputs) 2025-08-14T21:41:25.7413851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7413915Z outputs = self.model( 2025-08-14T21:41:25.7414151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7414228Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7414470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7414544Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7414776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7414849Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7415090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7415189Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7415425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7415511Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7415514Z 2025-08-14T21:41:25.7415611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7415838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7415900Z return mod(**inputs) 2025-08-14T21:41:25.7416141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7416217Z outputs = self.model( 2025-08-14T21:41:25.7416461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7416539Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7416778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7416848Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7417067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7417142Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7417383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7417487Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7417725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7417810Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7417814Z 2025-08-14T21:41:25.7417895Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7417978Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7418061Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7418135Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7418235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7418430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7418495Z return mod(**inputs) 2025-08-14T21:41:25.7418747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7418813Z outputs = self.model( 2025-08-14T21:41:25.7419056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7419136Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7419377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7419474Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7419693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7419770Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7420012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7420109Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7420344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7420470Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7420752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7420892Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7420898Z 2025-08-14T21:41:25.7420996Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7421184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7421253Z return mod(**inputs) 2025-08-14T21:41:25.7421490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7421556Z outputs = self.model( 2025-08-14T21:41:25.7421831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7421906Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7422154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7422225Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7422439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7422527Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7422773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7422874Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7423109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7423203Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7423491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7423603Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7423606Z 2025-08-14T21:41:25.7423704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7423898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7423966Z return mod(**inputs) 2025-08-14T21:41:25.7424207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7424274Z outputs = self.model( 2025-08-14T21:41:25.7424517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7424593Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7424821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7424897Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7425100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7425173Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7425435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7425529Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7425760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7425843Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7425847Z 2025-08-14T21:41:25.7425945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7426144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7426220Z return mod(**inputs) 2025-08-14T21:41:25.7426460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7426537Z outputs = self.model( 2025-08-14T21:41:25.7426773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7426844Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7427085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7427154Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7427374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7427481Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7427722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7427843Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7428079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7428235Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7428241Z 2025-08-14T21:41:25.7428344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7428539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7428610Z return mod(**inputs) 2025-08-14T21:41:25.7428854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7428935Z outputs = self.model( 2025-08-14T21:41:25.7429173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7429240Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7429475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7429542Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7429745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7429827Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7430055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7430167Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7430397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7430475Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7430480Z 2025-08-14T21:41:25.7430587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7430776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7430838Z return mod(**inputs) 2025-08-14T21:41:25.7431084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7431164Z outputs = self.model( 2025-08-14T21:41:25.7431408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7431477Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7431715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7431796Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7432014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7432117Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7432427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7432531Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7432776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7432859Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7432863Z 2025-08-14T21:41:25.7432942Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7433028Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7433102Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7433219Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7433322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7433520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7433592Z return mod(**inputs) 2025-08-14T21:41:25.7433840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7433907Z outputs = self.model( 2025-08-14T21:41:25.7434158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7434232Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7434483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7434552Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7434776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7434865Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7435131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7435245Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7435562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7435665Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7436074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7436224Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7436229Z 2025-08-14T21:41:25.7436332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7436571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7436640Z return mod(**inputs) 2025-08-14T21:41:25.7436901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7436979Z outputs = self.model( 2025-08-14T21:41:25.7437254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7437345Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7437583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7437662Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7437875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7437958Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7438198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7438301Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7438562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7438655Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7438947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7439049Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7439052Z 2025-08-14T21:41:25.7439146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7439334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7439396Z return mod(**inputs) 2025-08-14T21:41:25.7439653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7439724Z outputs = self.model( 2025-08-14T21:41:25.7439959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7440035Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7440272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7440341Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7440557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7440631Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7440869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7440978Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7441215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7441300Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7441303Z 2025-08-14T21:41:25.7441403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7441594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7441665Z return mod(**inputs) 2025-08-14T21:41:25.7441906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7441980Z outputs = self.model( 2025-08-14T21:41:25.7442226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7442294Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7442544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7442618Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7442835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7442916Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7443160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7443306Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7443310Z 2025-08-14T21:41:25.7443410Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7443604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7443677Z return mod(**inputs) 2025-08-14T21:41:25.7443925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7443998Z outputs = self.model( 2025-08-14T21:41:25.7444259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7444342Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7444585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7444652Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7444860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7444943Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7445175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7445329Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7445534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7445604Z return self.act(input) 2025-08-14T21:41:25.7445607Z 2025-08-14T21:41:25.7445716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7445902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7445966Z return mod(**inputs) 2025-08-14T21:41:25.7446211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7446277Z outputs = self.model( 2025-08-14T21:41:25.7446519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7446588Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7446829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7446904Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7447114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7447196Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7447425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7447504Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7447508Z 2025-08-14T21:41:25.7447613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7447802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7447864Z return mod(**inputs) 2025-08-14T21:41:25.7448105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7448174Z outputs = self.model( 2025-08-14T21:41:25.7448415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7448486Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7448720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7448797Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7449006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7449100Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7449341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7449437Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7449682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7449829Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7449850Z 2025-08-14T21:41:25.7449950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7450147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7450211Z return mod(**inputs) 2025-08-14T21:41:25.7450454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7450521Z outputs = self.model( 2025-08-14T21:41:25.7450755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7450833Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7451099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7451168Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7451386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7451461Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7451704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7451797Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7452034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7452119Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7452123Z 2025-08-14T21:41:25.7452220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7452418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7452483Z return mod(**inputs) 2025-08-14T21:41:25.7452722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7452795Z outputs = self.model( 2025-08-14T21:41:25.7453035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7453103Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7453347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7453415Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7453635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7453709Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7453951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7454054Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7454293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7454376Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7454386Z 2025-08-14T21:41:25.7454463Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7454538Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7454647Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7454721Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7454819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7455012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7455076Z return mod(**inputs) 2025-08-14T21:41:25.7455324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7455396Z outputs = self.model( 2025-08-14T21:41:25.7455631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7455723Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7455961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7456029Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7456246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7456323Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7456558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7456652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7456918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7457020Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7457298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7457425Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7457429Z 2025-08-14T21:41:25.7457532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7457719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7457788Z return mod(**inputs) 2025-08-14T21:41:25.7458027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7458090Z outputs = self.model( 2025-08-14T21:41:25.7458343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7458411Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7458641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7458717Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7458922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7459005Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7459233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7459324Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7459566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7459662Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7459945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7460049Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7460053Z 2025-08-14T21:41:25.7460150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7460346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7460427Z return mod(**inputs) 2025-08-14T21:41:25.7460671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7460745Z outputs = self.model( 2025-08-14T21:41:25.7460981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7461055Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7461341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7461424Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7461635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7461709Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7461949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7462042Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7462275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7462357Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7462360Z 2025-08-14T21:41:25.7462455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7462672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7462744Z return mod(**inputs) 2025-08-14T21:41:25.7462976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7463047Z outputs = self.model( 2025-08-14T21:41:25.7463277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7463346Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7463576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7463642Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7463840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7463922Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7464149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7464255Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7464482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7464623Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7464628Z 2025-08-14T21:41:25.7464731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7464912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7464977Z return mod(**inputs) 2025-08-14T21:41:25.7465211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7465273Z outputs = self.model( 2025-08-14T21:41:25.7465513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7465582Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7465812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7465886Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7466087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7466220Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7466446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7466545Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7466778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7466849Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7466852Z 2025-08-14T21:41:25.7466954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7467155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7467216Z return mod(**inputs) 2025-08-14T21:41:25.7467450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7467514Z outputs = self.model( 2025-08-14T21:41:25.7467746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7467823Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7468059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7468136Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7468375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7468454Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7468701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7468805Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7469052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7469141Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7469145Z 2025-08-14T21:41:25.7469226Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7469313Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7469392Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7469471Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7469588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7469791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7469859Z return mod(**inputs) 2025-08-14T21:41:25.7470118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7470189Z outputs = self.model( 2025-08-14T21:41:25.7470451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7470527Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7470782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7470862Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7471095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7471181Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7471433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7471543Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7471799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7471900Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7472224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7472371Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7472375Z 2025-08-14T21:41:25.7472480Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7472694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7472764Z return mod(**inputs) 2025-08-14T21:41:25.7473024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7473121Z outputs = self.model( 2025-08-14T21:41:25.7473381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7473466Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7473723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7473801Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7474043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7474127Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7474436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7474563Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7474827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7474939Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7475249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7475368Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7475371Z 2025-08-14T21:41:25.7475489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7475702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7475779Z return mod(**inputs) 2025-08-14T21:41:25.7476141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7476216Z outputs = self.model( 2025-08-14T21:41:25.7476485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7476561Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7476823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7476913Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7477147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7477240Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7477503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7477616Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7477890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7477979Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7477983Z 2025-08-14T21:41:25.7478094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7478315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7478387Z return mod(**inputs) 2025-08-14T21:41:25.7478688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7478762Z outputs = self.model( 2025-08-14T21:41:25.7479027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7479115Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7479382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7479467Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7479722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7479804Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7480077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7480207Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7480211Z 2025-08-14T21:41:25.7480321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7480537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7480605Z return mod(**inputs) 2025-08-14T21:41:25.7480908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7480980Z outputs = self.model( 2025-08-14T21:41:25.7481243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7481330Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7481591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7481667Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7481912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7481995Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7482265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7482391Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7482623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7482705Z return self.act(input) 2025-08-14T21:41:25.7482710Z 2025-08-14T21:41:25.7482819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7483035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7483104Z return mod(**inputs) 2025-08-14T21:41:25.7483367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7483448Z outputs = self.model( 2025-08-14T21:41:25.7483712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7483790Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7484062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7484143Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7484391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7484468Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7484710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7484798Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7484820Z 2025-08-14T21:41:25.7484923Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7485122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7485187Z return mod(**inputs) 2025-08-14T21:41:25.7485430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7485503Z outputs = self.model( 2025-08-14T21:41:25.7485749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7485853Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7486106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7486176Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7486402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7486479Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7486720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7486822Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7487094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7487245Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7487256Z 2025-08-14T21:41:25.7487358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7487550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7487620Z return mod(**inputs) 2025-08-14T21:41:25.7487863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7487930Z outputs = self.model( 2025-08-14T21:41:25.7488181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7488251Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7488503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7488577Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7488794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7488879Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7489121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7489217Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7489467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7489545Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7489549Z 2025-08-14T21:41:25.7489657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7489850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7489914Z return mod(**inputs) 2025-08-14T21:41:25.7490169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7490236Z outputs = self.model( 2025-08-14T21:41:25.7490478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7490559Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7490809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7490908Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7491134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7491212Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7491469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7491570Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7491826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7491937Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7491941Z 2025-08-14T21:41:25.7492021Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7492106Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7492182Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7492258Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7492367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7492561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7492632Z return mod(**inputs) 2025-08-14T21:41:25.7492875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7492974Z outputs = self.model( 2025-08-14T21:41:25.7493224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7493297Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7493544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7493623Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7493843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7493930Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7494170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7494265Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7494512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7494613Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7494913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7495045Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7495049Z 2025-08-14T21:41:25.7495150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7495359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7495425Z return mod(**inputs) 2025-08-14T21:41:25.7495680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7495747Z outputs = self.model( 2025-08-14T21:41:25.7495994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7496075Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7496373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7496449Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7496676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7496754Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7497021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7497120Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7497360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7497461Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7497747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7497873Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7497883Z 2025-08-14T21:41:25.7497983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7498174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7498247Z return mod(**inputs) 2025-08-14T21:41:25.7498492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7498558Z outputs = self.model( 2025-08-14T21:41:25.7498808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7498878Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7499170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7499243Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7499462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7499546Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7499788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:41:25.7499884Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:25.7500134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7500214Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7500217Z 2025-08-14T21:41:25.7500325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7500523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7500586Z return mod(**inputs) 2025-08-14T21:41:25.7500841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7500906Z outputs = self.model( 2025-08-14T21:41:25.7501163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7501231Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7501474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7501549Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7501760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7501834Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7502084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7502189Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7502437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:41:25.7502582Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:25.7502586Z 2025-08-14T21:41:25.7502701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7502898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7502962Z return mod(**inputs) 2025-08-14T21:41:25.7503195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7503267Z outputs = self.model( 2025-08-14T21:41:25.7503505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7503581Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7503833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7503902Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7504118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7504195Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7504434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7504537Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7504769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:41:25.7504882Z key_states = self.k_proj(current_states) 2025-08-14T21:41:25.7504886Z 2025-08-14T21:41:25.7504988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7505180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7505251Z return mod(**inputs) 2025-08-14T21:41:25.7505496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7505568Z outputs = self.model( 2025-08-14T21:41:25.7505814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7505885Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7506138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7506209Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7506434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7506511Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7506763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7506872Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7507108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:41:25.7507191Z value_states = self.v_proj(current_states) 2025-08-14T21:41:25.7507195Z 2025-08-14T21:41:25.7507279Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7507355Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7507436Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7507510Z cudagraph partition due to non gpu ops 2025-08-14T21:41:25.7507609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7507810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7507874Z return mod(**inputs) 2025-08-14T21:41:25.7508116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7508189Z outputs = self.model( 2025-08-14T21:41:25.7508428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7508525Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7508903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7508979Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7509209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7509293Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7509545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7509703Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7509943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7510049Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7510335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:25.7510474Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:25.7510478Z 2025-08-14T21:41:25.7510584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7510774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7510888Z return mod(**inputs) 2025-08-14T21:41:25.7511131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7511199Z outputs = self.model( 2025-08-14T21:41:25.7511446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7511518Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7511757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7511836Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7512047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7512128Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7512367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7512471Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7512713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:41:25.7512808Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:25.7513101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:25.7513210Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:25.7513214Z 2025-08-14T21:41:25.7513316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7513517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7513581Z return mod(**inputs) 2025-08-14T21:41:25.7513830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7513909Z outputs = self.model( 2025-08-14T21:41:25.7514165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7514253Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7514512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7514588Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7514871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7514952Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7515205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:41:25.7515325Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:41:25.7515581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:41:25.7515671Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:25.7515693Z 2025-08-14T21:41:25.7515808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7516087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7516162Z return mod(**inputs) 2025-08-14T21:41:25.7516427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7516510Z outputs = self.model( 2025-08-14T21:41:25.7516773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7516852Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7517157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7517236Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7517480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7517557Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7517789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7517914Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7517919Z 2025-08-14T21:41:25.7518028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7518243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7518322Z return mod(**inputs) 2025-08-14T21:41:25.7518592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7518675Z outputs = self.model( 2025-08-14T21:41:25.7518941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7519020Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7519295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7519370Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7519611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7519696Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7519957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:41:25.7520091Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:25.7520320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:25.7520396Z return self.act(input) 2025-08-14T21:41:25.7520400Z 2025-08-14T21:41:25.7520519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7520730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7520809Z return mod(**inputs) 2025-08-14T21:41:25.7521073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:41:25.7521166Z outputs = self.model( 2025-08-14T21:41:25.7521441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:41:25.7521519Z decoder_outputs = self.decoder( 2025-08-14T21:41:25.7521789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:41:25.7521875Z layer_outputs = decoder_layer( 2025-08-14T21:41:25.7522115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:25.7522232Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:25.7522500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:41:25.7522590Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:25.7522594Z 2025-08-14T21:41:25.7522717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7522933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7523012Z return mod(**inputs) 2025-08-14T21:41:25.7523282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1490, in forward 2025-08-14T21:41:25.7523372Z lm_logits = self.lm_head(outputs[0]) 2025-08-14T21:41:25.7523375Z 2025-08-14T21:41:25.7523507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:25.7523690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:25.7523751Z return mod(**inputs) 2025-08-14T21:41:25.7523989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1497, in forward 2025-08-14T21:41:25.7524150Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:41:25.7524155Z 2025-08-14T21:41:38.4105129Z Compilation time (from dynamo_timed): 27.069366154 2025-08-14T21:41:38.4206249Z pass 2025-08-14T21:41:38.4206729Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:38.4207619Z TIMING: _recursive_pre_grad_passes:0.01444 _recursive_joint_graph_passes:1.17517 _recursive_post_grad_passes:0.18236 async_compile.wait:0.82764 code_gen:10.88731 inductor_compile:13.92699 backend_compile:21.22305 gc:0.00085 entire_frame_compile:27.06937 total_wall_time:27.06937 2025-08-14T21:41:38.4208618Z STATS: call_* op count: 980 | FakeTensorMode.__torch_dispatch__:33505 | FakeTensor.__torch_dispatch__:11921 | ProxyTorchDispatchMode.__torch_dispatch__:12370 2025-08-14T21:41:38.4209372Z Dynamo produced 1 graphs covering 980 ops with 0 graph breaks (0 unique) 2025-08-14T21:41:44.1197948Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:41:44.1198963Z from pkg_resources import resource_filename 2025-08-14T21:41:44.7088734Z 2025-08-14T21:41:46.0957803Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:41:46.0958125Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:41:46.0973679Z cpu eval BertForMaskedLM 2025-08-14T21:41:46.5981353Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:46.8484667Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:47.0926274Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:54.7489833Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7496229Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7497767Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7498525Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7502979Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7503294Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7503544Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7503854Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7504267Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7504598Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7504947Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7505293Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7505915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7506535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7506971Z return mod(**inputs) 2025-08-14T21:41:54.7507547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7508050Z outputs = self.bert( 2025-08-14T21:41:54.7508475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7509005Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7509390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7509925Z layer_outputs = layer_module( 2025-08-14T21:41:54.7510338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7510752Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7511201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7511632Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7512130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7512769Z return func(*args, **kwargs) 2025-08-14T21:41:54.7513183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7513687Z self_outputs = self.self( 2025-08-14T21:41:54.7514103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7514527Z return func(*args, **kwargs) 2025-08-14T21:41:54.7514939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.7515675Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.7516108Z 2025-08-14T21:41:54.7516233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7516649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7517006Z return mod(**inputs) 2025-08-14T21:41:54.7517460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7517829Z outputs = self.bert( 2025-08-14T21:41:54.7518197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7518596Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7518963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7519332Z layer_outputs = layer_module( 2025-08-14T21:41:54.7519933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7520317Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7520886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7521524Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7521951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7522441Z return func(*args, **kwargs) 2025-08-14T21:41:54.7522833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7523240Z self_outputs = self.self( 2025-08-14T21:41:54.7523658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7524051Z return func(*args, **kwargs) 2025-08-14T21:41:54.7524433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.7524834Z self.key(current_states) 2025-08-14T21:41:54.7524967Z 2025-08-14T21:41:54.7525087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7525654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7526054Z return mod(**inputs) 2025-08-14T21:41:54.7527068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7527675Z outputs = self.bert( 2025-08-14T21:41:54.7528246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7528888Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7529500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7529934Z layer_outputs = layer_module( 2025-08-14T21:41:54.7530305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7530676Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7531078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7531498Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7531908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7532291Z return func(*args, **kwargs) 2025-08-14T21:41:54.7532680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7533154Z self_outputs = self.self( 2025-08-14T21:41:54.7533545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7533940Z return func(*args, **kwargs) 2025-08-14T21:41:54.7534330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.7534747Z self.value(current_states) 2025-08-14T21:41:54.7534876Z 2025-08-14T21:41:54.7534963Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7535225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7535614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7535951Z return mod(**inputs) 2025-08-14T21:41:54.7536331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7536733Z outputs = self.bert( 2025-08-14T21:41:54.7537115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7537527Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7537950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7538352Z layer_outputs = layer_module( 2025-08-14T21:41:54.7538712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7539097Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7539508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7539916Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7540329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7540729Z return func(*args, **kwargs) 2025-08-14T21:41:54.7541121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7541519Z self_outputs = self.self( 2025-08-14T21:41:54.7541890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7542278Z return func(*args, **kwargs) 2025-08-14T21:41:54.7542656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.7543146Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.7543352Z 2025-08-14T21:41:54.7543465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7543856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7544200Z return mod(**inputs) 2025-08-14T21:41:54.7544573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7544977Z outputs = self.bert( 2025-08-14T21:41:54.7545354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7545748Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7546130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7546532Z layer_outputs = layer_module( 2025-08-14T21:41:54.7546980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7547360Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7547768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7548180Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7548585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7548963Z return func(*args, **kwargs) 2025-08-14T21:41:54.7549330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.7549768Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.7550192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.7550585Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7550731Z 2025-08-14T21:41:54.7550837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7551205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7551539Z return mod(**inputs) 2025-08-14T21:41:54.7551918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7552338Z outputs = self.bert( 2025-08-14T21:41:54.7552704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7553121Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7553520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7553929Z layer_outputs = layer_module( 2025-08-14T21:41:54.7554291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7554670Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7555094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7555516Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7556323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7556771Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7557223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7557721Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7558246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.7558646Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7558788Z 2025-08-14T21:41:54.7558902Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7559265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7559594Z return mod(**inputs) 2025-08-14T21:41:54.7559955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7560331Z outputs = self.bert( 2025-08-14T21:41:54.7560688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7561076Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7561455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7561830Z layer_outputs = layer_module( 2025-08-14T21:41:54.7562200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7562558Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7562935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7563320Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7563720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7564115Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7564530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7565015Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7565438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.7565876Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.7566272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.7566633Z return self.act(input) 2025-08-14T21:41:54.7566752Z 2025-08-14T21:41:54.7566870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7567247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7567608Z return mod(**inputs) 2025-08-14T21:41:54.7567986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7568382Z outputs = self.bert( 2025-08-14T21:41:54.7568750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7569154Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7569550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7569979Z layer_outputs = layer_module( 2025-08-14T21:41:54.7570338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7570717Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7571118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7571527Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7571951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7572367Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7572833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.7573328Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.7573794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.7574209Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7574353Z 2025-08-14T21:41:54.7574471Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7574851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7575197Z return mod(**inputs) 2025-08-14T21:41:54.7575577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7575969Z outputs = self.bert( 2025-08-14T21:41:54.7576355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7576757Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7577132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7577507Z layer_outputs = layer_module( 2025-08-14T21:41:54.7577852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7578217Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7578592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7578988Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7579372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7579746Z return func(*args, **kwargs) 2025-08-14T21:41:54.7580110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7580485Z self_outputs = self.self( 2025-08-14T21:41:54.7580847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7581213Z return func(*args, **kwargs) 2025-08-14T21:41:54.7581581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.7582126Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.7582393Z 2025-08-14T21:41:54.7582505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7582871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7583201Z return mod(**inputs) 2025-08-14T21:41:54.7583563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7583952Z outputs = self.bert( 2025-08-14T21:41:54.7584321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7584703Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7585078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7585449Z layer_outputs = layer_module( 2025-08-14T21:41:54.7585803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7586183Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7586588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7586990Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7587429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7587823Z return func(*args, **kwargs) 2025-08-14T21:41:54.7588179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7588556Z self_outputs = self.self( 2025-08-14T21:41:54.7588913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7589285Z return func(*args, **kwargs) 2025-08-14T21:41:54.7589643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.7590018Z self.key(current_states) 2025-08-14T21:41:54.7590135Z 2025-08-14T21:41:54.7590248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7590623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7590939Z return mod(**inputs) 2025-08-14T21:41:54.7591293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7591669Z outputs = self.bert( 2025-08-14T21:41:54.7592039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7592449Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7592827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7593208Z layer_outputs = layer_module( 2025-08-14T21:41:54.7593545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7593903Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7594297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7594696Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7595099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7595502Z return func(*args, **kwargs) 2025-08-14T21:41:54.7595996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7596435Z self_outputs = self.self( 2025-08-14T21:41:54.7596819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7597238Z return func(*args, **kwargs) 2025-08-14T21:41:54.7597640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.7598030Z self.value(current_states) 2025-08-14T21:41:54.7598159Z 2025-08-14T21:41:54.7598243Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7598482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7598855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7599176Z return mod(**inputs) 2025-08-14T21:41:54.7599526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7599892Z outputs = self.bert( 2025-08-14T21:41:54.7600239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7600626Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7600999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7601370Z layer_outputs = layer_module( 2025-08-14T21:41:54.7601748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7602115Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7602499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7602889Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7603304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7603677Z return func(*args, **kwargs) 2025-08-14T21:41:54.7604036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7604420Z self_outputs = self.self( 2025-08-14T21:41:54.7604787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7605162Z return func(*args, **kwargs) 2025-08-14T21:41:54.7605521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.7605964Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.7606148Z 2025-08-14T21:41:54.7606263Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7606619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7606949Z return mod(**inputs) 2025-08-14T21:41:54.7607304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7607679Z outputs = self.bert( 2025-08-14T21:41:54.7608028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7608411Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7608962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7609343Z layer_outputs = layer_module( 2025-08-14T21:41:54.7609691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7610072Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7610478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7610907Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7611357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7611738Z return func(*args, **kwargs) 2025-08-14T21:41:54.7612129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.7612590Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.7613031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.7613494Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7613639Z 2025-08-14T21:41:54.7613748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7614130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7614474Z return mod(**inputs) 2025-08-14T21:41:54.7614851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7615239Z outputs = self.bert( 2025-08-14T21:41:54.7615742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7616163Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7616643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7617045Z layer_outputs = layer_module( 2025-08-14T21:41:54.7617414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7617812Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7618204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7618623Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7619049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7619473Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7619903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7620382Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7620827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.7621232Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7621389Z 2025-08-14T21:41:54.7621501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7621880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7622221Z return mod(**inputs) 2025-08-14T21:41:54.7622589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7622985Z outputs = self.bert( 2025-08-14T21:41:54.7623355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7623765Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7624176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7624596Z layer_outputs = layer_module( 2025-08-14T21:41:54.7624941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7625308Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7625676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7626071Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7626460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7626837Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7627240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7627697Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7628132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.7628538Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.7628907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.7629251Z return self.act(input) 2025-08-14T21:41:54.7629364Z 2025-08-14T21:41:54.7629468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7629829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7630160Z return mod(**inputs) 2025-08-14T21:41:54.7630548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7630924Z outputs = self.bert( 2025-08-14T21:41:54.7631280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7631663Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7632029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7632406Z layer_outputs = layer_module( 2025-08-14T21:41:54.7632756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7633146Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7633542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7633958Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7634386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7634794Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7635222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.7635712Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.7636243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.7636660Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7636815Z 2025-08-14T21:41:54.7636928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7637313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7637626Z return mod(**inputs) 2025-08-14T21:41:54.7637970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7638336Z outputs = self.bert( 2025-08-14T21:41:54.7638680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7639048Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7639421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7639810Z layer_outputs = layer_module( 2025-08-14T21:41:54.7640148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7640545Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7640919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7641307Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7641687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7642051Z return func(*args, **kwargs) 2025-08-14T21:41:54.7642441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7642823Z self_outputs = self.self( 2025-08-14T21:41:54.7643166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7643532Z return func(*args, **kwargs) 2025-08-14T21:41:54.7643888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.7644379Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.7644658Z 2025-08-14T21:41:54.7644768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7645218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7645565Z return mod(**inputs) 2025-08-14T21:41:54.7645952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7646331Z outputs = self.bert( 2025-08-14T21:41:54.7646695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7647078Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7647497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7647905Z layer_outputs = layer_module( 2025-08-14T21:41:54.7648273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7648653Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7649059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7649476Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7649885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7650272Z return func(*args, **kwargs) 2025-08-14T21:41:54.7650660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7651059Z self_outputs = self.self( 2025-08-14T21:41:54.7651434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7651829Z return func(*args, **kwargs) 2025-08-14T21:41:54.7652216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.7652612Z self.key(current_states) 2025-08-14T21:41:54.7652734Z 2025-08-14T21:41:54.7652843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7653223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7653548Z return mod(**inputs) 2025-08-14T21:41:54.7653884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7654265Z outputs = self.bert( 2025-08-14T21:41:54.7654611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7654987Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7655344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7655711Z layer_outputs = layer_module( 2025-08-14T21:41:54.7656051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7656402Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7656786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7657168Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7657545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7657922Z return func(*args, **kwargs) 2025-08-14T21:41:54.7658268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7658631Z self_outputs = self.self( 2025-08-14T21:41:54.7658978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7659363Z return func(*args, **kwargs) 2025-08-14T21:41:54.7659724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.7660096Z self.value(current_states) 2025-08-14T21:41:54.7660211Z 2025-08-14T21:41:54.7660302Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7660532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7660874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7661187Z return mod(**inputs) 2025-08-14T21:41:54.7661517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7661880Z outputs = self.bert( 2025-08-14T21:41:54.7662215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7662567Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7662929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7663290Z layer_outputs = layer_module( 2025-08-14T21:41:54.7663616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7663948Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7664307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7664677Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7665037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7665385Z return func(*args, **kwargs) 2025-08-14T21:41:54.7665739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7666111Z self_outputs = self.self( 2025-08-14T21:41:54.7666463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7666835Z return func(*args, **kwargs) 2025-08-14T21:41:54.7667205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.7667658Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.7667859Z 2025-08-14T21:41:54.7667961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7668310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7668623Z return mod(**inputs) 2025-08-14T21:41:54.7668963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7669325Z outputs = self.bert( 2025-08-14T21:41:54.7669669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7670115Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7670467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7670837Z layer_outputs = layer_module( 2025-08-14T21:41:54.7671183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7671534Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7671911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7672300Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7672679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7673073Z return func(*args, **kwargs) 2025-08-14T21:41:54.7673450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.7673917Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.7674387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.7674808Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7674964Z 2025-08-14T21:41:54.7675074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7675459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7675796Z return mod(**inputs) 2025-08-14T21:41:54.7676250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7676682Z outputs = self.bert( 2025-08-14T21:41:54.7677069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7677470Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7677865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7678284Z layer_outputs = layer_module( 2025-08-14T21:41:54.7678613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7678964Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7679334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7679714Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7680100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7680481Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7680882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7681328Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7681731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.7682131Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7682264Z 2025-08-14T21:41:54.7682373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7682714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7683030Z return mod(**inputs) 2025-08-14T21:41:54.7683379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7683744Z outputs = self.bert( 2025-08-14T21:41:54.7684083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7684475Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7684837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7685205Z layer_outputs = layer_module( 2025-08-14T21:41:54.7685551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7685911Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7686291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7686674Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7687110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7687493Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7687889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7688320Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7688731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.7689156Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.7689527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.7689865Z return self.act(input) 2025-08-14T21:41:54.7689985Z 2025-08-14T21:41:54.7690090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7690451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7690766Z return mod(**inputs) 2025-08-14T21:41:54.7691125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7691503Z outputs = self.bert( 2025-08-14T21:41:54.7691847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7692232Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7692612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7693332Z layer_outputs = layer_module( 2025-08-14T21:41:54.7693701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7694091Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7694505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7694921Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7695343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7695773Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7696182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.7696674Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.7697106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.7697507Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7697645Z 2025-08-14T21:41:54.7697756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7698109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7698433Z return mod(**inputs) 2025-08-14T21:41:54.7698808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7699180Z outputs = self.bert( 2025-08-14T21:41:54.7699533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7699940Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7700331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7700811Z layer_outputs = layer_module( 2025-08-14T21:41:54.7701163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7701555Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7702002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7702409Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7702821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7703217Z return func(*args, **kwargs) 2025-08-14T21:41:54.7703601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7704010Z self_outputs = self.self( 2025-08-14T21:41:54.7704401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7704804Z return func(*args, **kwargs) 2025-08-14T21:41:54.7705186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.7705734Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.7706011Z 2025-08-14T21:41:54.7706128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7706513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7706860Z return mod(**inputs) 2025-08-14T21:41:54.7707254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7707668Z outputs = self.bert( 2025-08-14T21:41:54.7708049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7708470Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7709009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7709431Z layer_outputs = layer_module( 2025-08-14T21:41:54.7709798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7710187Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7710596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7711013Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7711429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7711907Z return func(*args, **kwargs) 2025-08-14T21:41:54.7712291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7712690Z self_outputs = self.self( 2025-08-14T21:41:54.7713082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7713471Z return func(*args, **kwargs) 2025-08-14T21:41:54.7713855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.7714287Z self.key(current_states) 2025-08-14T21:41:54.7714416Z 2025-08-14T21:41:54.7714530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7714913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7715249Z return mod(**inputs) 2025-08-14T21:41:54.7715635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7716213Z outputs = self.bert( 2025-08-14T21:41:54.7716624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7717036Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7717494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7717900Z layer_outputs = layer_module( 2025-08-14T21:41:54.7718259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7718650Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7719057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7719468Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7719866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7720268Z return func(*args, **kwargs) 2025-08-14T21:41:54.7720656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7721063Z self_outputs = self.self( 2025-08-14T21:41:54.7721449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7721845Z return func(*args, **kwargs) 2025-08-14T21:41:54.7722232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.7722624Z self.value(current_states) 2025-08-14T21:41:54.7722760Z 2025-08-14T21:41:54.7722847Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7723101Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7723484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7723811Z return mod(**inputs) 2025-08-14T21:41:54.7724166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7724541Z outputs = self.bert( 2025-08-14T21:41:54.7724895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7725270Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7725634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7725996Z layer_outputs = layer_module( 2025-08-14T21:41:54.7726331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7726722Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7727094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7727468Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7727841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7728204Z return func(*args, **kwargs) 2025-08-14T21:41:54.7728570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7728958Z self_outputs = self.self( 2025-08-14T21:41:54.7729317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7729688Z return func(*args, **kwargs) 2025-08-14T21:41:54.7730046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.7730482Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.7730672Z 2025-08-14T21:41:54.7730777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7731135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7731490Z return mod(**inputs) 2025-08-14T21:41:54.7731845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7732222Z outputs = self.bert( 2025-08-14T21:41:54.7732566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7732953Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7733331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7733716Z layer_outputs = layer_module( 2025-08-14T21:41:54.7734058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7734413Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7734796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7735189Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7735548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7735909Z return func(*args, **kwargs) 2025-08-14T21:41:54.7736265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.7736675Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.7737090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.7737472Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7737604Z 2025-08-14T21:41:54.7737713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7738057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7738371Z return mod(**inputs) 2025-08-14T21:41:54.7738716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7739081Z outputs = self.bert( 2025-08-14T21:41:54.7739433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7739814Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7740203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7740570Z layer_outputs = layer_module( 2025-08-14T21:41:54.7740909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7741266Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7741639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7742035Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7742435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7742856Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7743265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7743730Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7744163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.7744565Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7744707Z 2025-08-14T21:41:54.7744813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7745215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7745540Z return mod(**inputs) 2025-08-14T21:41:54.7745887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7746261Z outputs = self.bert( 2025-08-14T21:41:54.7746615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7747003Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7747375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7747754Z layer_outputs = layer_module( 2025-08-14T21:41:54.7748099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7748461Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7748875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7749269Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7749670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7750054Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7750455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7750910Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7751386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.7751797Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.7752176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.7752515Z return self.act(input) 2025-08-14T21:41:54.7752627Z 2025-08-14T21:41:54.7752730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7753091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7753421Z return mod(**inputs) 2025-08-14T21:41:54.7753777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7754184Z outputs = self.bert( 2025-08-14T21:41:54.7754567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7754978Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7755385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7755879Z layer_outputs = layer_module( 2025-08-14T21:41:54.7756352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7756745Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7757176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7757586Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7758005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7758400Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7758799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.7759288Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.7759781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.7760203Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7760360Z 2025-08-14T21:41:54.7760474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7760857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7761205Z return mod(**inputs) 2025-08-14T21:41:54.7761573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7761975Z outputs = self.bert( 2025-08-14T21:41:54.7762354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7762764Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7763151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7763555Z layer_outputs = layer_module( 2025-08-14T21:41:54.7763920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7764293Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7764700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7765111Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7765516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7765903Z return func(*args, **kwargs) 2025-08-14T21:41:54.7766290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7766671Z self_outputs = self.self( 2025-08-14T21:41:54.7767029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7767398Z return func(*args, **kwargs) 2025-08-14T21:41:54.7767767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.7768279Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.7768539Z 2025-08-14T21:41:54.7768643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7769031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7769360Z return mod(**inputs) 2025-08-14T21:41:54.7769721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7770099Z outputs = self.bert( 2025-08-14T21:41:54.7770459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7770859Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7771229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7771615Z layer_outputs = layer_module( 2025-08-14T21:41:54.7771952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7772302Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7772671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7773053Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7773425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7773786Z return func(*args, **kwargs) 2025-08-14T21:41:54.7774192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7774556Z self_outputs = self.self( 2025-08-14T21:41:54.7774933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7775288Z return func(*args, **kwargs) 2025-08-14T21:41:54.7775641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.7776202Z self.key(current_states) 2025-08-14T21:41:54.7776351Z 2025-08-14T21:41:54.7776461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7776952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7777301Z return mod(**inputs) 2025-08-14T21:41:54.7777677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7778069Z outputs = self.bert( 2025-08-14T21:41:54.7778448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7778861Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7779265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7779668Z layer_outputs = layer_module( 2025-08-14T21:41:54.7780045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7780446Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7780843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7781256Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7781637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7782006Z return func(*args, **kwargs) 2025-08-14T21:41:54.7782376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7782785Z self_outputs = self.self( 2025-08-14T21:41:54.7783179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7783582Z return func(*args, **kwargs) 2025-08-14T21:41:54.7784012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.7784440Z self.value(current_states) 2025-08-14T21:41:54.7784569Z 2025-08-14T21:41:54.7784667Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7784927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7785323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7785675Z return mod(**inputs) 2025-08-14T21:41:54.7786056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7786484Z outputs = self.bert( 2025-08-14T21:41:54.7786868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7787291Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7787695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7788104Z layer_outputs = layer_module( 2025-08-14T21:41:54.7788475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7788871Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7789319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7789917Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7790399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7790807Z return func(*args, **kwargs) 2025-08-14T21:41:54.7791244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7791670Z self_outputs = self.self( 2025-08-14T21:41:54.7792073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7792476Z return func(*args, **kwargs) 2025-08-14T21:41:54.7792888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.7793380Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.7793585Z 2025-08-14T21:41:54.7793758Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7794271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7794625Z return mod(**inputs) 2025-08-14T21:41:54.7795012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7795412Z outputs = self.bert( 2025-08-14T21:41:54.7795876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7796323Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7796730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7797145Z layer_outputs = layer_module( 2025-08-14T21:41:54.7797539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7797946Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7798369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7798808Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7799235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7799688Z return func(*args, **kwargs) 2025-08-14T21:41:54.7800076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.7800536Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.7800988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.7801408Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7801564Z 2025-08-14T21:41:54.7801674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7802075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7802420Z return mod(**inputs) 2025-08-14T21:41:54.7802789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7803187Z outputs = self.bert( 2025-08-14T21:41:54.7803566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7803967Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7804365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7804772Z layer_outputs = layer_module( 2025-08-14T21:41:54.7805169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7805545Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7805951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7806372Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7806797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7807213Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7807647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7808132Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7808577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.7809324Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7809485Z 2025-08-14T21:41:54.7809597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7809980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7810316Z return mod(**inputs) 2025-08-14T21:41:54.7810697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7811095Z outputs = self.bert( 2025-08-14T21:41:54.7811465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7811871Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7812271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7812671Z layer_outputs = layer_module( 2025-08-14T21:41:54.7813032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7813413Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7813792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7814184Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7814576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7815013Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7815417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7815862Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7816287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.7816706Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.7817089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.7817451Z return self.act(input) 2025-08-14T21:41:54.7817574Z 2025-08-14T21:41:54.7817680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7818043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7818359Z return mod(**inputs) 2025-08-14T21:41:54.7818713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7819089Z outputs = self.bert( 2025-08-14T21:41:54.7819443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7819817Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7820242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7820624Z layer_outputs = layer_module( 2025-08-14T21:41:54.7820981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7821339Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7821732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7822135Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7822536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7822947Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7823376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.7823871Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.7824332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.7824758Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7824908Z 2025-08-14T21:41:54.7825035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7825415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7825751Z return mod(**inputs) 2025-08-14T21:41:54.7826103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7826481Z outputs = self.bert( 2025-08-14T21:41:54.7826835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7827225Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7827607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7827992Z layer_outputs = layer_module( 2025-08-14T21:41:54.7828343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7828699Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7829088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7829461Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7829835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7830198Z return func(*args, **kwargs) 2025-08-14T21:41:54.7830560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7830930Z self_outputs = self.self( 2025-08-14T21:41:54.7831292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7831683Z return func(*args, **kwargs) 2025-08-14T21:41:54.7832041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.7832558Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.7832837Z 2025-08-14T21:41:54.7832941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7833300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7833619Z return mod(**inputs) 2025-08-14T21:41:54.7834041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7834437Z outputs = self.bert( 2025-08-14T21:41:54.7834823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7835218Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7835630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7836142Z layer_outputs = layer_module( 2025-08-14T21:41:54.7836507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7836892Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7837288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7837676Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7838083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7838479Z return func(*args, **kwargs) 2025-08-14T21:41:54.7838847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7839218Z self_outputs = self.self( 2025-08-14T21:41:54.7839574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7839951Z return func(*args, **kwargs) 2025-08-14T21:41:54.7840308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.7840666Z self.key(current_states) 2025-08-14T21:41:54.7840785Z 2025-08-14T21:41:54.7840894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7841236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7841544Z return mod(**inputs) 2025-08-14T21:41:54.7841872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7842224Z outputs = self.bert( 2025-08-14T21:41:54.7842561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7842915Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7843298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7843665Z layer_outputs = layer_module( 2025-08-14T21:41:54.7844000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7844341Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7844715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7845090Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7845470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7845828Z return func(*args, **kwargs) 2025-08-14T21:41:54.7846181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7846544Z self_outputs = self.self( 2025-08-14T21:41:54.7846884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7847246Z return func(*args, **kwargs) 2025-08-14T21:41:54.7847603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.7847966Z self.value(current_states) 2025-08-14T21:41:54.7848090Z 2025-08-14T21:41:54.7848206Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7848456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7848821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7849127Z return mod(**inputs) 2025-08-14T21:41:54.7849479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7849853Z outputs = self.bert( 2025-08-14T21:41:54.7850203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7850564Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7850916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7851275Z layer_outputs = layer_module( 2025-08-14T21:41:54.7851595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7851934Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7852301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7852665Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7853016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7853368Z return func(*args, **kwargs) 2025-08-14T21:41:54.7853713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7854061Z self_outputs = self.self( 2025-08-14T21:41:54.7854398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7854747Z return func(*args, **kwargs) 2025-08-14T21:41:54.7855094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.7855510Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.7855698Z 2025-08-14T21:41:54.7855798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7856148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7856464Z return mod(**inputs) 2025-08-14T21:41:54.7856823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7857175Z outputs = self.bert( 2025-08-14T21:41:54.7857513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7857874Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7858240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7858607Z layer_outputs = layer_module( 2025-08-14T21:41:54.7858951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7859302Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7859679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7860054Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7860406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7860766Z return func(*args, **kwargs) 2025-08-14T21:41:54.7861124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.7861585Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.7861993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.7862373Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7862506Z 2025-08-14T21:41:54.7862615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7862954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7863274Z return mod(**inputs) 2025-08-14T21:41:54.7863619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7863984Z outputs = self.bert( 2025-08-14T21:41:54.7864321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7864692Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7865061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7865432Z layer_outputs = layer_module( 2025-08-14T21:41:54.7865794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7866177Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7866575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7866978Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7867391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7867772Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7868164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7868596Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7869019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.7869410Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7869548Z 2025-08-14T21:41:54.7869653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7870014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7870364Z return mod(**inputs) 2025-08-14T21:41:54.7870721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7871089Z outputs = self.bert( 2025-08-14T21:41:54.7871446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7871857Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7872249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7872669Z layer_outputs = layer_module( 2025-08-14T21:41:54.7873034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7873426Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7873836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7874265Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7874693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7875115Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7875607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7876305Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7876776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.7877254Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.7877677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.7878047Z return self.act(input) 2025-08-14T21:41:54.7878169Z 2025-08-14T21:41:54.7878287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7878665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7879009Z return mod(**inputs) 2025-08-14T21:41:54.7879397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7879810Z outputs = self.bert( 2025-08-14T21:41:54.7880178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7880588Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7880988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7881393Z layer_outputs = layer_module( 2025-08-14T21:41:54.7881767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7882159Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7882566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7882981Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7883407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7883823Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7884302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.7884796Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.7885252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.7885695Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7885840Z 2025-08-14T21:41:54.7885950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7886329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7886673Z return mod(**inputs) 2025-08-14T21:41:54.7887053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7887444Z outputs = self.bert( 2025-08-14T21:41:54.7887820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7888244Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7888617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7889024Z layer_outputs = layer_module( 2025-08-14T21:41:54.7889387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7889749Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7890126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7890526Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7890963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7891347Z return func(*args, **kwargs) 2025-08-14T21:41:54.7891739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7892143Z self_outputs = self.self( 2025-08-14T21:41:54.7892524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7892910Z return func(*args, **kwargs) 2025-08-14T21:41:54.7893297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.7893848Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.7894122Z 2025-08-14T21:41:54.7894241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7894620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7894965Z return mod(**inputs) 2025-08-14T21:41:54.7895340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7895728Z outputs = self.bert( 2025-08-14T21:41:54.7896102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7896501Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7896892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7897280Z layer_outputs = layer_module( 2025-08-14T21:41:54.7897644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7898021Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7898418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7898824Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7899230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7899305Z return func(*args, **kwargs) 2025-08-14T21:41:54.7899568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7899664Z self_outputs = self.self( 2025-08-14T21:41:54.7899917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7899998Z return func(*args, **kwargs) 2025-08-14T21:41:54.7900254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.7900338Z self.key(current_states) 2025-08-14T21:41:54.7900342Z 2025-08-14T21:41:54.7900453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7900684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7900760Z return mod(**inputs) 2025-08-14T21:41:54.7901026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7901093Z outputs = self.bert( 2025-08-14T21:41:54.7901366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7901442Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7901708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7901785Z layer_outputs = layer_module( 2025-08-14T21:41:54.7902088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7902185Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7902442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7902529Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7902787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7902862Z return func(*args, **kwargs) 2025-08-14T21:41:54.7903123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7903196Z self_outputs = self.self( 2025-08-14T21:41:54.7903442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7903521Z return func(*args, **kwargs) 2025-08-14T21:41:54.7903778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.7903863Z self.value(current_states) 2025-08-14T21:41:54.7903867Z 2025-08-14T21:41:54.7903954Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7904066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7904283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7904353Z return mod(**inputs) 2025-08-14T21:41:54.7904612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7904688Z outputs = self.bert( 2025-08-14T21:41:54.7904946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7905033Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7905294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7905369Z layer_outputs = layer_module( 2025-08-14T21:41:54.7905606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7905690Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7905942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7906058Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7906309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7906389Z return func(*args, **kwargs) 2025-08-14T21:41:54.7906643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7906719Z self_outputs = self.self( 2025-08-14T21:41:54.7906977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7907070Z return func(*args, **kwargs) 2025-08-14T21:41:54.7907325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.7907475Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.7907481Z 2025-08-14T21:41:54.7907592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7907806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7907872Z return mod(**inputs) 2025-08-14T21:41:54.7908131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7908207Z outputs = self.bert( 2025-08-14T21:41:54.7908495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7908582Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7909130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7909244Z layer_outputs = layer_module( 2025-08-14T21:41:54.7909488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7909575Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7909829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7909924Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7910175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7910259Z return func(*args, **kwargs) 2025-08-14T21:41:54.7910514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.7910656Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.7910919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.7911010Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7911016Z 2025-08-14T21:41:54.7911133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7911340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7911409Z return mod(**inputs) 2025-08-14T21:41:54.7911676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7911747Z outputs = self.bert( 2025-08-14T21:41:54.7912010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7912096Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7912353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7912434Z layer_outputs = layer_module( 2025-08-14T21:41:54.7912664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7912793Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7913055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7913146Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7913423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7913518Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7913819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7913985Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7914249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.7914339Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7914345Z 2025-08-14T21:41:54.7914465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7914684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7914762Z return mod(**inputs) 2025-08-14T21:41:54.7915032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7915153Z outputs = self.bert( 2025-08-14T21:41:54.7915435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7915517Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7915788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7915921Z layer_outputs = layer_module( 2025-08-14T21:41:54.7916167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7916261Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7916536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7916627Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7916925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7917009Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7917323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7917456Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7917732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.7917865Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.7918091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.7918164Z return self.act(input) 2025-08-14T21:41:54.7918168Z 2025-08-14T21:41:54.7918286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7918497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7918572Z return mod(**inputs) 2025-08-14T21:41:54.7918847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7918917Z outputs = self.bert( 2025-08-14T21:41:54.7919190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7919268Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7919581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7919661Z layer_outputs = layer_module( 2025-08-14T21:41:54.7919899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7919988Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7920253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7920336Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7920617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7920694Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7920976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.7921110Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.7921353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.7921442Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7921446Z 2025-08-14T21:41:54.7921548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7921782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7921849Z return mod(**inputs) 2025-08-14T21:41:54.7922093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7922166Z outputs = self.bert( 2025-08-14T21:41:54.7922409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7922482Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7922732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7922801Z layer_outputs = layer_module( 2025-08-14T21:41:54.7923027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7923104Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7923347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7923436Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7923677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7923748Z return func(*args, **kwargs) 2025-08-14T21:41:54.7923996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7924067Z self_outputs = self.self( 2025-08-14T21:41:54.7924310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7924382Z return func(*args, **kwargs) 2025-08-14T21:41:54.7924628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.7924847Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.7924850Z 2025-08-14T21:41:54.7924953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7925156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7925221Z return mod(**inputs) 2025-08-14T21:41:54.7925468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7925560Z outputs = self.bert( 2025-08-14T21:41:54.7925813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7925885Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7926140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7926210Z layer_outputs = layer_module( 2025-08-14T21:41:54.7926445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7926543Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7926789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7926877Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7927105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7927174Z return func(*args, **kwargs) 2025-08-14T21:41:54.7927416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7927484Z self_outputs = self.self( 2025-08-14T21:41:54.7927719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7927823Z return func(*args, **kwargs) 2025-08-14T21:41:54.7928058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.7928135Z self.key(current_states) 2025-08-14T21:41:54.7928138Z 2025-08-14T21:41:54.7928237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7928434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7928499Z return mod(**inputs) 2025-08-14T21:41:54.7928738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7928807Z outputs = self.bert( 2025-08-14T21:41:54.7929046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7929115Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7929360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7929428Z layer_outputs = layer_module( 2025-08-14T21:41:54.7929648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7929725Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7929959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7930048Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7930278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7930343Z return func(*args, **kwargs) 2025-08-14T21:41:54.7930589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7930657Z self_outputs = self.self( 2025-08-14T21:41:54.7930895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7930963Z return func(*args, **kwargs) 2025-08-14T21:41:54.7931197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.7931276Z self.value(current_states) 2025-08-14T21:41:54.7931279Z 2025-08-14T21:41:54.7931359Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7931481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7931678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7931742Z return mod(**inputs) 2025-08-14T21:41:54.7931988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7932052Z outputs = self.bert( 2025-08-14T21:41:54.7932298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7932396Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7932640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7932716Z layer_outputs = layer_module( 2025-08-14T21:41:54.7932941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7933019Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7933270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7933350Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7933591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7933700Z return func(*args, **kwargs) 2025-08-14T21:41:54.7933953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7934028Z self_outputs = self.self( 2025-08-14T21:41:54.7934258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7934324Z return func(*args, **kwargs) 2025-08-14T21:41:54.7934568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.7934695Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.7934699Z 2025-08-14T21:41:54.7934797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7934996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7935057Z return mod(**inputs) 2025-08-14T21:41:54.7935308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7935374Z outputs = self.bert( 2025-08-14T21:41:54.7935612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7935689Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7935925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7936002Z layer_outputs = layer_module( 2025-08-14T21:41:54.7936215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7936290Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7936533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7936613Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7936845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7936921Z return func(*args, **kwargs) 2025-08-14T21:41:54.7937157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.7937293Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.7937551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.7937633Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7937636Z 2025-08-14T21:41:54.7937742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7937935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7938003Z return mod(**inputs) 2025-08-14T21:41:54.7938249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7938329Z outputs = self.bert( 2025-08-14T21:41:54.7938581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7938652Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7938893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7938972Z layer_outputs = layer_module( 2025-08-14T21:41:54.7939187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7939280Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7939560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7939642Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7939892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7939967Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7940228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7940354Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7940589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.7940678Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7940682Z 2025-08-14T21:41:54.7940782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7940970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7941043Z return mod(**inputs) 2025-08-14T21:41:54.7941278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7941353Z outputs = self.bert( 2025-08-14T21:41:54.7941587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7941670Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7941920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7941993Z layer_outputs = layer_module( 2025-08-14T21:41:54.7942207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7942294Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7942533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7942624Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7942881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7942960Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7943241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7943375Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7943626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.7943735Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.7943937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.7944014Z return self.act(input) 2025-08-14T21:41:54.7944017Z 2025-08-14T21:41:54.7944117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7944325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7944397Z return mod(**inputs) 2025-08-14T21:41:54.7944634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7944703Z outputs = self.bert( 2025-08-14T21:41:54.7944945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7945018Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7945268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7945338Z layer_outputs = layer_module( 2025-08-14T21:41:54.7945586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7945672Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7945913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7946003Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7946254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7946333Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7946612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.7946742Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.7946992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.7947076Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7947080Z 2025-08-14T21:41:54.7947180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7947382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7947445Z return mod(**inputs) 2025-08-14T21:41:54.7947689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7947763Z outputs = self.bert( 2025-08-14T21:41:54.7948012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7948087Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7948320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7948387Z layer_outputs = layer_module( 2025-08-14T21:41:54.7948615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7948691Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7948916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7949000Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7949234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7949331Z return func(*args, **kwargs) 2025-08-14T21:41:54.7949575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7949648Z self_outputs = self.self( 2025-08-14T21:41:54.7949905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7949982Z return func(*args, **kwargs) 2025-08-14T21:41:54.7950242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.7950479Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.7950483Z 2025-08-14T21:41:54.7950591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7950804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7950875Z return mod(**inputs) 2025-08-14T21:41:54.7951135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7951211Z outputs = self.bert( 2025-08-14T21:41:54.7951473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7951589Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7951845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7951921Z layer_outputs = layer_module( 2025-08-14T21:41:54.7952155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7952235Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7952494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7952583Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7952831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7952912Z return func(*args, **kwargs) 2025-08-14T21:41:54.7953169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7953241Z self_outputs = self.self( 2025-08-14T21:41:54.7953500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7953577Z return func(*args, **kwargs) 2025-08-14T21:41:54.7953845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.7953922Z self.key(current_states) 2025-08-14T21:41:54.7953927Z 2025-08-14T21:41:54.7954040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7954262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7954333Z return mod(**inputs) 2025-08-14T21:41:54.7954596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7954674Z outputs = self.bert( 2025-08-14T21:41:54.7955083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7955502Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7955992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7956485Z layer_outputs = layer_module( 2025-08-14T21:41:54.7956903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7957327Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7957738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7958162Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7958576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7958973Z return func(*args, **kwargs) 2025-08-14T21:41:54.7959377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7959813Z self_outputs = self.self( 2025-08-14T21:41:54.7960210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7960611Z return func(*args, **kwargs) 2025-08-14T21:41:54.7961014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.7961432Z self.value(current_states) 2025-08-14T21:41:54.7961566Z 2025-08-14T21:41:54.7961658Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.7961929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7962335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7962691Z return mod(**inputs) 2025-08-14T21:41:54.7963032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7963409Z outputs = self.bert( 2025-08-14T21:41:54.7963774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7964144Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7964510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7964881Z layer_outputs = layer_module( 2025-08-14T21:41:54.7965224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7965590Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7965996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7966407Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7966803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7967193Z return func(*args, **kwargs) 2025-08-14T21:41:54.7967569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.7967937Z self_outputs = self.self( 2025-08-14T21:41:54.7968281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7968646Z return func(*args, **kwargs) 2025-08-14T21:41:54.7969006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.7969436Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.7969617Z 2025-08-14T21:41:54.7969858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7970210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7970532Z return mod(**inputs) 2025-08-14T21:41:54.7970876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7971244Z outputs = self.bert( 2025-08-14T21:41:54.7971594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7971992Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7972353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7972721Z layer_outputs = layer_module( 2025-08-14T21:41:54.7973089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7973470Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7973864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.7974287Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.7974666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.7975031Z return func(*args, **kwargs) 2025-08-14T21:41:54.7975397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.7975834Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.7976270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.7976641Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7976824Z 2025-08-14T21:41:54.7976928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7977280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7977589Z return mod(**inputs) 2025-08-14T21:41:54.7977941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7978314Z outputs = self.bert( 2025-08-14T21:41:54.7978669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7979098Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7979460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7979830Z layer_outputs = layer_module( 2025-08-14T21:41:54.7980168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7980514Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7980893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7981285Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7981680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7982066Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7982457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7982904Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7983339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.7983765Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.7983913Z 2025-08-14T21:41:54.7984034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7984426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7984745Z return mod(**inputs) 2025-08-14T21:41:54.7985099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7985475Z outputs = self.bert( 2025-08-14T21:41:54.7985881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7986295Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7986763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7987174Z layer_outputs = layer_module( 2025-08-14T21:41:54.7987543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7987928Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7988354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7988764Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7989192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7989613Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7990047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.7990521Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.7990973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.7991465Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.7991867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.7992223Z return self.act(input) 2025-08-14T21:41:54.7992347Z 2025-08-14T21:41:54.7992460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.7992850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.7993192Z return mod(**inputs) 2025-08-14T21:41:54.7993575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.7993975Z outputs = self.bert( 2025-08-14T21:41:54.7994356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.7994770Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.7995174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.7995579Z layer_outputs = layer_module( 2025-08-14T21:41:54.7996011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.7996403Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.7996804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.7997231Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.7997646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.7998066Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.7998497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.7998992Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.7999471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.7999889Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8000031Z 2025-08-14T21:41:54.8000148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8000518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8000885Z return mod(**inputs) 2025-08-14T21:41:54.8001259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8001661Z outputs = self.bert( 2025-08-14T21:41:54.8002029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8002444Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8002841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8003249Z layer_outputs = layer_module( 2025-08-14T21:41:54.8003615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8003994Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8004395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8004801Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8005207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8005603Z return func(*args, **kwargs) 2025-08-14T21:41:54.8006020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8006422Z self_outputs = self.self( 2025-08-14T21:41:54.8006803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8007207Z return func(*args, **kwargs) 2025-08-14T21:41:54.8007592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.8008139Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.8008428Z 2025-08-14T21:41:54.8008539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8009240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8009629Z return mod(**inputs) 2025-08-14T21:41:54.8010009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8010412Z outputs = self.bert( 2025-08-14T21:41:54.8010781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8011196Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8011592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8011998Z layer_outputs = layer_module( 2025-08-14T21:41:54.8012361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8012744Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8013149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8013545Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8013925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8014308Z return func(*args, **kwargs) 2025-08-14T21:41:54.8014663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8015021Z self_outputs = self.self( 2025-08-14T21:41:54.8015374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8015808Z return func(*args, **kwargs) 2025-08-14T21:41:54.8016181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.8016557Z self.key(current_states) 2025-08-14T21:41:54.8016686Z 2025-08-14T21:41:54.8016796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8017170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8017509Z return mod(**inputs) 2025-08-14T21:41:54.8017883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8018292Z outputs = self.bert( 2025-08-14T21:41:54.8018644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8019015Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8019390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8019840Z layer_outputs = layer_module( 2025-08-14T21:41:54.8020169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8020524Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8021872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8022277Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8022644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8023011Z return func(*args, **kwargs) 2025-08-14T21:41:54.8023369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8023739Z self_outputs = self.self( 2025-08-14T21:41:54.8024104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8024511Z return func(*args, **kwargs) 2025-08-14T21:41:54.8024909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.8025333Z self.value(current_states) 2025-08-14T21:41:54.8025468Z 2025-08-14T21:41:54.8025559Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.8025821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8026209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8026534Z return mod(**inputs) 2025-08-14T21:41:54.8026899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8027281Z outputs = self.bert( 2025-08-14T21:41:54.8027634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8028025Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8028402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8028777Z layer_outputs = layer_module( 2025-08-14T21:41:54.8029119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8029483Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8029864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8030249Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8030636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8031060Z return func(*args, **kwargs) 2025-08-14T21:41:54.8031442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8031883Z self_outputs = self.self( 2025-08-14T21:41:54.8032261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8032659Z return func(*args, **kwargs) 2025-08-14T21:41:54.8033037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.8033500Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.8033772Z 2025-08-14T21:41:54.8033883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8034266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8034604Z return mod(**inputs) 2025-08-14T21:41:54.8034982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8035384Z outputs = self.bert( 2025-08-14T21:41:54.8035754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8036282Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8036744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8037160Z layer_outputs = layer_module( 2025-08-14T21:41:54.8037558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8037933Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8038314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8038700Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8039068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8039439Z return func(*args, **kwargs) 2025-08-14T21:41:54.8039802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.8040225Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.8040653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.8041065Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8041210Z 2025-08-14T21:41:54.8041328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8041697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8042036Z return mod(**inputs) 2025-08-14T21:41:54.8042413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8042807Z outputs = self.bert( 2025-08-14T21:41:54.8043175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8043577Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8043957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8044332Z layer_outputs = layer_module( 2025-08-14T21:41:54.8044683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8045049Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8045452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.8045888Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.8046311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.8046727Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.8047155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.8047651Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.8048109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.8048552Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8048705Z 2025-08-14T21:41:54.8048822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8049233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8049577Z return mod(**inputs) 2025-08-14T21:41:54.8049952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8050352Z outputs = self.bert( 2025-08-14T21:41:54.8050725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8051132Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8051553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8051957Z layer_outputs = layer_module( 2025-08-14T21:41:54.8052322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8052707Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8053118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.8053560Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.8053984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.8054393Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.8054827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.8055309Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.8055753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.8056188Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.8056593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.8056950Z return self.act(input) 2025-08-14T21:41:54.8057065Z 2025-08-14T21:41:54.8057176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8057534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8057860Z return mod(**inputs) 2025-08-14T21:41:54.8058216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8058587Z outputs = self.bert( 2025-08-14T21:41:54.8058941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8059334Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8059706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8060078Z layer_outputs = layer_module( 2025-08-14T21:41:54.8060419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8060810Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8061185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.8061577Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.8061985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.8062380Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.8062779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.8063265Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.8063696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.8064089Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8064227Z 2025-08-14T21:41:54.8064332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8064694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8065024Z return mod(**inputs) 2025-08-14T21:41:54.8065372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8065782Z outputs = self.bert( 2025-08-14T21:41:54.8066141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8066523Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8066895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8067274Z layer_outputs = layer_module( 2025-08-14T21:41:54.8067622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8067977Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8068381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8068792Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8069201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8069580Z return func(*args, **kwargs) 2025-08-14T21:41:54.8069949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8070329Z self_outputs = self.self( 2025-08-14T21:41:54.8070691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8071086Z return func(*args, **kwargs) 2025-08-14T21:41:54.8071474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.8072029Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.8072303Z 2025-08-14T21:41:54.8072411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8072802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8073142Z return mod(**inputs) 2025-08-14T21:41:54.8073518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8073914Z outputs = self.bert( 2025-08-14T21:41:54.8074288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8074691Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8075110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8075516Z layer_outputs = layer_module( 2025-08-14T21:41:54.8075960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8076371Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8076792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8077227Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8077667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8078083Z return func(*args, **kwargs) 2025-08-14T21:41:54.8078462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8078867Z self_outputs = self.self( 2025-08-14T21:41:54.8079245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8079627Z return func(*args, **kwargs) 2025-08-14T21:41:54.8080019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.8080419Z self.key(current_states) 2025-08-14T21:41:54.8080579Z 2025-08-14T21:41:54.8080700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8081100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8081454Z return mod(**inputs) 2025-08-14T21:41:54.8081874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8082308Z outputs = self.bert( 2025-08-14T21:41:54.8082718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8083157Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8083586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8084026Z layer_outputs = layer_module( 2025-08-14T21:41:54.8084437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8084853Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8085280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8085724Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8086157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8086576Z return func(*args, **kwargs) 2025-08-14T21:41:54.8086985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8087410Z self_outputs = self.self( 2025-08-14T21:41:54.8087834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8088236Z return func(*args, **kwargs) 2025-08-14T21:41:54.8088647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.8089060Z self.value(current_states) 2025-08-14T21:41:54.8089190Z 2025-08-14T21:41:54.8089289Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.8089556Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8089923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8090267Z return mod(**inputs) 2025-08-14T21:41:54.8090695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8091113Z outputs = self.bert( 2025-08-14T21:41:54.8091495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8091928Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8092303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8092685Z layer_outputs = layer_module( 2025-08-14T21:41:54.8093052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8093430Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8093835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8094256Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8094660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8095060Z return func(*args, **kwargs) 2025-08-14T21:41:54.8095455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8095869Z self_outputs = self.self( 2025-08-14T21:41:54.8096287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8096677Z return func(*args, **kwargs) 2025-08-14T21:41:54.8097044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.8097485Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.8097672Z 2025-08-14T21:41:54.8097784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8098165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8098511Z return mod(**inputs) 2025-08-14T21:41:54.8098887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8099279Z outputs = self.bert( 2025-08-14T21:41:54.8099658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8100069Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8100470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8100876Z layer_outputs = layer_module( 2025-08-14T21:41:54.8101239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8101623Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8102017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8102428Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8102831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8103221Z return func(*args, **kwargs) 2025-08-14T21:41:54.8103606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.8104064Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.8104520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.8104926Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8105079Z 2025-08-14T21:41:54.8105224Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8105602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8105941Z return mod(**inputs) 2025-08-14T21:41:54.8106306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8106705Z outputs = self.bert( 2025-08-14T21:41:54.8107079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8107483Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8107888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8108289Z layer_outputs = layer_module( 2025-08-14T21:41:54.8108802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8109285Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8109693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.8110119Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.8110553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.8111050Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.8111491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.8111989Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.8112426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.8112852Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8113006Z 2025-08-14T21:41:54.8113115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8113490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8113832Z return mod(**inputs) 2025-08-14T21:41:54.8114219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8114628Z outputs = self.bert( 2025-08-14T21:41:54.8115015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8115423Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8116056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8116503Z layer_outputs = layer_module( 2025-08-14T21:41:54.8116876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8117277Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8117682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.8118105Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.8118501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.8118898Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.8119305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.8119749Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.8120170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.8120590Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.8121011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.8121344Z return self.act(input) 2025-08-14T21:41:54.8121463Z 2025-08-14T21:41:54.8121570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8121929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8122254Z return mod(**inputs) 2025-08-14T21:41:54.8122602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8122995Z outputs = self.bert( 2025-08-14T21:41:54.8123352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8123732Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8124116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8124504Z layer_outputs = layer_module( 2025-08-14T21:41:54.8124859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8125223Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8125652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.8126043Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.8126428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.8126818Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.8127224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.8127685Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.8128109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.8128493Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8128629Z 2025-08-14T21:41:54.8128744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8129102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8129425Z return mod(**inputs) 2025-08-14T21:41:54.8129775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8130145Z outputs = self.bert( 2025-08-14T21:41:54.8130486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8130861Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8131241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8131600Z layer_outputs = layer_module( 2025-08-14T21:41:54.8131930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8132276Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8132643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8133009Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8133381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8133739Z return func(*args, **kwargs) 2025-08-14T21:41:54.8134090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8134488Z self_outputs = self.self( 2025-08-14T21:41:54.8134841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8135208Z return func(*args, **kwargs) 2025-08-14T21:41:54.8135564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:54.8136087Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:54.8136354Z 2025-08-14T21:41:54.8136456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8136840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8137159Z return mod(**inputs) 2025-08-14T21:41:54.8137510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8137881Z outputs = self.bert( 2025-08-14T21:41:54.8138226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8138592Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8138957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8139331Z layer_outputs = layer_module( 2025-08-14T21:41:54.8139711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8140080Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8140454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8140836Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8141204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8141570Z return func(*args, **kwargs) 2025-08-14T21:41:54.8141931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8142299Z self_outputs = self.self( 2025-08-14T21:41:54.8142643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8143009Z return func(*args, **kwargs) 2025-08-14T21:41:54.8143367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:54.8143728Z self.key(current_states) 2025-08-14T21:41:54.8143849Z 2025-08-14T21:41:54.8143953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8144304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8144619Z return mod(**inputs) 2025-08-14T21:41:54.8144963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8145329Z outputs = self.bert( 2025-08-14T21:41:54.8145679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8146046Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8146420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8146790Z layer_outputs = layer_module( 2025-08-14T21:41:54.8147127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8147473Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8147847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8148246Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8148612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8148983Z return func(*args, **kwargs) 2025-08-14T21:41:54.8149348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8149726Z self_outputs = self.self( 2025-08-14T21:41:54.8149980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8150072Z return func(*args, **kwargs) 2025-08-14T21:41:54.8150346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:54.8150418Z self.value(current_states) 2025-08-14T21:41:54.8150421Z 2025-08-14T21:41:54.8150510Z cudagraph partition due to non gpu ops 2025-08-14T21:41:54.8150618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8150815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8150889Z return mod(**inputs) 2025-08-14T21:41:54.8151144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8151212Z outputs = self.bert( 2025-08-14T21:41:54.8151512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8151591Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8151856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8151929Z layer_outputs = layer_module( 2025-08-14T21:41:54.8152157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8152250Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8152501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8152587Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8152849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8152927Z return func(*args, **kwargs) 2025-08-14T21:41:54.8153197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:54.8153274Z self_outputs = self.self( 2025-08-14T21:41:54.8153527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8153609Z return func(*args, **kwargs) 2025-08-14T21:41:54.8153873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:54.8154022Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:54.8154026Z 2025-08-14T21:41:54.8154136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8154347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8154424Z return mod(**inputs) 2025-08-14T21:41:54.8154693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8154762Z outputs = self.bert( 2025-08-14T21:41:54.8155037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8155115Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8155395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8155496Z layer_outputs = layer_module( 2025-08-14T21:41:54.8155727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8155897Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8156233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:54.8156325Z self_attention_outputs = self.attention( 2025-08-14T21:41:54.8156588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:54.8156690Z return func(*args, **kwargs) 2025-08-14T21:41:54.8156965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:54.8157104Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:54.8157367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:54.8157471Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8157475Z 2025-08-14T21:41:54.8157587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8157817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8157888Z return mod(**inputs) 2025-08-14T21:41:54.8158172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8158253Z outputs = self.bert( 2025-08-14T21:41:54.8158505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8158581Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8158836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8158912Z layer_outputs = layer_module( 2025-08-14T21:41:54.8159144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8159223Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8159471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.8159569Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.8159835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.8159918Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.8160206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.8160329Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.8160587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:54.8160672Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8160675Z 2025-08-14T21:41:54.8160779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8160989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8161058Z return mod(**inputs) 2025-08-14T21:41:54.8161318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8161386Z outputs = self.bert( 2025-08-14T21:41:54.8161749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8161870Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8162118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8162215Z layer_outputs = layer_module( 2025-08-14T21:41:54.8162444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8162523Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8162782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.8162867Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.8163132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.8163236Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.8163517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:54.8163647Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:54.8163910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:54.8164025Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:54.8164249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:54.8164323Z return self.act(input) 2025-08-14T21:41:54.8164360Z 2025-08-14T21:41:54.8164469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8164676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8164744Z return mod(**inputs) 2025-08-14T21:41:54.8165016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:41:54.8165082Z outputs = self.bert( 2025-08-14T21:41:54.8165334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:54.8165418Z encoder_outputs = self.encoder( 2025-08-14T21:41:54.8165684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:54.8165767Z layer_outputs = layer_module( 2025-08-14T21:41:54.8165996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:54.8166075Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:54.8166327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:54.8166413Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:54.8166669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:54.8166756Z return forward_fn(*input_tensors) 2025-08-14T21:41:54.8167029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:54.8167168Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:54.8167415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:54.8167500Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8167503Z 2025-08-14T21:41:54.8167614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8167811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8167876Z return mod(**inputs) 2025-08-14T21:41:54.8168133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward 2025-08-14T21:41:54.8168249Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:41:54.8168502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward 2025-08-14T21:41:54.8168616Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:41:54.8168860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 769, in forward 2025-08-14T21:41:54.8168965Z hidden_states = self.transform(hidden_states) 2025-08-14T21:41:54.8169213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 745, in forward 2025-08-14T21:41:54.8169323Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:54.8169327Z 2025-08-14T21:41:54.8169429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8169622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8169695Z return mod(**inputs) 2025-08-14T21:41:54.8169942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward 2025-08-14T21:41:54.8170032Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:41:54.8170280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward 2025-08-14T21:41:54.8170387Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:41:54.8170676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 770, in forward 2025-08-14T21:41:54.8170769Z hidden_states = self.decoder(hidden_states) 2025-08-14T21:41:54.8170772Z 2025-08-14T21:41:54.8170873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:54.8171074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:54.8171138Z return mod(**inputs) 2025-08-14T21:41:54.8171391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1328, in forward 2025-08-14T21:41:54.8171581Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:41:54.8171584Z 2025-08-14T21:42:03.4410073Z Compilation time (from dynamo_timed): 14.994064649 2025-08-14T21:42:03.4491996Z pass 2025-08-14T21:42:03.4497527Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:03.4503479Z TIMING: _recursive_pre_grad_passes:0.00686 _recursive_joint_graph_passes:0.66504 _recursive_post_grad_passes:0.08164 async_compile.wait:0.75046 code_gen:7.47877 inductor_compile:8.65862 backend_compile:11.84603 gc:0.00043 entire_frame_compile:14.99406 total_wall_time:14.99406 2025-08-14T21:42:03.4504526Z STATS: call_* op count: 289 | FakeTensorMode.__torch_dispatch__:12337 | FakeTensor.__torch_dispatch__:4686 | ProxyTorchDispatchMode.__torch_dispatch__:4495 2025-08-14T21:42:03.4505078Z Dynamo produced 1 graphs covering 289 ops with 0 graph breaks (0 unique) 2025-08-14T21:42:08.7148073Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:42:08.7149029Z from pkg_resources import resource_filename 2025-08-14T21:42:09.3632828Z 2025-08-14T21:42:10.5486373Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:42:10.5489001Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:42:10.5499478Z cpu eval BertForQuestionAnswering 2025-08-14T21:42:10.9671822Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:11.1626546Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:11.3512974Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:18.9814012Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9816516Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9816903Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9820836Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9821188Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9821575Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9821886Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9827431Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9828063Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9828305Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9828527Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9828747Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9829009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9829427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9829802Z return mod(**inputs) 2025-08-14T21:42:18.9830227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9830652Z outputs = self.bert( 2025-08-14T21:42:18.9831137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9831567Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9831987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9832406Z layer_outputs = layer_module( 2025-08-14T21:42:18.9832802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9833214Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9833659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9834096Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9834538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9834966Z return func(*args, **kwargs) 2025-08-14T21:42:18.9835380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9835983Z self_outputs = self.self( 2025-08-14T21:42:18.9836437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9836869Z return func(*args, **kwargs) 2025-08-14T21:42:18.9837290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:18.9837878Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:18.9838187Z 2025-08-14T21:42:18.9838305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9838703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9839046Z return mod(**inputs) 2025-08-14T21:42:18.9839444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9839850Z outputs = self.bert( 2025-08-14T21:42:18.9840229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9840659Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9841064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9841540Z layer_outputs = layer_module( 2025-08-14T21:42:18.9841909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9842301Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9842717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9843143Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9843547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9843976Z return func(*args, **kwargs) 2025-08-14T21:42:18.9844379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9844785Z self_outputs = self.self( 2025-08-14T21:42:18.9845181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9845586Z return func(*args, **kwargs) 2025-08-14T21:42:18.9845971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:18.9846347Z self.key(current_states) 2025-08-14T21:42:18.9846471Z 2025-08-14T21:42:18.9846579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9847012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9847337Z return mod(**inputs) 2025-08-14T21:42:18.9847689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9848065Z outputs = self.bert( 2025-08-14T21:42:18.9848416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9848795Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9849451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9849853Z layer_outputs = layer_module( 2025-08-14T21:42:18.9850217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9850569Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9850949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9851335Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9851709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9852109Z return func(*args, **kwargs) 2025-08-14T21:42:18.9852502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9852899Z self_outputs = self.self( 2025-08-14T21:42:18.9853276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9853690Z return func(*args, **kwargs) 2025-08-14T21:42:18.9854096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:18.9854504Z self.value(current_states) 2025-08-14T21:42:18.9854639Z 2025-08-14T21:42:18.9854727Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9854986Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9855369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9855686Z return mod(**inputs) 2025-08-14T21:42:18.9856046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9856452Z outputs = self.bert( 2025-08-14T21:42:18.9856800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9857187Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9857588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9857964Z layer_outputs = layer_module( 2025-08-14T21:42:18.9858306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9858683Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9859069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9859459Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9859830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9860212Z return func(*args, **kwargs) 2025-08-14T21:42:18.9860602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9860996Z self_outputs = self.self( 2025-08-14T21:42:18.9861376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9861806Z return func(*args, **kwargs) 2025-08-14T21:42:18.9862199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:18.9862665Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:18.9862860Z 2025-08-14T21:42:18.9862966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9863338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9863679Z return mod(**inputs) 2025-08-14T21:42:18.9864062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9864458Z outputs = self.bert( 2025-08-14T21:42:18.9864830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9865235Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9865614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9865993Z layer_outputs = layer_module( 2025-08-14T21:42:18.9882445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9883044Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9883497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9883945Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9884368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9884778Z return func(*args, **kwargs) 2025-08-14T21:42:18.9885173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:18.9885646Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:18.9886108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:18.9886534Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:18.9886689Z 2025-08-14T21:42:18.9886807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9887202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9887652Z return mod(**inputs) 2025-08-14T21:42:18.9888039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9888449Z outputs = self.bert( 2025-08-14T21:42:18.9888829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9889251Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9889651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9890093Z layer_outputs = layer_module( 2025-08-14T21:42:18.9890466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9890858Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9891256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:18.9891674Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:18.9892099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:18.9892513Z return forward_fn(*input_tensors) 2025-08-14T21:42:18.9892996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:18.9893482Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:18.9893931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:18.9894341Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:18.9894498Z 2025-08-14T21:42:18.9894614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9894999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9895343Z return mod(**inputs) 2025-08-14T21:42:18.9895724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9896122Z outputs = self.bert( 2025-08-14T21:42:18.9896503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9896909Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9897310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9897712Z layer_outputs = layer_module( 2025-08-14T21:42:18.9898079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9898458Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9898840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:18.9899231Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:18.9899627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:18.9900027Z return forward_fn(*input_tensors) 2025-08-14T21:42:18.9900433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:18.9900888Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:18.9901307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:18.9901725Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:18.9902103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:18.9902456Z return self.act(input) 2025-08-14T21:42:18.9902580Z 2025-08-14T21:42:18.9902687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9903050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9903375Z return mod(**inputs) 2025-08-14T21:42:18.9903725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9904103Z outputs = self.bert( 2025-08-14T21:42:18.9904460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9904895Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9905268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9905652Z layer_outputs = layer_module( 2025-08-14T21:42:18.9906018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9906391Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9906797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:18.9907208Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:18.9907648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:18.9908066Z return forward_fn(*input_tensors) 2025-08-14T21:42:18.9908502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:18.9909326Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:18.9909787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:18.9910214Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:18.9910371Z 2025-08-14T21:42:18.9910487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9910875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9911216Z return mod(**inputs) 2025-08-14T21:42:18.9911604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9912007Z outputs = self.bert( 2025-08-14T21:42:18.9912391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9912849Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9913258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9913671Z layer_outputs = layer_module( 2025-08-14T21:42:18.9914040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9914442Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9914858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9915290Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9915707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9916210Z return func(*args, **kwargs) 2025-08-14T21:42:18.9916616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9917021Z self_outputs = self.self( 2025-08-14T21:42:18.9917421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9918046Z return func(*args, **kwargs) 2025-08-14T21:42:18.9918447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:18.9919020Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:18.9919330Z 2025-08-14T21:42:18.9919451Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9919849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9920249Z return mod(**inputs) 2025-08-14T21:42:18.9920631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9921041Z outputs = self.bert( 2025-08-14T21:42:18.9921426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9921834Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9922240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9922653Z layer_outputs = layer_module( 2025-08-14T21:42:18.9923026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9923465Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9923882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9924305Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9924712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9925127Z return func(*args, **kwargs) 2025-08-14T21:42:18.9925512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9925910Z self_outputs = self.self( 2025-08-14T21:42:18.9926282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9926676Z return func(*args, **kwargs) 2025-08-14T21:42:18.9927068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:18.9927469Z self.key(current_states) 2025-08-14T21:42:18.9927592Z 2025-08-14T21:42:18.9927703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9928086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9928428Z return mod(**inputs) 2025-08-14T21:42:18.9928795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9929193Z outputs = self.bert( 2025-08-14T21:42:18.9929564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9929981Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9930366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9930808Z layer_outputs = layer_module( 2025-08-14T21:42:18.9931180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9931564Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9931964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9932378Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9932777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9933226Z return func(*args, **kwargs) 2025-08-14T21:42:18.9933603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9934008Z self_outputs = self.self( 2025-08-14T21:42:18.9934387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9934779Z return func(*args, **kwargs) 2025-08-14T21:42:18.9935158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:18.9935574Z self.value(current_states) 2025-08-14T21:42:18.9935700Z 2025-08-14T21:42:18.9935799Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9936049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9936431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9936774Z return mod(**inputs) 2025-08-14T21:42:18.9937148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9937541Z outputs = self.bert( 2025-08-14T21:42:18.9937896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9938313Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9938679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9939054Z layer_outputs = layer_module( 2025-08-14T21:42:18.9939398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9939761Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9940135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9940526Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9940909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9941276Z return func(*args, **kwargs) 2025-08-14T21:42:18.9941638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9942019Z self_outputs = self.self( 2025-08-14T21:42:18.9942376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9942735Z return func(*args, **kwargs) 2025-08-14T21:42:18.9943093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:18.9943530Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:18.9943717Z 2025-08-14T21:42:18.9943826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9944174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9944496Z return mod(**inputs) 2025-08-14T21:42:18.9944850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9945220Z outputs = self.bert( 2025-08-14T21:42:18.9945573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9945957Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9946328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9946702Z layer_outputs = layer_module( 2025-08-14T21:42:18.9947065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9947462Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9947851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9948258Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9948663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9949034Z return func(*args, **kwargs) 2025-08-14T21:42:18.9949391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:18.9949838Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:18.9950264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:18.9950657Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:18.9950797Z 2025-08-14T21:42:18.9950901Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9951260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9951582Z return mod(**inputs) 2025-08-14T21:42:18.9951925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9952343Z outputs = self.bert( 2025-08-14T21:42:18.9952722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9953124Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9953506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9953906Z layer_outputs = layer_module( 2025-08-14T21:42:18.9954271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9954645Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9955047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:18.9955470Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:18.9955999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:18.9956427Z return forward_fn(*input_tensors) 2025-08-14T21:42:18.9956876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:18.9957375Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:18.9957821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:18.9958226Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:18.9958379Z 2025-08-14T21:42:18.9958490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9958870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9959208Z return mod(**inputs) 2025-08-14T21:42:18.9959591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9959987Z outputs = self.bert( 2025-08-14T21:42:18.9960362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9960752Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9961140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9961541Z layer_outputs = layer_module( 2025-08-14T21:42:18.9961956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9962347Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9962747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:18.9963157Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:18.9963581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:18.9963997Z return forward_fn(*input_tensors) 2025-08-14T21:42:18.9964421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:18.9964866Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:18.9965277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:18.9965687Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:18.9966059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:18.9966386Z return self.act(input) 2025-08-14T21:42:18.9966502Z 2025-08-14T21:42:18.9966604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9966984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9967301Z return mod(**inputs) 2025-08-14T21:42:18.9967639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9968010Z outputs = self.bert( 2025-08-14T21:42:18.9968358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9968788Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9969157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9969527Z layer_outputs = layer_module( 2025-08-14T21:42:18.9969863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9970205Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9970577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:18.9970959Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:18.9971352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:18.9971727Z return forward_fn(*input_tensors) 2025-08-14T21:42:18.9972124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:18.9972583Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:18.9973006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:18.9973402Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:18.9973551Z 2025-08-14T21:42:18.9973663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9974044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9974376Z return mod(**inputs) 2025-08-14T21:42:18.9974734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9975115Z outputs = self.bert( 2025-08-14T21:42:18.9975481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9975914Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9976279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9976650Z layer_outputs = layer_module( 2025-08-14T21:42:18.9976982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9977335Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9977707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9978110Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9978479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9978854Z return func(*args, **kwargs) 2025-08-14T21:42:18.9979228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9979588Z self_outputs = self.self( 2025-08-14T21:42:18.9979938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9980300Z return func(*args, **kwargs) 2025-08-14T21:42:18.9980706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:18.9981241Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:18.9981505Z 2025-08-14T21:42:18.9981611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9981964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9982279Z return mod(**inputs) 2025-08-14T21:42:18.9982620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9982989Z outputs = self.bert( 2025-08-14T21:42:18.9983338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9983709Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9984084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9984469Z layer_outputs = layer_module( 2025-08-14T21:42:18.9984820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9985181Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9985560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9985947Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9986324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9986713Z return func(*args, **kwargs) 2025-08-14T21:42:18.9987100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9987496Z self_outputs = self.self( 2025-08-14T21:42:18.9987873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9988259Z return func(*args, **kwargs) 2025-08-14T21:42:18.9988640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:18.9989013Z self.key(current_states) 2025-08-14T21:42:18.9989135Z 2025-08-14T21:42:18.9989241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9989607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9989968Z return mod(**inputs) 2025-08-14T21:42:18.9990338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9990733Z outputs = self.bert( 2025-08-14T21:42:18.9991103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9991510Z encoder_outputs = self.encoder( 2025-08-14T21:42:18.9991903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:18.9992319Z layer_outputs = layer_module( 2025-08-14T21:42:18.9992677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:18.9993059Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:18.9993464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:18.9993873Z self_attention_outputs = self.attention( 2025-08-14T21:42:18.9994273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9994666Z return func(*args, **kwargs) 2025-08-14T21:42:18.9995092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:18.9995495Z self_outputs = self.self( 2025-08-14T21:42:18.9995973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:18.9996384Z return func(*args, **kwargs) 2025-08-14T21:42:18.9996774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:18.9997204Z self.value(current_states) 2025-08-14T21:42:18.9997345Z 2025-08-14T21:42:18.9997434Z cudagraph partition due to non gpu ops 2025-08-14T21:42:18.9997693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:18.9998074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:18.9998424Z return mod(**inputs) 2025-08-14T21:42:18.9998807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:18.9999203Z outputs = self.bert( 2025-08-14T21:42:18.9999579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:18.9999986Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0000383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0000792Z layer_outputs = layer_module( 2025-08-14T21:42:19.0001165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0001559Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0001964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0002369Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0002778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0003173Z return func(*args, **kwargs) 2025-08-14T21:42:19.0003559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0003964Z self_outputs = self.self( 2025-08-14T21:42:19.0004341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0004731Z return func(*args, **kwargs) 2025-08-14T21:42:19.0005139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0005598Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0005792Z 2025-08-14T21:42:19.0005911Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0006286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0006628Z return mod(**inputs) 2025-08-14T21:42:19.0007004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0007421Z outputs = self.bert( 2025-08-14T21:42:19.0007785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0008190Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0008588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0009125Z layer_outputs = layer_module( 2025-08-14T21:42:19.0009497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0009878Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0010389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0010792Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0011201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0011598Z return func(*args, **kwargs) 2025-08-14T21:42:19.0011984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0012436Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0012897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0013313Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0013460Z 2025-08-14T21:42:19.0013574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0013956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0014299Z return mod(**inputs) 2025-08-14T21:42:19.0014674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0015066Z outputs = self.bert( 2025-08-14T21:42:19.0015440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0015851Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0016242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0016650Z layer_outputs = layer_module( 2025-08-14T21:42:19.0017015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0017410Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0017819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0018249Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0018685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0019105Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0019508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0020003Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0020428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0020831Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0020970Z 2025-08-14T21:42:19.0021075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0021442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0021767Z return mod(**inputs) 2025-08-14T21:42:19.0022114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0022523Z outputs = self.bert( 2025-08-14T21:42:19.0022876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0023258Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0023622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0024000Z layer_outputs = layer_module( 2025-08-14T21:42:19.0024349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0024711Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0025126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0025511Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0025919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0026329Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0026754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0027236Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0027655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0028070Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0028454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0028795Z return self.act(input) 2025-08-14T21:42:19.0028907Z 2025-08-14T21:42:19.0029024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0029400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0029742Z return mod(**inputs) 2025-08-14T21:42:19.0030114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0030507Z outputs = self.bert( 2025-08-14T21:42:19.0030879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0031288Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0031680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0032074Z layer_outputs = layer_module( 2025-08-14T21:42:19.0032439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0032824Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0033217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0033625Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0034046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0034491Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0034938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0035441Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0035994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0036426Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0036612Z 2025-08-14T21:42:19.0036728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0037126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0037488Z return mod(**inputs) 2025-08-14T21:42:19.0037860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0038262Z outputs = self.bert( 2025-08-14T21:42:19.0038640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0039054Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0039447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0039891Z layer_outputs = layer_module( 2025-08-14T21:42:19.0040260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0040636Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0041041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0041457Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0041863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0042248Z return func(*args, **kwargs) 2025-08-14T21:42:19.0042635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0043040Z self_outputs = self.self( 2025-08-14T21:42:19.0043420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0043813Z return func(*args, **kwargs) 2025-08-14T21:42:19.0044200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:19.0044745Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:19.0045022Z 2025-08-14T21:42:19.0045129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0045514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0045852Z return mod(**inputs) 2025-08-14T21:42:19.0046226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0046613Z outputs = self.bert( 2025-08-14T21:42:19.0046972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0047356Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0047723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0048109Z layer_outputs = layer_module( 2025-08-14T21:42:19.0048455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0048819Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0049229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0049615Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0049996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0050359Z return func(*args, **kwargs) 2025-08-14T21:42:19.0050729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0051111Z self_outputs = self.self( 2025-08-14T21:42:19.0051498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0051859Z return func(*args, **kwargs) 2025-08-14T21:42:19.0052226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:19.0052601Z self.key(current_states) 2025-08-14T21:42:19.0052718Z 2025-08-14T21:42:19.0052826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0053177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0053499Z return mod(**inputs) 2025-08-14T21:42:19.0053858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0054284Z outputs = self.bert( 2025-08-14T21:42:19.0054655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0055037Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0055429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0055825Z layer_outputs = layer_module( 2025-08-14T21:42:19.0056171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0056536Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0056907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0057321Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0057724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0058117Z return func(*args, **kwargs) 2025-08-14T21:42:19.0058495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0058891Z self_outputs = self.self( 2025-08-14T21:42:19.0059249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0059606Z return func(*args, **kwargs) 2025-08-14T21:42:19.0059974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:19.0060353Z self.value(current_states) 2025-08-14T21:42:19.0060471Z 2025-08-14T21:42:19.0060561Z cudagraph partition due to non gpu ops 2025-08-14T21:42:19.0060794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0061153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0061491Z return mod(**inputs) 2025-08-14T21:42:19.0061858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0062253Z outputs = self.bert( 2025-08-14T21:42:19.0062619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0063026Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0063441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0063834Z layer_outputs = layer_module( 2025-08-14T21:42:19.0064189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0064568Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0064963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0065365Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0065786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0066181Z return func(*args, **kwargs) 2025-08-14T21:42:19.0066565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0066971Z self_outputs = self.self( 2025-08-14T21:42:19.0067353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0067734Z return func(*args, **kwargs) 2025-08-14T21:42:19.0068122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0068636Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0068834Z 2025-08-14T21:42:19.0068945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0069330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0069679Z return mod(**inputs) 2025-08-14T21:42:19.0070052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0070442Z outputs = self.bert( 2025-08-14T21:42:19.0070819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0071221Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0071607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0072002Z layer_outputs = layer_module( 2025-08-14T21:42:19.0072373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0072770Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0073180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0073600Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0074011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0074417Z return func(*args, **kwargs) 2025-08-14T21:42:19.0074804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0075274Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0075741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0076266Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0076427Z 2025-08-14T21:42:19.0076543Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0076942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0077304Z return mod(**inputs) 2025-08-14T21:42:19.0077674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0078073Z outputs = self.bert( 2025-08-14T21:42:19.0078487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0078865Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0079239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0079613Z layer_outputs = layer_module( 2025-08-14T21:42:19.0079958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0080307Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0080710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0081133Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0081547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0081959Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0082388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0082866Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0083309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0083785Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0083940Z 2025-08-14T21:42:19.0084051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0084433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0084767Z return mod(**inputs) 2025-08-14T21:42:19.0085142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0085542Z outputs = self.bert( 2025-08-14T21:42:19.0085912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0086327Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0086721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0087177Z layer_outputs = layer_module( 2025-08-14T21:42:19.0087536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0087921Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0088321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0088732Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0089144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0089565Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0089991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0090459Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0090915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0091355Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0091753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0092102Z return self.act(input) 2025-08-14T21:42:19.0092230Z 2025-08-14T21:42:19.0092341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0092721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0093090Z return mod(**inputs) 2025-08-14T21:42:19.0093468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0093868Z outputs = self.bert( 2025-08-14T21:42:19.0094245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0094647Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0095054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0095472Z layer_outputs = layer_module( 2025-08-14T21:42:19.0095842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0096217Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0096617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0097033Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0097458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0097889Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0098371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0098876Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0099338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0099784Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0099936Z 2025-08-14T21:42:19.0100057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0100445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0100788Z return mod(**inputs) 2025-08-14T21:42:19.0101175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0101591Z outputs = self.bert( 2025-08-14T21:42:19.0101974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0102384Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0102790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0103202Z layer_outputs = layer_module( 2025-08-14T21:42:19.0103568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0103961Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0104380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0104804Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0105226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0105629Z return func(*args, **kwargs) 2025-08-14T21:42:19.0106029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0106434Z self_outputs = self.self( 2025-08-14T21:42:19.0106830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0107233Z return func(*args, **kwargs) 2025-08-14T21:42:19.0107617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:19.0108199Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:19.0108492Z 2025-08-14T21:42:19.0108605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0109156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0109505Z return mod(**inputs) 2025-08-14T21:42:19.0109903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0110311Z outputs = self.bert( 2025-08-14T21:42:19.0110709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0111205Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0111640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0112070Z layer_outputs = layer_module( 2025-08-14T21:42:19.0112451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0112852Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0113283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0113718Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0114187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0114592Z return func(*args, **kwargs) 2025-08-14T21:42:19.0114992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0115402Z self_outputs = self.self( 2025-08-14T21:42:19.0115910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0116369Z return func(*args, **kwargs) 2025-08-14T21:42:19.0116765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:19.0117178Z self.key(current_states) 2025-08-14T21:42:19.0117313Z 2025-08-14T21:42:19.0117427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0117823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0118181Z return mod(**inputs) 2025-08-14T21:42:19.0118571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0118982Z outputs = self.bert( 2025-08-14T21:42:19.0119371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0119783Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0120192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0120599Z layer_outputs = layer_module( 2025-08-14T21:42:19.0120974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0121363Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0121780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0122202Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0122605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0122985Z return func(*args, **kwargs) 2025-08-14T21:42:19.0123358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0123791Z self_outputs = self.self( 2025-08-14T21:42:19.0124151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0124529Z return func(*args, **kwargs) 2025-08-14T21:42:19.0124902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:19.0125299Z self.value(current_states) 2025-08-14T21:42:19.0125429Z 2025-08-14T21:42:19.0125522Z cudagraph partition due to non gpu ops 2025-08-14T21:42:19.0125774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0126182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0126504Z return mod(**inputs) 2025-08-14T21:42:19.0126867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0127252Z outputs = self.bert( 2025-08-14T21:42:19.0127608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0128024Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0128407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0128814Z layer_outputs = layer_module( 2025-08-14T21:42:19.0129198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0129574Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0129966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0130371Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0130753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0131149Z return func(*args, **kwargs) 2025-08-14T21:42:19.0131533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0131925Z self_outputs = self.self( 2025-08-14T21:42:19.0132293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0132674Z return func(*args, **kwargs) 2025-08-14T21:42:19.0133046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0133493Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0133694Z 2025-08-14T21:42:19.0133803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0134173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0134501Z return mod(**inputs) 2025-08-14T21:42:19.0134870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0135247Z outputs = self.bert( 2025-08-14T21:42:19.0135599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0135974Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0136351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0136726Z layer_outputs = layer_module( 2025-08-14T21:42:19.0137066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0137419Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0137798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0138203Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0138580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0138971Z return func(*args, **kwargs) 2025-08-14T21:42:19.0139351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0139783Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0140203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0140612Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0140749Z 2025-08-14T21:42:19.0140861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0141209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0141538Z return mod(**inputs) 2025-08-14T21:42:19.0141893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0142266Z outputs = self.bert( 2025-08-14T21:42:19.0142610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0142993Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0143404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0143777Z layer_outputs = layer_module( 2025-08-14T21:42:19.0144122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0144480Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0144862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0145250Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0145653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0146054Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0146460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0146909Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0147332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0147725Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0147863Z 2025-08-14T21:42:19.0147966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0148325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0148656Z return mod(**inputs) 2025-08-14T21:42:19.0149010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0149380Z outputs = self.bert( 2025-08-14T21:42:19.0149743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0150122Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0150518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0150928Z layer_outputs = layer_module( 2025-08-14T21:42:19.0151300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0151688Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0152083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0152519Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0152940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0153353Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0153786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0154282Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0154797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0155237Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0155634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0156095Z return self.act(input) 2025-08-14T21:42:19.0156220Z 2025-08-14T21:42:19.0156341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0156722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0157082Z return mod(**inputs) 2025-08-14T21:42:19.0157484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0157926Z outputs = self.bert( 2025-08-14T21:42:19.0158264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0158639Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0159007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0159369Z layer_outputs = layer_module( 2025-08-14T21:42:19.0159711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0160065Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0160439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0160818Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0161213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0161603Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0162005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0162483Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0162941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0163352Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0163495Z 2025-08-14T21:42:19.0163606Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0163990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0164336Z return mod(**inputs) 2025-08-14T21:42:19.0164689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0165055Z outputs = self.bert( 2025-08-14T21:42:19.0165409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0165787Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0166154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0166522Z layer_outputs = layer_module( 2025-08-14T21:42:19.0166881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0167235Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0167604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0167992Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0168373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0168738Z return func(*args, **kwargs) 2025-08-14T21:42:19.0169127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0169500Z self_outputs = self.self( 2025-08-14T21:42:19.0169856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0170214Z return func(*args, **kwargs) 2025-08-14T21:42:19.0170574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:19.0171084Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:19.0171342Z 2025-08-14T21:42:19.0171452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0171833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0172158Z return mod(**inputs) 2025-08-14T21:42:19.0172513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0172878Z outputs = self.bert( 2025-08-14T21:42:19.0173234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0173629Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0174024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0174413Z layer_outputs = layer_module( 2025-08-14T21:42:19.0174780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0175161Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0175552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0175938Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0176315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0176684Z return func(*args, **kwargs) 2025-08-14T21:42:19.0177043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0177419Z self_outputs = self.self( 2025-08-14T21:42:19.0177775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0178131Z return func(*args, **kwargs) 2025-08-14T21:42:19.0178494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:19.0178873Z self.key(current_states) 2025-08-14T21:42:19.0178988Z 2025-08-14T21:42:19.0179098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0179452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0179795Z return mod(**inputs) 2025-08-14T21:42:19.0180170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0180598Z outputs = self.bert( 2025-08-14T21:42:19.0180964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0181367Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0181752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0182125Z layer_outputs = layer_module( 2025-08-14T21:42:19.0182470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0182828Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0183234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0183639Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0184035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0184425Z return func(*args, **kwargs) 2025-08-14T21:42:19.0184802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0185207Z self_outputs = self.self( 2025-08-14T21:42:19.0185586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0186017Z return func(*args, **kwargs) 2025-08-14T21:42:19.0186399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:19.0186801Z self.value(current_states) 2025-08-14T21:42:19.0186923Z 2025-08-14T21:42:19.0187018Z cudagraph partition due to non gpu ops 2025-08-14T21:42:19.0187265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0187641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0187982Z return mod(**inputs) 2025-08-14T21:42:19.0188353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0188740Z outputs = self.bert( 2025-08-14T21:42:19.0189112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0189513Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0189899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0190289Z layer_outputs = layer_module( 2025-08-14T21:42:19.0190651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0191039Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0191432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0191839Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0192245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0192642Z return func(*args, **kwargs) 2025-08-14T21:42:19.0193030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0193429Z self_outputs = self.self( 2025-08-14T21:42:19.0193822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0194215Z return func(*args, **kwargs) 2025-08-14T21:42:19.0194610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0195081Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0195295Z 2025-08-14T21:42:19.0195413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0195867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0196237Z return mod(**inputs) 2025-08-14T21:42:19.0196625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0197029Z outputs = self.bert( 2025-08-14T21:42:19.0197470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0197904Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0198304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0198711Z layer_outputs = layer_module( 2025-08-14T21:42:19.0199083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0199517Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0199921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0200345Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0200764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0201199Z return func(*args, **kwargs) 2025-08-14T21:42:19.0201579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0202048Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0202521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0202944Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0203095Z 2025-08-14T21:42:19.0203209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0203600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0203952Z return mod(**inputs) 2025-08-14T21:42:19.0204341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0204734Z outputs = self.bert( 2025-08-14T21:42:19.0205116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0205531Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0205927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0206335Z layer_outputs = layer_module( 2025-08-14T21:42:19.0206711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0207094Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0207504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0207928Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0208364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0208936Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0209383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0209882Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0210340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0210811Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0210968Z 2025-08-14T21:42:19.0211085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0211478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0211834Z return mod(**inputs) 2025-08-14T21:42:19.0212228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0212645Z outputs = self.bert( 2025-08-14T21:42:19.0213041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0213482Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0213894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0214310Z layer_outputs = layer_module( 2025-08-14T21:42:19.0214674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0215062Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0215483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0215873Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0216317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0216717Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0217124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0217581Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0218002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0218450Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0218855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0219186Z return self.act(input) 2025-08-14T21:42:19.0219308Z 2025-08-14T21:42:19.0219414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0219774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0220099Z return mod(**inputs) 2025-08-14T21:42:19.0220452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0220833Z outputs = self.bert( 2025-08-14T21:42:19.0221186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0221566Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0221935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0222318Z layer_outputs = layer_module( 2025-08-14T21:42:19.0222683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0223058Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0223468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0223880Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0224304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0224715Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0225145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0225652Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0226099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0226498Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0226641Z 2025-08-14T21:42:19.0226744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0227103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0227417Z return mod(**inputs) 2025-08-14T21:42:19.0227791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0228163Z outputs = self.bert( 2025-08-14T21:42:19.0228516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0228886Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0229261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0229639Z layer_outputs = layer_module( 2025-08-14T21:42:19.0229985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0230364Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0230819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0231245Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0231653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0232065Z return func(*args, **kwargs) 2025-08-14T21:42:19.0232450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0232849Z self_outputs = self.self( 2025-08-14T21:42:19.0233242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0233648Z return func(*args, **kwargs) 2025-08-14T21:42:19.0234047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:19.0234598Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:19.0234890Z 2025-08-14T21:42:19.0235005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0235398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0235747Z return mod(**inputs) 2025-08-14T21:42:19.0236207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0236627Z outputs = self.bert( 2025-08-14T21:42:19.0237024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0237435Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0237848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0238269Z layer_outputs = layer_module( 2025-08-14T21:42:19.0238649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0239039Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0239462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0239892Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0240305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0240418Z return func(*args, **kwargs) 2025-08-14T21:42:19.0240687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0240772Z self_outputs = self.self( 2025-08-14T21:42:19.0241037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0241114Z return func(*args, **kwargs) 2025-08-14T21:42:19.0241391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:19.0241490Z self.key(current_states) 2025-08-14T21:42:19.0241495Z 2025-08-14T21:42:19.0241617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0241834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0241906Z return mod(**inputs) 2025-08-14T21:42:19.0242184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0242254Z outputs = self.bert( 2025-08-14T21:42:19.0242523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0242613Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0242911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0242998Z layer_outputs = layer_module( 2025-08-14T21:42:19.0243233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0243318Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0243584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0243674Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0243929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0244009Z return func(*args, **kwargs) 2025-08-14T21:42:19.0244268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0244351Z self_outputs = self.self( 2025-08-14T21:42:19.0244606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0244682Z return func(*args, **kwargs) 2025-08-14T21:42:19.0244950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:19.0245028Z self.value(current_states) 2025-08-14T21:42:19.0245032Z 2025-08-14T21:42:19.0245129Z cudagraph partition due to non gpu ops 2025-08-14T21:42:19.0245244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0245457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0245534Z return mod(**inputs) 2025-08-14T21:42:19.0245797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0245870Z outputs = self.bert( 2025-08-14T21:42:19.0246147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0246223Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0246467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0246537Z layer_outputs = layer_module( 2025-08-14T21:42:19.0246756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0246858Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0247104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0247185Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0247436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0247505Z return func(*args, **kwargs) 2025-08-14T21:42:19.0247758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0247847Z self_outputs = self.self( 2025-08-14T21:42:19.0248081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0248158Z return func(*args, **kwargs) 2025-08-14T21:42:19.0248399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0248530Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0248542Z 2025-08-14T21:42:19.0248646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0248839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0248941Z return mod(**inputs) 2025-08-14T21:42:19.0249188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0249256Z outputs = self.bert( 2025-08-14T21:42:19.0249506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0249580Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0249825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0249900Z layer_outputs = layer_module( 2025-08-14T21:42:19.0250115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0250200Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0250447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0250530Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0250781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0250853Z return func(*args, **kwargs) 2025-08-14T21:42:19.0251106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0251238Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0251486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0251581Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0251585Z 2025-08-14T21:42:19.0251691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0251899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0251969Z return mod(**inputs) 2025-08-14T21:42:19.0252220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0252293Z outputs = self.bert( 2025-08-14T21:42:19.0252545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0252618Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0252873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0252969Z layer_outputs = layer_module( 2025-08-14T21:42:19.0253198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0253278Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0253526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0253622Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0253886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0253984Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0254271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0254396Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0254716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0254801Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0254805Z 2025-08-14T21:42:19.0254910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0255147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0255216Z return mod(**inputs) 2025-08-14T21:42:19.0255498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0255566Z outputs = self.bert( 2025-08-14T21:42:19.0255839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0255922Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0256170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0256242Z layer_outputs = layer_module( 2025-08-14T21:42:19.0256483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0256561Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0256836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0256920Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0257186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0257269Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0257525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0257639Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0257880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0257986Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0258191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0258259Z return self.act(input) 2025-08-14T21:42:19.0258263Z 2025-08-14T21:42:19.0258361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0258554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0258613Z return mod(**inputs) 2025-08-14T21:42:19.0258853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0258932Z outputs = self.bert( 2025-08-14T21:42:19.0259177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0259256Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0259544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0259610Z layer_outputs = layer_module( 2025-08-14T21:42:19.0259830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0259903Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0260152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0260231Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0260479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0260562Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0260836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0260975Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0261286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0261369Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0261372Z 2025-08-14T21:42:19.0261482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0261679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0261745Z return mod(**inputs) 2025-08-14T21:42:19.0262001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0262071Z outputs = self.bert( 2025-08-14T21:42:19.0262327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0262402Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0262647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0262730Z layer_outputs = layer_module( 2025-08-14T21:42:19.0262954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0263038Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0263290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0263374Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0263624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0263701Z return func(*args, **kwargs) 2025-08-14T21:42:19.0263946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0264028Z self_outputs = self.self( 2025-08-14T21:42:19.0264269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0264352Z return func(*args, **kwargs) 2025-08-14T21:42:19.0264593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:19.0264804Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:19.0264808Z 2025-08-14T21:42:19.0264921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0265120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0265208Z return mod(**inputs) 2025-08-14T21:42:19.0265460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0265526Z outputs = self.bert( 2025-08-14T21:42:19.0265778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0265853Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0266095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0266208Z layer_outputs = layer_module( 2025-08-14T21:42:19.0266424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0266508Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0266749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0266829Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0267077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0267146Z return func(*args, **kwargs) 2025-08-14T21:42:19.0267432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0267508Z self_outputs = self.self( 2025-08-14T21:42:19.0267742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0267818Z return func(*args, **kwargs) 2025-08-14T21:42:19.0268052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:19.0268121Z self.key(current_states) 2025-08-14T21:42:19.0268125Z 2025-08-14T21:42:19.0268232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0268422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0268485Z return mod(**inputs) 2025-08-14T21:42:19.0268732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0268797Z outputs = self.bert( 2025-08-14T21:42:19.0269039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0269112Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0269344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0269420Z layer_outputs = layer_module( 2025-08-14T21:42:19.0269629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0269705Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0269947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0270026Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0270265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0270333Z return func(*args, **kwargs) 2025-08-14T21:42:19.0270570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0270647Z self_outputs = self.self( 2025-08-14T21:42:19.0270881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0270955Z return func(*args, **kwargs) 2025-08-14T21:42:19.0271214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:19.0271288Z self.value(current_states) 2025-08-14T21:42:19.0271291Z 2025-08-14T21:42:19.0271381Z cudagraph partition due to non gpu ops 2025-08-14T21:42:19.0271484Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0271691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0271770Z return mod(**inputs) 2025-08-14T21:42:19.0272029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0272125Z outputs = self.bert( 2025-08-14T21:42:19.0272385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0272462Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0272725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0272801Z layer_outputs = layer_module( 2025-08-14T21:42:19.0273030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0273119Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0273408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0273499Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0273749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0273824Z return func(*args, **kwargs) 2025-08-14T21:42:19.0274084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0274157Z self_outputs = self.self( 2025-08-14T21:42:19.0274416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0274489Z return func(*args, **kwargs) 2025-08-14T21:42:19.0274741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0274886Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0274893Z 2025-08-14T21:42:19.0275001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0275203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0275281Z return mod(**inputs) 2025-08-14T21:42:19.0275539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0275614Z outputs = self.bert( 2025-08-14T21:42:19.0275947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0276034Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0276297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0276375Z layer_outputs = layer_module( 2025-08-14T21:42:19.0276604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0276695Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0276951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0277045Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0277293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0277365Z return func(*args, **kwargs) 2025-08-14T21:42:19.0277651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0277787Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0278047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0278136Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0278144Z 2025-08-14T21:42:19.0278252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0278465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0278557Z return mod(**inputs) 2025-08-14T21:42:19.0278814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0278892Z outputs = self.bert( 2025-08-14T21:42:19.0279152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0279235Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0279488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0279563Z layer_outputs = layer_module( 2025-08-14T21:42:19.0279832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0279916Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0280171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0280269Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0280537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0280627Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0280915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0281040Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0281303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0281392Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0281396Z 2025-08-14T21:42:19.0281510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0281720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0281789Z return mod(**inputs) 2025-08-14T21:42:19.0282056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0282127Z outputs = self.bert( 2025-08-14T21:42:19.0282384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0282466Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0282719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0282801Z layer_outputs = layer_module( 2025-08-14T21:42:19.0283031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0283111Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0283383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0283465Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0283725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0283818Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0284090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0284214Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0284460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0284573Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0284790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0284889Z return self.act(input) 2025-08-14T21:42:19.0284893Z 2025-08-14T21:42:19.0285004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0285201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0285270Z return mod(**inputs) 2025-08-14T21:42:19.0285530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0285597Z outputs = self.bert( 2025-08-14T21:42:19.0285861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0285935Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0286210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0286291Z layer_outputs = layer_module( 2025-08-14T21:42:19.0286509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0286587Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0286838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0286922Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0287195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0287269Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0287542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0287678Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0287916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0287998Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0288007Z 2025-08-14T21:42:19.0288106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0288297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0288370Z return mod(**inputs) 2025-08-14T21:42:19.0288610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0288675Z outputs = self.bert( 2025-08-14T21:42:19.0288930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0289005Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0289255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0289328Z layer_outputs = layer_module( 2025-08-14T21:42:19.0289549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0289634Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0289877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0289981Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0290227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0290297Z return func(*args, **kwargs) 2025-08-14T21:42:19.0290549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0290619Z self_outputs = self.self( 2025-08-14T21:42:19.0290857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0290954Z return func(*args, **kwargs) 2025-08-14T21:42:19.0291193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:19.0291395Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:19.0291407Z 2025-08-14T21:42:19.0291510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0291704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0291775Z return mod(**inputs) 2025-08-14T21:42:19.0292049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0292116Z outputs = self.bert( 2025-08-14T21:42:19.0292363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0292438Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0292682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0292752Z layer_outputs = layer_module( 2025-08-14T21:42:19.0292971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0293056Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0293306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0293391Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0293652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0293726Z return func(*args, **kwargs) 2025-08-14T21:42:19.0293985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0294059Z self_outputs = self.self( 2025-08-14T21:42:19.0294307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0294390Z return func(*args, **kwargs) 2025-08-14T21:42:19.0294647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:19.0294728Z self.key(current_states) 2025-08-14T21:42:19.0294731Z 2025-08-14T21:42:19.0294839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0295047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0295121Z return mod(**inputs) 2025-08-14T21:42:19.0295364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0295431Z outputs = self.bert( 2025-08-14T21:42:19.0295680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0295753Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0295997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0296088Z layer_outputs = layer_module( 2025-08-14T21:42:19.0296300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0296386Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0296625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0296709Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0296963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0297055Z return func(*args, **kwargs) 2025-08-14T21:42:19.0297314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0297388Z self_outputs = self.self( 2025-08-14T21:42:19.0297637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0297724Z return func(*args, **kwargs) 2025-08-14T21:42:19.0297978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:19.0298053Z self.value(current_states) 2025-08-14T21:42:19.0298063Z 2025-08-14T21:42:19.0298178Z cudagraph partition due to non gpu ops 2025-08-14T21:42:19.0298290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0298501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0298573Z return mod(**inputs) 2025-08-14T21:42:19.0298832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0298903Z outputs = self.bert( 2025-08-14T21:42:19.0299148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0299228Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0299469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0299540Z layer_outputs = layer_module( 2025-08-14T21:42:19.0299766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0299845Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0300087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0300177Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0300414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0300495Z return func(*args, **kwargs) 2025-08-14T21:42:19.0300751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0300823Z self_outputs = self.self( 2025-08-14T21:42:19.0301080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0301153Z return func(*args, **kwargs) 2025-08-14T21:42:19.0301408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0301550Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0301553Z 2025-08-14T21:42:19.0301657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0301859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0301924Z return mod(**inputs) 2025-08-14T21:42:19.0302187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0302260Z outputs = self.bert( 2025-08-14T21:42:19.0302513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0302596Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0302851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0302926Z layer_outputs = layer_module( 2025-08-14T21:42:19.0303181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0303262Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0303512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0303605Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0303854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0303933Z return func(*args, **kwargs) 2025-08-14T21:42:19.0304187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0304378Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0304639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0304728Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0304732Z 2025-08-14T21:42:19.0304846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0305050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0305122Z return mod(**inputs) 2025-08-14T21:42:19.0305383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0305452Z outputs = self.bert( 2025-08-14T21:42:19.0305706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0305790Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0306044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0306126Z layer_outputs = layer_module( 2025-08-14T21:42:19.0306355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0306437Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0306694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0306785Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0307054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0307142Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0307438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0307569Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0307814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0307901Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0307904Z 2025-08-14T21:42:19.0308018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0308224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0308327Z return mod(**inputs) 2025-08-14T21:42:19.0308588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0308782Z outputs = self.bert( 2025-08-14T21:42:19.0309062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0309140Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0309398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0309535Z layer_outputs = layer_module( 2025-08-14T21:42:19.0309767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0309856Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0310113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0310200Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0310482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0310563Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0310922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0311059Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0311315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0311442Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0311661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0311735Z return self.act(input) 2025-08-14T21:42:19.0311741Z 2025-08-14T21:42:19.0311857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0312062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0312139Z return mod(**inputs) 2025-08-14T21:42:19.0312398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0312468Z outputs = self.bert( 2025-08-14T21:42:19.0312733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0312810Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0313064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0313145Z layer_outputs = layer_module( 2025-08-14T21:42:19.0313369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0313461Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0313713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0313803Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0314084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0314164Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0314457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0314596Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0314849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0314972Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0314976Z 2025-08-14T21:42:19.0315083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0315288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0315364Z return mod(**inputs) 2025-08-14T21:42:19.0315623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0315698Z outputs = self.bert( 2025-08-14T21:42:19.0316013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0316125Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0316404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0316481Z layer_outputs = layer_module( 2025-08-14T21:42:19.0316732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0316822Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0317083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0317176Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0317473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0317548Z return func(*args, **kwargs) 2025-08-14T21:42:19.0317808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0317879Z self_outputs = self.self( 2025-08-14T21:42:19.0318132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0318206Z return func(*args, **kwargs) 2025-08-14T21:42:19.0318453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:19.0318671Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:19.0318675Z 2025-08-14T21:42:19.0318783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0318999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0319067Z return mod(**inputs) 2025-08-14T21:42:19.0319326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0319403Z outputs = self.bert( 2025-08-14T21:42:19.0319660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0319735Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0319995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0320069Z layer_outputs = layer_module( 2025-08-14T21:42:19.0320303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0320385Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0320637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0320730Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0320985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0321058Z return func(*args, **kwargs) 2025-08-14T21:42:19.0321316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0321406Z self_outputs = self.self( 2025-08-14T21:42:19.0321662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0321735Z return func(*args, **kwargs) 2025-08-14T21:42:19.0321988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:19.0322069Z self.key(current_states) 2025-08-14T21:42:19.0322075Z 2025-08-14T21:42:19.0322182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0322391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0322485Z return mod(**inputs) 2025-08-14T21:42:19.0322747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0322822Z outputs = self.bert( 2025-08-14T21:42:19.0323085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0323161Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0323427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0323502Z layer_outputs = layer_module( 2025-08-14T21:42:19.0323768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0323849Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0324106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0324198Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0324445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0324518Z return func(*args, **kwargs) 2025-08-14T21:42:19.0324783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0324848Z self_outputs = self.self( 2025-08-14T21:42:19.0325083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0325148Z return func(*args, **kwargs) 2025-08-14T21:42:19.0325382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:19.0325460Z self.value(current_states) 2025-08-14T21:42:19.0325465Z 2025-08-14T21:42:19.0325544Z cudagraph partition due to non gpu ops 2025-08-14T21:42:19.0325643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0325841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0325906Z return mod(**inputs) 2025-08-14T21:42:19.0326154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0326218Z outputs = self.bert( 2025-08-14T21:42:19.0326453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0326532Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0326766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0326834Z layer_outputs = layer_module( 2025-08-14T21:42:19.0327049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0327123Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0327360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0327458Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0327686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0327760Z return func(*args, **kwargs) 2025-08-14T21:42:19.0327993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0328070Z self_outputs = self.self( 2025-08-14T21:42:19.0328299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0328386Z return func(*args, **kwargs) 2025-08-14T21:42:19.0328635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0328764Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0328767Z 2025-08-14T21:42:19.0328870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0329072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0329135Z return mod(**inputs) 2025-08-14T21:42:19.0329390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0329453Z outputs = self.bert( 2025-08-14T21:42:19.0329735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0329815Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0330051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0330118Z layer_outputs = layer_module( 2025-08-14T21:42:19.0330335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0330413Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0330652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0330730Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0330960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0331038Z return func(*args, **kwargs) 2025-08-14T21:42:19.0331271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0331405Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0331638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0331718Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0331723Z 2025-08-14T21:42:19.0331831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0332020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0332086Z return mod(**inputs) 2025-08-14T21:42:19.0332340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0332403Z outputs = self.bert( 2025-08-14T21:42:19.0332660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0332735Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0332977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0333056Z layer_outputs = layer_module( 2025-08-14T21:42:19.0333272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0333369Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0333613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0333695Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0333958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0334036Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0334308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0334451Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0334692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0334780Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0334785Z 2025-08-14T21:42:19.0334886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0335079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0335151Z return mod(**inputs) 2025-08-14T21:42:19.0335396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0335492Z outputs = self.bert( 2025-08-14T21:42:19.0335751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0335823Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0336060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0336128Z layer_outputs = layer_module( 2025-08-14T21:42:19.0336340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0336426Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0336667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0336757Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0337017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0337093Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0337372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0337491Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0337738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0337857Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0338067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0338144Z return self.act(input) 2025-08-14T21:42:19.0338147Z 2025-08-14T21:42:19.0338250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0338444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0338520Z return mod(**inputs) 2025-08-14T21:42:19.0338768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0338844Z outputs = self.bert( 2025-08-14T21:42:19.0339090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0339163Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0339416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0339509Z layer_outputs = layer_module( 2025-08-14T21:42:19.0339729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0339815Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0340062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0340153Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0340408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0340509Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0340787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0340920Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0341168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0341259Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0341262Z 2025-08-14T21:42:19.0341364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0341599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0341670Z return mod(**inputs) 2025-08-14T21:42:19.0341952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0342032Z outputs = self.bert( 2025-08-14T21:42:19.0342305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0342384Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0342631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0342701Z layer_outputs = layer_module( 2025-08-14T21:42:19.0342929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0343008Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0343281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0343374Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0343641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0343721Z return func(*args, **kwargs) 2025-08-14T21:42:19.0343991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0344065Z self_outputs = self.self( 2025-08-14T21:42:19.0344329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0344402Z return func(*args, **kwargs) 2025-08-14T21:42:19.0344671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:19.0344899Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:19.0344903Z 2025-08-14T21:42:19.0345011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0345228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0345297Z return mod(**inputs) 2025-08-14T21:42:19.0345574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0345670Z outputs = self.bert( 2025-08-14T21:42:19.0345944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0346027Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0346297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0346369Z layer_outputs = layer_module( 2025-08-14T21:42:19.0346607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0346707Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0346971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0347064Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0347326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0347408Z return func(*args, **kwargs) 2025-08-14T21:42:19.0347663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0347734Z self_outputs = self.self( 2025-08-14T21:42:19.0347993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0348101Z return func(*args, **kwargs) 2025-08-14T21:42:19.0348362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:19.0348438Z self.key(current_states) 2025-08-14T21:42:19.0348442Z 2025-08-14T21:42:19.0348550Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0348771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0348836Z return mod(**inputs) 2025-08-14T21:42:19.0349081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0349153Z outputs = self.bert( 2025-08-14T21:42:19.0349397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0349481Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0349734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0349810Z layer_outputs = layer_module( 2025-08-14T21:42:19.0350044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0350124Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0350375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0350468Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0350719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0350795Z return func(*args, **kwargs) 2025-08-14T21:42:19.0351047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0351121Z self_outputs = self.self( 2025-08-14T21:42:19.0351378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0351451Z return func(*args, **kwargs) 2025-08-14T21:42:19.0351709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:19.0351784Z self.value(current_states) 2025-08-14T21:42:19.0351788Z 2025-08-14T21:42:19.0351873Z cudagraph partition due to non gpu ops 2025-08-14T21:42:19.0352008Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0352214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0352282Z return mod(**inputs) 2025-08-14T21:42:19.0352549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0352617Z outputs = self.bert( 2025-08-14T21:42:19.0352890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0352989Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0353252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0353335Z layer_outputs = layer_module( 2025-08-14T21:42:19.0353573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0353659Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0353926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0354012Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0354279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0354402Z return func(*args, **kwargs) 2025-08-14T21:42:19.0354661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0354744Z self_outputs = self.self( 2025-08-14T21:42:19.0354995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0355069Z return func(*args, **kwargs) 2025-08-14T21:42:19.0355332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0355472Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0355476Z 2025-08-14T21:42:19.0355592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0355878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0355958Z return mod(**inputs) 2025-08-14T21:42:19.0356240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0356312Z outputs = self.bert( 2025-08-14T21:42:19.0356584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0356666Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0356928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0357017Z layer_outputs = layer_module( 2025-08-14T21:42:19.0357253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0357332Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0357586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0357671Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0357919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0357991Z return func(*args, **kwargs) 2025-08-14T21:42:19.0358234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0358370Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0358632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0358730Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0358734Z 2025-08-14T21:42:19.0358838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0359034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0359109Z return mod(**inputs) 2025-08-14T21:42:19.0359364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0359447Z outputs = self.bert( 2025-08-14T21:42:19.0359690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0359761Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0360002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0360073Z layer_outputs = layer_module( 2025-08-14T21:42:19.0360282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0360365Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0360595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0360710Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0360971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0361047Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0361316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0361431Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0361664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0361751Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0361755Z 2025-08-14T21:42:19.0361855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0362056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0362125Z return mod(**inputs) 2025-08-14T21:42:19.0362367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0362441Z outputs = self.bert( 2025-08-14T21:42:19.0362684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0362757Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0363004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0363076Z layer_outputs = layer_module( 2025-08-14T21:42:19.0363298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0363375Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0363618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0363719Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0363967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0364049Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0364314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0365371Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0365608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0365713Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0365908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0365985Z return self.act(input) 2025-08-14T21:42:19.0365988Z 2025-08-14T21:42:19.0366085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0366274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0366363Z return mod(**inputs) 2025-08-14T21:42:19.0366600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0366669Z outputs = self.bert( 2025-08-14T21:42:19.0366912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0366984Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0367227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0367296Z layer_outputs = layer_module( 2025-08-14T21:42:19.0367554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0367629Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0367856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0367942Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0368187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0368268Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0368528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0368654Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0368895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0368976Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0368980Z 2025-08-14T21:42:19.0369171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0369361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0369432Z return mod(**inputs) 2025-08-14T21:42:19.0369669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0369740Z outputs = self.bert( 2025-08-14T21:42:19.0369981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0370053Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0370297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0370367Z layer_outputs = layer_module( 2025-08-14T21:42:19.0370584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0370669Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0370910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0371000Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0371248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0371334Z return func(*args, **kwargs) 2025-08-14T21:42:19.0371580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0371649Z self_outputs = self.self( 2025-08-14T21:42:19.0371884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0371965Z return func(*args, **kwargs) 2025-08-14T21:42:19.0372204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:42:19.0372425Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:19.0372429Z 2025-08-14T21:42:19.0372529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0372722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0372796Z return mod(**inputs) 2025-08-14T21:42:19.0373041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0373114Z outputs = self.bert( 2025-08-14T21:42:19.0373361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0373467Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0373726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0373797Z layer_outputs = layer_module( 2025-08-14T21:42:19.0374006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0374089Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0374319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0374409Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0374636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0374702Z return func(*args, **kwargs) 2025-08-14T21:42:19.0374944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0375011Z self_outputs = self.self( 2025-08-14T21:42:19.0375245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0375314Z return func(*args, **kwargs) 2025-08-14T21:42:19.0375545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:42:19.0375620Z self.key(current_states) 2025-08-14T21:42:19.0375625Z 2025-08-14T21:42:19.0375724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0375914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0375984Z return mod(**inputs) 2025-08-14T21:42:19.0376220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0376293Z outputs = self.bert( 2025-08-14T21:42:19.0376530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0376603Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0376847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0376915Z layer_outputs = layer_module( 2025-08-14T21:42:19.0377129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0377238Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0377476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0377563Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0377797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0377867Z return func(*args, **kwargs) 2025-08-14T21:42:19.0378111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0378207Z self_outputs = self.self( 2025-08-14T21:42:19.0378451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0378519Z return func(*args, **kwargs) 2025-08-14T21:42:19.0378761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:42:19.0378840Z self.value(current_states) 2025-08-14T21:42:19.0378844Z 2025-08-14T21:42:19.0378925Z cudagraph partition due to non gpu ops 2025-08-14T21:42:19.0379028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0379231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0379326Z return mod(**inputs) 2025-08-14T21:42:19.0379584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0379652Z outputs = self.bert( 2025-08-14T21:42:19.0379922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0380004Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0380262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0380336Z layer_outputs = layer_module( 2025-08-14T21:42:19.0380571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0380647Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0380893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0380975Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0381209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0381286Z return func(*args, **kwargs) 2025-08-14T21:42:19.0381578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:42:19.0381651Z self_outputs = self.self( 2025-08-14T21:42:19.0381904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0381976Z return func(*args, **kwargs) 2025-08-14T21:42:19.0382240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:42:19.0382380Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:19.0382384Z 2025-08-14T21:42:19.0382495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0382707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0382778Z return mod(**inputs) 2025-08-14T21:42:19.0383053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0383121Z outputs = self.bert( 2025-08-14T21:42:19.0383390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0383497Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0383761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0383835Z layer_outputs = layer_module( 2025-08-14T21:42:19.0384111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0384192Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0384439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:42:19.0384537Z self_attention_outputs = self.attention( 2025-08-14T21:42:19.0384829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:19.0384913Z return func(*args, **kwargs) 2025-08-14T21:42:19.0385169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:42:19.0385303Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:19.0385567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:42:19.0385649Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0385653Z 2025-08-14T21:42:19.0385793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0385991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0386057Z return mod(**inputs) 2025-08-14T21:42:19.0386315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0386378Z outputs = self.bert( 2025-08-14T21:42:19.0386631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0386705Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0386963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0387046Z layer_outputs = layer_module( 2025-08-14T21:42:19.0387277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0387359Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0387629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0387720Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0388000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0388082Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0388372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0388504Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0388759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:42:19.0388851Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0388858Z 2025-08-14T21:42:19.0388967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0389174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0389251Z return mod(**inputs) 2025-08-14T21:42:19.0389509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0389578Z outputs = self.bert( 2025-08-14T21:42:19.0389869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0389944Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0390208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0390281Z layer_outputs = layer_module( 2025-08-14T21:42:19.0390517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0390606Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0390881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0390967Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0391243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0391326Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0391618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:42:19.0391742Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:19.0391997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:42:19.0392155Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:19.0392377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:19.0392461Z return self.act(input) 2025-08-14T21:42:19.0392465Z 2025-08-14T21:42:19.0392572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0392778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0392857Z return mod(**inputs) 2025-08-14T21:42:19.0393114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:42:19.0393184Z outputs = self.bert( 2025-08-14T21:42:19.0393447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:42:19.0393523Z encoder_outputs = self.encoder( 2025-08-14T21:42:19.0393787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:42:19.0393863Z layer_outputs = layer_module( 2025-08-14T21:42:19.0394093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:19.0394181Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:19.0394434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:42:19.0394529Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:19.0394796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:19.0394875Z return forward_fn(*input_tensors) 2025-08-14T21:42:19.0395169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:42:19.0395310Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:19.0395564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:42:19.0395657Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:19.0395661Z 2025-08-14T21:42:19.0395767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0396076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0396174Z return mod(**inputs) 2025-08-14T21:42:19.0396442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1781, in forward 2025-08-14T21:42:19.0396541Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:42:19.0396545Z 2025-08-14T21:42:19.0396658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0396881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0396953Z return mod(**inputs) 2025-08-14T21:42:19.0397219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1799, in forward 2025-08-14T21:42:19.0397359Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:42:19.0397363Z 2025-08-14T21:42:19.0397474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:19.0397689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:19.0397768Z return mod(**inputs) 2025-08-14T21:42:19.0398025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1800, in forward 2025-08-14T21:42:19.0398131Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:42:19.0398135Z 2025-08-14T21:42:26.9050800Z Compilation time (from dynamo_timed): 14.271465772 2025-08-14T21:42:26.9051263Z pass 2025-08-14T21:42:26.9054351Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:26.9056411Z TIMING: _recursive_pre_grad_passes:0.00737 _recursive_joint_graph_passes:0.37478 _recursive_post_grad_passes:0.08854 async_compile.wait:0.00223 code_gen:6.79383 inductor_compile:8.00111 backend_compile:11.16096 gc:0.00017 entire_frame_compile:14.27147 total_wall_time:14.27147 2025-08-14T21:42:26.9057400Z STATS: call_* op count: 296 | FakeTensorMode.__torch_dispatch__:12371 | FakeTensor.__torch_dispatch__:4710 | ProxyTorchDispatchMode.__torch_dispatch__:4531 2025-08-14T21:42:26.9057938Z Dynamo produced 1 graphs covering 296 ops with 0 graph breaks (0 unique) 2025-08-14T21:42:32.2159314Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:42:32.2160260Z from pkg_resources import resource_filename 2025-08-14T21:42:32.7952338Z 2025-08-14T21:42:52.0828001Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:42:52.0829482Z loading model: 0it [00:19, ?it/s] 2025-08-14T21:42:52.0860039Z cpu eval BlenderbotForCausalLM 2025-08-14T21:42:52.2878739Z Compilation time (from dynamo_timed): 0 2025-08-14T21:42:52.2879217Z pass_due_to_skip 2025-08-14T21:42:52.2885857Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:52.2886464Z TIMING: total_wall_time:0 2025-08-14T21:42:52.2886791Z STATS: call_* op count: 0 2025-08-14T21:42:52.2887660Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-08-14T21:42:57.0319001Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:42:57.0322004Z from pkg_resources import resource_filename 2025-08-14T21:42:57.6219955Z 2025-08-14T21:42:58.4986792Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:42:58.4991592Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:42:58.4995958Z cpu eval BlenderbotSmallForCausalLM 2025-08-14T21:42:58.6645212Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:58.7183439Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:58.7694195Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:04.4836971Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4841121Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4841527Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4846234Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4851297Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4856529Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4861864Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4862163Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4862428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4862895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4863301Z return mod(**inputs) 2025-08-14T21:43:04.4863772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4864234Z outputs = self.model.decoder( 2025-08-14T21:43:04.4864695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4865398Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4865784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4866159Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4866615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4867091Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4867572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:04.4868091Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:04.4868309Z 2025-08-14T21:43:04.4868421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4868810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4869132Z return mod(**inputs) 2025-08-14T21:43:04.4869558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4870010Z outputs = self.model.decoder( 2025-08-14T21:43:04.4870466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4870934Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4871316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4871724Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4872195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4872681Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4873172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:04.4873653Z key_states = self.k_proj(current_states) 2025-08-14T21:43:04.4873805Z 2025-08-14T21:43:04.4873922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4874318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4874776Z return mod(**inputs) 2025-08-14T21:43:04.4875235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4875701Z outputs = self.model.decoder( 2025-08-14T21:43:04.4876384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4876873Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4877258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4877691Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4878152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4878625Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4879107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:04.4879616Z value_states = self.v_proj(current_states) 2025-08-14T21:43:04.4879782Z 2025-08-14T21:43:04.4879872Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4880109Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4880331Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4880636Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4880898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4881280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4881636Z return mod(**inputs) 2025-08-14T21:43:04.4882088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4882565Z outputs = self.model.decoder( 2025-08-14T21:43:04.4883028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4883503Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4883881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4884274Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4884752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4885251Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4885693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.4886136Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.4886560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:04.4887019Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:04.4887194Z 2025-08-14T21:43:04.4887301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4887636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4887945Z return mod(**inputs) 2025-08-14T21:43:04.4888338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4888753Z outputs = self.model.decoder( 2025-08-14T21:43:04.4889152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4889572Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4889933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4890273Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4890696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4891143Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4891585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.4892051Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.4892491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:04.4892948Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:04.4893120Z 2025-08-14T21:43:04.4893238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4893614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4893940Z return mod(**inputs) 2025-08-14T21:43:04.4894356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4894794Z outputs = self.model.decoder( 2025-08-14T21:43:04.4895252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4895716Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4896052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4896392Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4896830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4897286Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4897738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:04.4898169Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:04.4898310Z 2025-08-14T21:43:04.4898416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4898764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4899085Z return mod(**inputs) 2025-08-14T21:43:04.4899482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4899915Z outputs = self.model.decoder( 2025-08-14T21:43:04.4900342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4900767Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4901106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4901462Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4901893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.4902381Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.4902560Z 2025-08-14T21:43:04.4902665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4903016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4903333Z return mod(**inputs) 2025-08-14T21:43:04.4903733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4904179Z outputs = self.model.decoder( 2025-08-14T21:43:04.4904645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4905062Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4905405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4905754Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4906195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.4906661Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.4907036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:04.4907372Z return self.act(input) 2025-08-14T21:43:04.4907480Z 2025-08-14T21:43:04.4907581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4907924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4908239Z return mod(**inputs) 2025-08-14T21:43:04.4909083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4909526Z outputs = self.model.decoder( 2025-08-14T21:43:04.4909958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4910419Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4910775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4911139Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4911589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:04.4912043Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:04.4912185Z 2025-08-14T21:43:04.4912293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4912664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4912998Z return mod(**inputs) 2025-08-14T21:43:04.4913424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4913874Z outputs = self.model.decoder( 2025-08-14T21:43:04.4914314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4914761Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4915131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4915513Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4916046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4916549Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4917029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:04.4917586Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:04.4917800Z 2025-08-14T21:43:04.4917905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4918262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4918611Z return mod(**inputs) 2025-08-14T21:43:04.4919025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4919463Z outputs = self.model.decoder( 2025-08-14T21:43:04.4919897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4920326Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4920672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4921061Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4921492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4921954Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4922417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:04.4922861Z key_states = self.k_proj(current_states) 2025-08-14T21:43:04.4922996Z 2025-08-14T21:43:04.4923101Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4923497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4923826Z return mod(**inputs) 2025-08-14T21:43:04.4924238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4924672Z outputs = self.model.decoder( 2025-08-14T21:43:04.4925097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4925534Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4925877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4926239Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4926674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4927136Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4927583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:04.4928032Z value_states = self.v_proj(current_states) 2025-08-14T21:43:04.4928179Z 2025-08-14T21:43:04.4928264Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4928478Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4928681Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4928892Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4929128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4929477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4929799Z return mod(**inputs) 2025-08-14T21:43:04.4930213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4930708Z outputs = self.model.decoder( 2025-08-14T21:43:04.4931140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4931582Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4931935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4932292Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4932749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4933211Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4933683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.4934167Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.4934609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:04.4935126Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:04.4935311Z 2025-08-14T21:43:04.4935424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4935779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4936106Z return mod(**inputs) 2025-08-14T21:43:04.4936522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4936959Z outputs = self.model.decoder( 2025-08-14T21:43:04.4937470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4937954Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4938306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4938663Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4939138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4939602Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4940061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.4940532Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.4941000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:04.4941482Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:04.4941658Z 2025-08-14T21:43:04.4941778Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4942160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4942499Z return mod(**inputs) 2025-08-14T21:43:04.4942901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4943317Z outputs = self.model.decoder( 2025-08-14T21:43:04.4943738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4944189Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4944537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4944891Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4945330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4945787Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4946233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:04.4946676Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:04.4946818Z 2025-08-14T21:43:04.4946944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4947303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4947620Z return mod(**inputs) 2025-08-14T21:43:04.4948035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4948472Z outputs = self.model.decoder( 2025-08-14T21:43:04.4948907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4949351Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4949689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4950048Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4950478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.4950966Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.4951146Z 2025-08-14T21:43:04.4951250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4951609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4951925Z return mod(**inputs) 2025-08-14T21:43:04.4952391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4952862Z outputs = self.model.decoder( 2025-08-14T21:43:04.4953325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4953788Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4954158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4954538Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4954993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.4955500Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.4955973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:04.4956346Z return self.act(input) 2025-08-14T21:43:04.4956466Z 2025-08-14T21:43:04.4956580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4956977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4957331Z return mod(**inputs) 2025-08-14T21:43:04.4957785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4958269Z outputs = self.model.decoder( 2025-08-14T21:43:04.4958733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4959206Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4959729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4960116Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4960590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:04.4961067Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:04.4961215Z 2025-08-14T21:43:04.4961326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4961708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4962072Z return mod(**inputs) 2025-08-14T21:43:04.4962506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4962959Z outputs = self.model.decoder( 2025-08-14T21:43:04.4963420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4963878Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4964235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4964644Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4965107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4965554Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4965999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:04.4966511Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:04.4966721Z 2025-08-14T21:43:04.4966826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4967215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4967540Z return mod(**inputs) 2025-08-14T21:43:04.4967943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4968368Z outputs = self.model.decoder( 2025-08-14T21:43:04.4968784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4969204Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4969539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4969886Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4970299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4970747Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4971191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:04.4971622Z key_states = self.k_proj(current_states) 2025-08-14T21:43:04.4971754Z 2025-08-14T21:43:04.4971855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4972202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4972524Z return mod(**inputs) 2025-08-14T21:43:04.4972920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4973350Z outputs = self.model.decoder( 2025-08-14T21:43:04.4973772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4974203Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4974540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4974901Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4975337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4975797Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4976263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:04.4976709Z value_states = self.v_proj(current_states) 2025-08-14T21:43:04.4976850Z 2025-08-14T21:43:04.4976940Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4977154Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4977361Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4977568Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.4977799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4979111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4979436Z return mod(**inputs) 2025-08-14T21:43:04.4979856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4980295Z outputs = self.model.decoder( 2025-08-14T21:43:04.4980730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4981169Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4981510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4981899Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4982358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4982844Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4983322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.4983797Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.4984262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:04.4984737Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:04.4984920Z 2025-08-14T21:43:04.4985030Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4985382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4985704Z return mod(**inputs) 2025-08-14T21:43:04.4986114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4986544Z outputs = self.model.decoder( 2025-08-14T21:43:04.4986977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4987413Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4987759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4988114Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4988549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4989010Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4989459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.4990042Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.4990489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:04.4990941Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:04.4991129Z 2025-08-14T21:43:04.4991235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4991595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4991937Z return mod(**inputs) 2025-08-14T21:43:04.4992382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4992845Z outputs = self.model.decoder( 2025-08-14T21:43:04.4993304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4993793Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4994166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.4994544Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.4995017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.4995511Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.4996077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:04.4996564Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:04.4996758Z 2025-08-14T21:43:04.4996873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.4997241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.4997561Z return mod(**inputs) 2025-08-14T21:43:04.4998044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.4998534Z outputs = self.model.decoder( 2025-08-14T21:43:04.4999017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.4999497Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.4999888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5000295Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5000771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5001303Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5001498Z 2025-08-14T21:43:04.5001613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5002000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5002357Z return mod(**inputs) 2025-08-14T21:43:04.5002819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5003304Z outputs = self.model.decoder( 2025-08-14T21:43:04.5003785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5004264Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5004658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5005038Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5005460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5005932Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5006308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:04.5006655Z return self.act(input) 2025-08-14T21:43:04.5006763Z 2025-08-14T21:43:04.5006862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5007211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5007528Z return mod(**inputs) 2025-08-14T21:43:04.5007927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5008356Z outputs = self.model.decoder( 2025-08-14T21:43:04.5008954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5009391Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5009724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5010115Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5010583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:04.5011040Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:04.5011183Z 2025-08-14T21:43:04.5011290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5011741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5012061Z return mod(**inputs) 2025-08-14T21:43:04.5012459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5012945Z outputs = self.model.decoder( 2025-08-14T21:43:04.5013378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5013811Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5014150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5014514Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5014962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5015413Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5015854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:04.5016359Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:04.5016558Z 2025-08-14T21:43:04.5016666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5017015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5017321Z return mod(**inputs) 2025-08-14T21:43:04.5017723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5018145Z outputs = self.model.decoder( 2025-08-14T21:43:04.5018558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5018985Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5019328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5019700Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5020156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5020677Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5021139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:04.5021566Z key_states = self.k_proj(current_states) 2025-08-14T21:43:04.5021697Z 2025-08-14T21:43:04.5021798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5022175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5022513Z return mod(**inputs) 2025-08-14T21:43:04.5022943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5023429Z outputs = self.model.decoder( 2025-08-14T21:43:04.5023886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5024352Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5024709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5025090Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5025552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5026068Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5026556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:04.5027035Z value_states = self.v_proj(current_states) 2025-08-14T21:43:04.5027183Z 2025-08-14T21:43:04.5027278Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5027501Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5027727Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5027948Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5028189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5028571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5028915Z return mod(**inputs) 2025-08-14T21:43:04.5029366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5029797Z outputs = self.model.decoder( 2025-08-14T21:43:04.5030234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5030680Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5031028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5031407Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5031877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5032363Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5032837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5033328Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5033797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:04.5034308Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:04.5034505Z 2025-08-14T21:43:04.5034616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5034998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5035357Z return mod(**inputs) 2025-08-14T21:43:04.5035800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5036326Z outputs = self.model.decoder( 2025-08-14T21:43:04.5036811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5037296Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5037679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5038049Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5038477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5038926Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5039368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5039857Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5040321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:04.5040836Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:04.5041009Z 2025-08-14T21:43:04.5041118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5041498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5041811Z return mod(**inputs) 2025-08-14T21:43:04.5042205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5042634Z outputs = self.model.decoder( 2025-08-14T21:43:04.5043054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5043478Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5043809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5044167Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5044605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5045073Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5045523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:04.5045969Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:04.5046106Z 2025-08-14T21:43:04.5046219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5046574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5046896Z return mod(**inputs) 2025-08-14T21:43:04.5047312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5047755Z outputs = self.model.decoder( 2025-08-14T21:43:04.5048181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5048622Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5048969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5049332Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5049782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5050265Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5050438Z 2025-08-14T21:43:04.5050548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5050902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5051218Z return mod(**inputs) 2025-08-14T21:43:04.5051635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5052092Z outputs = self.model.decoder( 2025-08-14T21:43:04.5052522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5052963Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5053324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5053714Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5054166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5054650Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5055075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:04.5055412Z return self.act(input) 2025-08-14T21:43:04.5055537Z 2025-08-14T21:43:04.5055644Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5056009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5056341Z return mod(**inputs) 2025-08-14T21:43:04.5056756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5057206Z outputs = self.model.decoder( 2025-08-14T21:43:04.5057646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5058089Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5058437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5058802Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5059248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:04.5059686Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:04.5059831Z 2025-08-14T21:43:04.5059935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5060286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5060616Z return mod(**inputs) 2025-08-14T21:43:04.5061004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5061432Z outputs = self.model.decoder( 2025-08-14T21:43:04.5061857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5062283Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5062631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5062996Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5063439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5063914Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5064370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:04.5064878Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:04.5065085Z 2025-08-14T21:43:04.5065192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5065531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5065847Z return mod(**inputs) 2025-08-14T21:43:04.5066263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5066684Z outputs = self.model.decoder( 2025-08-14T21:43:04.5067092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5067517Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5067854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5068198Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5068665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5069118Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5069563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:04.5070000Z key_states = self.k_proj(current_states) 2025-08-14T21:43:04.5070145Z 2025-08-14T21:43:04.5070250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5070621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5070964Z return mod(**inputs) 2025-08-14T21:43:04.5071391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5071855Z outputs = self.model.decoder( 2025-08-14T21:43:04.5072313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5072767Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5073135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5073520Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5073983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5074466Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5074959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:04.5075448Z value_states = self.v_proj(current_states) 2025-08-14T21:43:04.5075602Z 2025-08-14T21:43:04.5075700Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5076006Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5076240Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5076465Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5076715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5077113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5077471Z return mod(**inputs) 2025-08-14T21:43:04.5077930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5078411Z outputs = self.model.decoder( 2025-08-14T21:43:04.5078877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5079337Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5079699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5080083Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5080544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5081078Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5081559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5082049Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5082513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:04.5083018Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:04.5083213Z 2025-08-14T21:43:04.5083321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5083733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5084093Z return mod(**inputs) 2025-08-14T21:43:04.5084533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5085008Z outputs = self.model.decoder( 2025-08-14T21:43:04.5085472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5085950Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5086290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5086653Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5087098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5087557Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5088007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5088462Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5088902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:04.5089350Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:04.5089511Z 2025-08-14T21:43:04.5089615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5089968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5090293Z return mod(**inputs) 2025-08-14T21:43:04.5090698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5091136Z outputs = self.model.decoder( 2025-08-14T21:43:04.5091567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5092002Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5092340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5092700Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5093168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5093619Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5094080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:04.5094535Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:04.5094673Z 2025-08-14T21:43:04.5094785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5095150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5095479Z return mod(**inputs) 2025-08-14T21:43:04.5095886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5096320Z outputs = self.model.decoder( 2025-08-14T21:43:04.5096740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5097170Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5097514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5097865Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5098334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5098813Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5098982Z 2025-08-14T21:43:04.5099093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5099440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5099764Z return mod(**inputs) 2025-08-14T21:43:04.5100175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5100609Z outputs = self.model.decoder( 2025-08-14T21:43:04.5101031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5101462Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5101801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5102149Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5102586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5103061Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5103463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:04.5103816Z return self.act(input) 2025-08-14T21:43:04.5103940Z 2025-08-14T21:43:04.5104049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5104423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5104767Z return mod(**inputs) 2025-08-14T21:43:04.5105208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5105644Z outputs = self.model.decoder( 2025-08-14T21:43:04.5106071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5106503Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5106837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5107215Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5107645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:04.5108073Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:04.5108218Z 2025-08-14T21:43:04.5108321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5108798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5109170Z return mod(**inputs) 2025-08-14T21:43:04.5109577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5110017Z outputs = self.model.decoder( 2025-08-14T21:43:04.5110447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5110879Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5111225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5111583Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5112144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5112632Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5113122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:04.5113670Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:04.5113886Z 2025-08-14T21:43:04.5114003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5114379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5114722Z return mod(**inputs) 2025-08-14T21:43:04.5115165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5115636Z outputs = self.model.decoder( 2025-08-14T21:43:04.5116180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5116684Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5117071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5117455Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5117902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5118382Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5118882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:04.5119358Z key_states = self.k_proj(current_states) 2025-08-14T21:43:04.5119514Z 2025-08-14T21:43:04.5119628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5120022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5120373Z return mod(**inputs) 2025-08-14T21:43:04.5120823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5121304Z outputs = self.model.decoder( 2025-08-14T21:43:04.5121774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5122273Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5122648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5123055Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5123535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5124031Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5124528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:04.5125042Z value_states = self.v_proj(current_states) 2025-08-14T21:43:04.5125192Z 2025-08-14T21:43:04.5125273Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5125485Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5125699Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5125906Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5126131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5126485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5126801Z return mod(**inputs) 2025-08-14T21:43:04.5127242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5127673Z outputs = self.model.decoder( 2025-08-14T21:43:04.5128097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5128525Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5128862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5129222Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5129654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5130104Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5130548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5131005Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5131463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:04.5131973Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:04.5132183Z 2025-08-14T21:43:04.5132297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5132690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5133045Z return mod(**inputs) 2025-08-14T21:43:04.5133491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5133971Z outputs = self.model.decoder( 2025-08-14T21:43:04.5134441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5134920Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5135282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5135667Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5136129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5136630Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5137113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5137574Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5138001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:04.5138439Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:04.5138618Z 2025-08-14T21:43:04.5138728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5139127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5139467Z return mod(**inputs) 2025-08-14T21:43:04.5139896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5140364Z outputs = self.model.decoder( 2025-08-14T21:43:04.5140782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5141198Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5141533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5141910Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5142343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5142793Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5143247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:04.5143691Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:04.5143826Z 2025-08-14T21:43:04.5143937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5144290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5144657Z return mod(**inputs) 2025-08-14T21:43:04.5145121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5145558Z outputs = self.model.decoder( 2025-08-14T21:43:04.5145993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5146428Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5146773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5147125Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5147564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5148046Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5148218Z 2025-08-14T21:43:04.5148327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5148677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5149001Z return mod(**inputs) 2025-08-14T21:43:04.5149413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5149849Z outputs = self.model.decoder( 2025-08-14T21:43:04.5150279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5150713Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5151088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5151441Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5151877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5152359Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5152753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:04.5153137Z return self.act(input) 2025-08-14T21:43:04.5153259Z 2025-08-14T21:43:04.5153367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5153742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5154073Z return mod(**inputs) 2025-08-14T21:43:04.5154509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5154980Z outputs = self.model.decoder( 2025-08-14T21:43:04.5155428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5155983Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5156402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5156799Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5157264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:04.5157703Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:04.5157848Z 2025-08-14T21:43:04.5157952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5158313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5158632Z return mod(**inputs) 2025-08-14T21:43:04.5159048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5159489Z outputs = self.model.decoder( 2025-08-14T21:43:04.5159925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5160354Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5160708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5161068Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5161498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5161962Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5162418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:04.5162944Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:04.5163140Z 2025-08-14T21:43:04.5163244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5163591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5163909Z return mod(**inputs) 2025-08-14T21:43:04.5164306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5164724Z outputs = self.model.decoder( 2025-08-14T21:43:04.5165146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5165594Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5165931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5166271Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5166702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5167153Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5167618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:04.5168059Z key_states = self.k_proj(current_states) 2025-08-14T21:43:04.5168199Z 2025-08-14T21:43:04.5168301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5168654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5168962Z return mod(**inputs) 2025-08-14T21:43:04.5169371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5169803Z outputs = self.model.decoder( 2025-08-14T21:43:04.5170261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5170677Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5171016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5171366Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5171785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5172235Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5172674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:04.5173107Z value_states = self.v_proj(current_states) 2025-08-14T21:43:04.5173242Z 2025-08-14T21:43:04.5173321Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5173533Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5173736Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5173931Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5174160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5174505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5174818Z return mod(**inputs) 2025-08-14T21:43:04.5175209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5175635Z outputs = self.model.decoder( 2025-08-14T21:43:04.5176054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5176471Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5176815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5177164Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5177594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5178046Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5178503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5178987Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5179423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:04.5179882Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:04.5180071Z 2025-08-14T21:43:04.5180174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5180530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5180855Z return mod(**inputs) 2025-08-14T21:43:04.5181299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5181736Z outputs = self.model.decoder( 2025-08-14T21:43:04.5182174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5182612Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5182957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5183312Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5183780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5184234Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5184691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5185175Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5185640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:04.5186114Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:04.5186292Z 2025-08-14T21:43:04.5186403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5186784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5187119Z return mod(**inputs) 2025-08-14T21:43:04.5187550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5187992Z outputs = self.model.decoder( 2025-08-14T21:43:04.5188427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5188884Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5189264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5189654Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5190118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5190617Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5191133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:04.5191605Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:04.5191750Z 2025-08-14T21:43:04.5191858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5192237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5192588Z return mod(**inputs) 2025-08-14T21:43:04.5193039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5193548Z outputs = self.model.decoder( 2025-08-14T21:43:04.5194032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5194516Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5194905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5195309Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5195788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5196422Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5196610Z 2025-08-14T21:43:04.5196723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5197116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5197474Z return mod(**inputs) 2025-08-14T21:43:04.5197927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5198413Z outputs = self.model.decoder( 2025-08-14T21:43:04.5198899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5199420Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5199807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5200206Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5200693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5201222Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5201640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:04.5202013Z return self.act(input) 2025-08-14T21:43:04.5202145Z 2025-08-14T21:43:04.5202258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5202650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5203014Z return mod(**inputs) 2025-08-14T21:43:04.5203483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5203967Z outputs = self.model.decoder( 2025-08-14T21:43:04.5204442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5204929Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5205310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5205707Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5206181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:04.5206674Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:04.5206831Z 2025-08-14T21:43:04.5206948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5207350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5207688Z return mod(**inputs) 2025-08-14T21:43:04.5208130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5208595Z outputs = self.model.decoder( 2025-08-14T21:43:04.5209187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5209709Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5210076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5210459Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5210920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5211411Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5211926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:04.5212469Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:04.5212686Z 2025-08-14T21:43:04.5212800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5213183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5213529Z return mod(**inputs) 2025-08-14T21:43:04.5213968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5214427Z outputs = self.model.decoder( 2025-08-14T21:43:04.5214930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5215390Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5215745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5216122Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5216584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5217070Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5217544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:04.5218007Z key_states = self.k_proj(current_states) 2025-08-14T21:43:04.5218148Z 2025-08-14T21:43:04.5218270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5218649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5218989Z return mod(**inputs) 2025-08-14T21:43:04.5219419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5219883Z outputs = self.model.decoder( 2025-08-14T21:43:04.5220327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5220757Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5221096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5221446Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5221865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5222314Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5222761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:04.5223194Z value_states = self.v_proj(current_states) 2025-08-14T21:43:04.5223332Z 2025-08-14T21:43:04.5223411Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5223640Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5223850Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5224045Z cudagraph partition due to non gpu ops 2025-08-14T21:43:04.5224277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5224627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5224938Z return mod(**inputs) 2025-08-14T21:43:04.5225346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5225774Z outputs = self.model.decoder( 2025-08-14T21:43:04.5226218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5226636Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5226979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5227330Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5227761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5228204Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5228681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5229128Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5229548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:04.5230017Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:04.5230203Z 2025-08-14T21:43:04.5230307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5230669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5230989Z return mod(**inputs) 2025-08-14T21:43:04.5231404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5231843Z outputs = self.model.decoder( 2025-08-14T21:43:04.5232287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5232747Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5233126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5233515Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5233971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5234460Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5234942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:04.5235428Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:04.5235950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:04.5236461Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:04.5236647Z 2025-08-14T21:43:04.5236762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5237152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5237519Z return mod(**inputs) 2025-08-14T21:43:04.5237977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5238498Z outputs = self.model.decoder( 2025-08-14T21:43:04.5238969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5239418Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5239789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5240169Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5240626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:04.5241130Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:04.5241630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:04.5242105Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:04.5242250Z 2025-08-14T21:43:04.5242361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5242738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5243084Z return mod(**inputs) 2025-08-14T21:43:04.5243555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5244014Z outputs = self.model.decoder( 2025-08-14T21:43:04.5244470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5244936Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5245296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5245676Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5246139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5246647Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5246831Z 2025-08-14T21:43:04.5246940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5247320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5247666Z return mod(**inputs) 2025-08-14T21:43:04.5248157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5248589Z outputs = self.model.decoder( 2025-08-14T21:43:04.5249020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5249565Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5249953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5250589Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5251104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:04.5251668Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:04.5252165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:04.5252580Z return self.act(input) 2025-08-14T21:43:04.5252717Z 2025-08-14T21:43:04.5252865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5253293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5253686Z return mod(**inputs) 2025-08-14T21:43:04.5254183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:43:04.5254720Z outputs = self.model.decoder( 2025-08-14T21:43:04.5255187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:04.5255682Z layer_outputs = decoder_layer( 2025-08-14T21:43:04.5256173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:04.5256647Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:04.5257109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:04.5257649Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:04.5257814Z 2025-08-14T21:43:04.5257968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5258347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5258778Z return mod(**inputs) 2025-08-14T21:43:04.5259258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1528, in forward 2025-08-14T21:43:04.5259791Z logits = self.lm_head(outputs[0]) 2025-08-14T21:43:04.5259993Z 2025-08-14T21:43:04.5260117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:04.5260537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:04.5260936Z return mod(**inputs) 2025-08-14T21:43:04.5275241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1534, in forward 2025-08-14T21:43:04.5276067Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:43:04.5276418Z 2025-08-14T21:43:12.1382609Z Compilation time (from dynamo_timed): 12.218161634 2025-08-14T21:43:12.1409514Z pass 2025-08-14T21:43:12.1410576Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:12.1417816Z TIMING: _recursive_pre_grad_passes:0.00637 _recursive_joint_graph_passes:0.29261 _recursive_post_grad_passes:0.36285 async_compile.wait:0.73849 code_gen:7.32921 inductor_compile:8.49002 backend_compile:10.63133 gc:0.00153 entire_frame_compile:12.21816 total_wall_time:12.21816 2025-08-14T21:43:12.1422832Z STATS: call_* op count: 252 | FakeTensorMode.__torch_dispatch__:9096 | FakeTensor.__torch_dispatch__:3327 | ProxyTorchDispatchMode.__torch_dispatch__:3279 2025-08-14T21:43:12.1423365Z Dynamo produced 1 graphs covering 252 ops with 0 graph breaks (0 unique) 2025-08-14T21:43:17.4086727Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:17.4089935Z from pkg_resources import resource_filename 2025-08-14T21:43:18.0611385Z 2025-08-14T21:43:19.2122231Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:43:19.2125309Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:43:19.2146211Z cpu eval BlenderbotSmallForConditionalGeneration 2025-08-14T21:43:19.4844690Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:19.5945440Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:19.6927749Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:31.6425058Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6425939Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6426197Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6426449Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6426686Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6426951Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6427191Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6427427Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6427708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6428219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6428708Z return mod(**inputs) 2025-08-14T21:43:31.6429233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6429718Z outputs = self.model( 2025-08-14T21:43:31.6430177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6430661Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6431138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6431611Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6432106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6432518Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6433016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6433528Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6434038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6434625Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6434854Z 2025-08-14T21:43:31.6434998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6435395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6435756Z return mod(**inputs) 2025-08-14T21:43:31.6436317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6436801Z outputs = self.model( 2025-08-14T21:43:31.6437264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6437741Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6438193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6438664Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6439046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6439442Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6439924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6440422Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6440913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6441379Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6441521Z 2025-08-14T21:43:31.6441638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6442047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6442430Z return mod(**inputs) 2025-08-14T21:43:31.6442878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6443338Z outputs = self.model( 2025-08-14T21:43:31.6443801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6444286Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6444758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6445245Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6445623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6446012Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6446488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6446987Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6447484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6448072Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6448222Z 2025-08-14T21:43:31.6448309Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6448540Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6448767Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6448985Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6449244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6449633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6450004Z return mod(**inputs) 2025-08-14T21:43:31.6450461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6450950Z outputs = self.model( 2025-08-14T21:43:31.6451409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6451899Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6452362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6452841Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6453224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6453673Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6454153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6454646Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6455136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6455638Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6456127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6456668Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6456883Z 2025-08-14T21:43:31.6457001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6457414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6457798Z return mod(**inputs) 2025-08-14T21:43:31.6458244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6458706Z outputs = self.model( 2025-08-14T21:43:31.6459144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6459674Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6460131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6460618Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6460988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6461372Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6461910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6462398Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6462877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6463373Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6463881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6464380Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6464556Z 2025-08-14T21:43:31.6464669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6465051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6465392Z return mod(**inputs) 2025-08-14T21:43:31.6465835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6466285Z outputs = self.model( 2025-08-14T21:43:31.6466726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6467191Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6467642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6468109Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6468478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6468855Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6469311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6469788Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6470271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6470754Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6470905Z 2025-08-14T21:43:31.6471023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6471417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6471774Z return mod(**inputs) 2025-08-14T21:43:31.6472214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6472689Z outputs = self.model( 2025-08-14T21:43:31.6473144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6473639Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6474100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6474570Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6474951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6475352Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6475983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6476550Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6476740Z 2025-08-14T21:43:31.6476865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6477254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6477600Z return mod(**inputs) 2025-08-14T21:43:31.6478056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6478525Z outputs = self.model( 2025-08-14T21:43:31.6479025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6479500Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6479965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6480439Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6480808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6481203Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6481683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6482206Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6482620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6482992Z return self.act(input) 2025-08-14T21:43:31.6483112Z 2025-08-14T21:43:31.6483232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6483616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6483970Z return mod(**inputs) 2025-08-14T21:43:31.6484428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6484899Z outputs = self.model( 2025-08-14T21:43:31.6485349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6485821Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6486282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6486748Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6487123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6487579Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6488035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:43:31.6488501Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6488656Z 2025-08-14T21:43:31.6488784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6489161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6489517Z return mod(**inputs) 2025-08-14T21:43:31.6489960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6490425Z outputs = self.model( 2025-08-14T21:43:31.6490869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6491352Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6491811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6492266Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6492631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6493014Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6493485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6493974Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6494493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6495039Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6495263Z 2025-08-14T21:43:31.6495374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6495780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6496126Z return mod(**inputs) 2025-08-14T21:43:31.6496551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6497015Z outputs = self.model( 2025-08-14T21:43:31.6497458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6497921Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6498379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6498840Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6499205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6499590Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6500051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6500541Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6501028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6501493Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6501642Z 2025-08-14T21:43:31.6502204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6502591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6502937Z return mod(**inputs) 2025-08-14T21:43:31.6503367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6503840Z outputs = self.model( 2025-08-14T21:43:31.6504290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6504787Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6505232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6505707Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6506087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6506475Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6506947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6508319Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6509008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6509493Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6509645Z 2025-08-14T21:43:31.6509733Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6509966Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6510188Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6510402Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6510654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6511142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6511487Z return mod(**inputs) 2025-08-14T21:43:31.6511935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6512403Z outputs = self.model( 2025-08-14T21:43:31.6512848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6513314Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6513791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6514263Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6514651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6515031Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6515500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6516061Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6516546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6517057Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6517532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6518034Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6518229Z 2025-08-14T21:43:31.6518341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6518725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6519072Z return mod(**inputs) 2025-08-14T21:43:31.6519511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6519968Z outputs = self.model( 2025-08-14T21:43:31.6520413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6520904Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6521358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6521818Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6522192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6522561Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6522976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6523428Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6523868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6524312Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6524742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6525201Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6525359Z 2025-08-14T21:43:31.6525477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6525811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6526153Z return mod(**inputs) 2025-08-14T21:43:31.6526546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6526969Z outputs = self.model( 2025-08-14T21:43:31.6527380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6527809Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6528226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6528650Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6528983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6529329Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6529739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6530167Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6530595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6531017Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6531147Z 2025-08-14T21:43:31.6531246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6531581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6531890Z return mod(**inputs) 2025-08-14T21:43:31.6532279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6532684Z outputs = self.model( 2025-08-14T21:43:31.6533076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6533496Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6533893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6534309Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6534644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6535052Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6535474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6535931Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6536101Z 2025-08-14T21:43:31.6536204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6536547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6536893Z return mod(**inputs) 2025-08-14T21:43:31.6537306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6537727Z outputs = self.model( 2025-08-14T21:43:31.6538124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6538545Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6538960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6539378Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6539735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6540080Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6540494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6540947Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6541304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6541629Z return self.act(input) 2025-08-14T21:43:31.6541734Z 2025-08-14T21:43:31.6541839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6542171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6542480Z return mod(**inputs) 2025-08-14T21:43:31.6542875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6543285Z outputs = self.model( 2025-08-14T21:43:31.6543668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6544080Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6544486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6544897Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6545225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6545571Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6545998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:43:31.6546451Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6546582Z 2025-08-14T21:43:31.6546684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6547032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6547347Z return mod(**inputs) 2025-08-14T21:43:31.6547740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6548158Z outputs = self.model( 2025-08-14T21:43:31.6548579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6549004Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6549423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6549857Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6550207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6550584Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6551012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6551466Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6551916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6552424Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6552636Z 2025-08-14T21:43:31.6552740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6553094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6553449Z return mod(**inputs) 2025-08-14T21:43:31.6553857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6554293Z outputs = self.model( 2025-08-14T21:43:31.6554707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6555138Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6555578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6556112Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6556479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6556870Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6557390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6557842Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6558298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6558719Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6558858Z 2025-08-14T21:43:31.6558960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6559310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6559627Z return mod(**inputs) 2025-08-14T21:43:31.6560020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6560436Z outputs = self.model( 2025-08-14T21:43:31.6560837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6561250Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6561670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6562091Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6562428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6562832Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6563258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6563708Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6564156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6564591Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6564762Z 2025-08-14T21:43:31.6564844Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6565057Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6565259Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6565467Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6565701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6566061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6566380Z return mod(**inputs) 2025-08-14T21:43:31.6566792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6567293Z outputs = self.model( 2025-08-14T21:43:31.6567723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6568159Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6568587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6569021Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6569361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6569725Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6570160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6570613Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6571055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6571514Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6571954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6572422Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6572614Z 2025-08-14T21:43:31.6572718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6573074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6573399Z return mod(**inputs) 2025-08-14T21:43:31.6573805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6574237Z outputs = self.model( 2025-08-14T21:43:31.6574653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6575090Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6575514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6575949Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6576296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6576668Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6577111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6577563Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6578014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6578468Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6578912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6579386Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6579547Z 2025-08-14T21:43:31.6579659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6580012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6580342Z return mod(**inputs) 2025-08-14T21:43:31.6580753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6581176Z outputs = self.model( 2025-08-14T21:43:31.6581594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6582064Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6582506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6582926Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6583274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6583631Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6584062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6584501Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6584943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6585383Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6585518Z 2025-08-14T21:43:31.6585621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6585973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6586295Z return mod(**inputs) 2025-08-14T21:43:31.6586704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6587127Z outputs = self.model( 2025-08-14T21:43:31.6587535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6587966Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6588391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6588811Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6589159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6589518Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6589945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6590420Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6590599Z 2025-08-14T21:43:31.6590731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6591082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6591392Z return mod(**inputs) 2025-08-14T21:43:31.6591801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6592230Z outputs = self.model( 2025-08-14T21:43:31.6592647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6593114Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6593569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6594028Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6594397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6594788Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6595246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6595750Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6596283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6596670Z return self.act(input) 2025-08-14T21:43:31.6596791Z 2025-08-14T21:43:31.6596913Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6597307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6597662Z return mod(**inputs) 2025-08-14T21:43:31.6598165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6598640Z outputs = self.model( 2025-08-14T21:43:31.6599077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6599551Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6600008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6600463Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6600820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6601200Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6601660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:43:31.6602138Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6602283Z 2025-08-14T21:43:31.6602391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6602766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6603104Z return mod(**inputs) 2025-08-14T21:43:31.6603530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6603986Z outputs = self.model( 2025-08-14T21:43:31.6604418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6604877Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6605329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6605801Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6606169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6606538Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6606999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6607475Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6607955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6608508Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6608957Z 2025-08-14T21:43:31.6609068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6609427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6609751Z return mod(**inputs) 2025-08-14T21:43:31.6610159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6610599Z outputs = self.model( 2025-08-14T21:43:31.6611011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6611514Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6611940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6612378Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6612729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6613083Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6613525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6613984Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6614441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6614880Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6615025Z 2025-08-14T21:43:31.6615131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6615493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6615820Z return mod(**inputs) 2025-08-14T21:43:31.6616229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6616696Z outputs = self.model( 2025-08-14T21:43:31.6617140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6617575Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6618013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6618452Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6618805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6619164Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6619616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6620073Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6620526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6620990Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6621137Z 2025-08-14T21:43:31.6621219Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6621434Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6621645Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6621856Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6622090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6622446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6622793Z return mod(**inputs) 2025-08-14T21:43:31.6623204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6623635Z outputs = self.model( 2025-08-14T21:43:31.6624039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6624476Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6624907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6625371Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6625770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6626153Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6626623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6627111Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6627575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6628045Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6628517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6629019Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6629219Z 2025-08-14T21:43:31.6629331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6629710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6630055Z return mod(**inputs) 2025-08-14T21:43:31.6630491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6630960Z outputs = self.model( 2025-08-14T21:43:31.6631403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6631867Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6632324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6632791Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6633167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6633550Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6634016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6634504Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6634988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6635487Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6636029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6636533Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6636707Z 2025-08-14T21:43:31.6636827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6637217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6637561Z return mod(**inputs) 2025-08-14T21:43:31.6638024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6638474Z outputs = self.model( 2025-08-14T21:43:31.6638916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6639380Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6639840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6640291Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6640659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6641082Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6641539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6642008Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6642456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6642896Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6643030Z 2025-08-14T21:43:31.6643135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6643499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6643832Z return mod(**inputs) 2025-08-14T21:43:31.6644283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6644736Z outputs = self.model( 2025-08-14T21:43:31.6645160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6645604Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6646039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6646481Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6646826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6647180Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6647603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6648077Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6648252Z 2025-08-14T21:43:31.6648359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6648711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6649025Z return mod(**inputs) 2025-08-14T21:43:31.6649433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6649859Z outputs = self.model( 2025-08-14T21:43:31.6650285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6650717Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6651135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6651561Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6651899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6652266Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6652691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6653159Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6653530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6653865Z return self.act(input) 2025-08-14T21:43:31.6653973Z 2025-08-14T21:43:31.6654082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6654435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6654760Z return mod(**inputs) 2025-08-14T21:43:31.6655228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6655688Z outputs = self.model( 2025-08-14T21:43:31.6656100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6656538Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6656974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6657420Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6657752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6658105Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6658533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:43:31.6658956Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6659099Z 2025-08-14T21:43:31.6659204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6659559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6659884Z return mod(**inputs) 2025-08-14T21:43:31.6660287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6660716Z outputs = self.model( 2025-08-14T21:43:31.6661134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6661573Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6662000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6662435Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6662785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6663139Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6663580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6664059Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6664509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6665013Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6665223Z 2025-08-14T21:43:31.6665328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6665692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6666011Z return mod(**inputs) 2025-08-14T21:43:31.6666416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6666868Z outputs = self.model( 2025-08-14T21:43:31.6667281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6667707Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6668131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6668564Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6668908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6669288Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6669723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6670174Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6670622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6671055Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6671199Z 2025-08-14T21:43:31.6671306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6671679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6672011Z return mod(**inputs) 2025-08-14T21:43:31.6672446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6672906Z outputs = self.model( 2025-08-14T21:43:31.6673345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6673798Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6674252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6674708Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6675073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6675445Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6675994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6676489Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6676962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6677439Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6677597Z 2025-08-14T21:43:31.6677684Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6677911Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6678129Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6678351Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6678634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6679005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6679353Z return mod(**inputs) 2025-08-14T21:43:31.6679790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6680250Z outputs = self.model( 2025-08-14T21:43:31.6680686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6681174Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6681631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6682096Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6682461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6682841Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6683308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6683784Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6684301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6684763Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6685216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6685687Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6685882Z 2025-08-14T21:43:31.6685990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6686356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6686684Z return mod(**inputs) 2025-08-14T21:43:31.6687093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6687532Z outputs = self.model( 2025-08-14T21:43:31.6687952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6688387Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6688819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6689259Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6689615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6689978Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6690421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6690889Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6691348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6691806Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6692255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6692717Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6692879Z 2025-08-14T21:43:31.6692986Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6693366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6693687Z return mod(**inputs) 2025-08-14T21:43:31.6694103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6694529Z outputs = self.model( 2025-08-14T21:43:31.6694949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6695389Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6695839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6696272Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6696620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6696989Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6697407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6697859Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6698341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6698780Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6698912Z 2025-08-14T21:43:31.6699016Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6699377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6699704Z return mod(**inputs) 2025-08-14T21:43:31.6700117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6700553Z outputs = self.model( 2025-08-14T21:43:31.6700973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6701415Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6701838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6702274Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6702623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6702987Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6703448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6703967Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6704142Z 2025-08-14T21:43:31.6704256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6704614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6704936Z return mod(**inputs) 2025-08-14T21:43:31.6705353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6705793Z outputs = self.model( 2025-08-14T21:43:31.6706202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6706641Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6707069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6707515Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6707851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6708205Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6708811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6709304Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6709683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6710081Z return self.act(input) 2025-08-14T21:43:31.6710200Z 2025-08-14T21:43:31.6710315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6710685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6711025Z return mod(**inputs) 2025-08-14T21:43:31.6711474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6711941Z outputs = self.model( 2025-08-14T21:43:31.6712381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6712842Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6713350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6713807Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6714164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6714546Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6715008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:43:31.6715473Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6715627Z 2025-08-14T21:43:31.6715736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6716173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6716522Z return mod(**inputs) 2025-08-14T21:43:31.6716965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6717434Z outputs = self.model( 2025-08-14T21:43:31.6717885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6718334Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6718788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6719241Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6719609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6719982Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6720447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6720927Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6721401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6721933Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6722153Z 2025-08-14T21:43:31.6722263Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6722676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6723013Z return mod(**inputs) 2025-08-14T21:43:31.6723437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6723888Z outputs = self.model( 2025-08-14T21:43:31.6724326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6724776Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6725252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6725706Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6726073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6726447Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6726907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6727394Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6727873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6728369Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6728521Z 2025-08-14T21:43:31.6728631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6729014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6729349Z return mod(**inputs) 2025-08-14T21:43:31.6729763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6730200Z outputs = self.model( 2025-08-14T21:43:31.6730609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6731042Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6731473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6731911Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6732255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6732608Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6733042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6733495Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6733940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6734389Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6734539Z 2025-08-14T21:43:31.6734622Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6734835Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6735039Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6735248Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6735479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6735836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6736151Z return mod(**inputs) 2025-08-14T21:43:31.6736557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6737004Z outputs = self.model( 2025-08-14T21:43:31.6737407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6737841Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6738271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6738708Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6739046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6739417Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6739838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6740267Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6740708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6741154Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6741582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6742034Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6742341Z 2025-08-14T21:43:31.6742446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6742795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6743106Z return mod(**inputs) 2025-08-14T21:43:31.6743509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6743928Z outputs = self.model( 2025-08-14T21:43:31.6744332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6744747Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6745164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6745595Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6745949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6746292Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6746722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6747169Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6747613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6748064Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6748500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6748946Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6749104Z 2025-08-14T21:43:31.6749212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6749569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6749891Z return mod(**inputs) 2025-08-14T21:43:31.6750301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6750721Z outputs = self.model( 2025-08-14T21:43:31.6751137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6751592Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6752009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6752437Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6752785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6753143Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6753597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6754072Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6754547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6755015Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6755160Z 2025-08-14T21:43:31.6755269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6755660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6756097Z return mod(**inputs) 2025-08-14T21:43:31.6756618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6757092Z outputs = self.model( 2025-08-14T21:43:31.6757607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6758050Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6758472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6758908Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6759260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6759619Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6760051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6760538Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6760709Z 2025-08-14T21:43:31.6760821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6761179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6761497Z return mod(**inputs) 2025-08-14T21:43:31.6761911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6762346Z outputs = self.model( 2025-08-14T21:43:31.6762750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6763182Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6763607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6764038Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6764420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6764785Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6765219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6765715Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6766081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6766411Z return self.act(input) 2025-08-14T21:43:31.6766519Z 2025-08-14T21:43:31.6766627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6766969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6767285Z return mod(**inputs) 2025-08-14T21:43:31.6767683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6768121Z outputs = self.model( 2025-08-14T21:43:31.6768514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6768935Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6769354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6769767Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6770100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6770449Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6770900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:43:31.6771330Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6771478Z 2025-08-14T21:43:31.6771583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6771948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6772281Z return mod(**inputs) 2025-08-14T21:43:31.6772695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6773159Z outputs = self.model( 2025-08-14T21:43:31.6773609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6774082Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6774545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6774996Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6775342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6775701Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6776135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6776590Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6777037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6777535Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6777744Z 2025-08-14T21:43:31.6777852Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6778212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6778528Z return mod(**inputs) 2025-08-14T21:43:31.6778934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6779359Z outputs = self.model( 2025-08-14T21:43:31.6779772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6780211Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6780631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6781049Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6781387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6781729Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6782176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6782623Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6783039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6783457Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6783598Z 2025-08-14T21:43:31.6783700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6784052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6784361Z return mod(**inputs) 2025-08-14T21:43:31.6784802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6785234Z outputs = self.model( 2025-08-14T21:43:31.6785649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6786065Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6786479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6786900Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6787241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6787580Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6787999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6788434Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6788862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6789296Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6789441Z 2025-08-14T21:43:31.6789525Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6789734Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6789933Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6790135Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6790362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6790700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6791015Z return mod(**inputs) 2025-08-14T21:43:31.6791419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6791871Z outputs = self.model( 2025-08-14T21:43:31.6792306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6792783Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6793247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6793730Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6794110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6794502Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6794974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6795458Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6796028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6796563Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6797055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6797497Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6797681Z 2025-08-14T21:43:31.6797780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6800602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6800926Z return mod(**inputs) 2025-08-14T21:43:31.6801331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6801776Z outputs = self.model( 2025-08-14T21:43:31.6802190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6802640Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6803079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6803523Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6803881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6804242Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6804704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6805158Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6805627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6806099Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6806546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6807012Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6807181Z 2025-08-14T21:43:31.6807287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6807643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6807968Z return mod(**inputs) 2025-08-14T21:43:31.6808374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6808983Z outputs = self.model( 2025-08-14T21:43:31.6809393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6809820Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6810241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6810668Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6811063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6811406Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6811841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6812294Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6812740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6813183Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6813355Z 2025-08-14T21:43:31.6813458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6813814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6814132Z return mod(**inputs) 2025-08-14T21:43:31.6814547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6814963Z outputs = self.model( 2025-08-14T21:43:31.6815438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6815896Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6816383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6816853Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6817196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6817564Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6817989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6818457Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6818627Z 2025-08-14T21:43:31.6818732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6819089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6819409Z return mod(**inputs) 2025-08-14T21:43:31.6819824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6820248Z outputs = self.model( 2025-08-14T21:43:31.6820668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6821101Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6821541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6822002Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6822348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6822706Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6823136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6823613Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6823999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6824339Z return self.act(input) 2025-08-14T21:43:31.6824449Z 2025-08-14T21:43:31.6824552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6824908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6825261Z return mod(**inputs) 2025-08-14T21:43:31.6825665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6826098Z outputs = self.model( 2025-08-14T21:43:31.6826509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6826948Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6827370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6827826Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6828174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6828533Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6828965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:43:31.6829406Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6829544Z 2025-08-14T21:43:31.6829693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6830043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6830382Z return mod(**inputs) 2025-08-14T21:43:31.6830813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6831271Z outputs = self.model( 2025-08-14T21:43:31.6831703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6832170Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6832633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6833100Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6833474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6833866Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6834341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6834826Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6835311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6835935Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6836167Z 2025-08-14T21:43:31.6836287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6836672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6837025Z return mod(**inputs) 2025-08-14T21:43:31.6837484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6837964Z outputs = self.model( 2025-08-14T21:43:31.6838419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6838909Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6839388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6839865Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6840249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6840664Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6841151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6841647Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6842152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6842648Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6842802Z 2025-08-14T21:43:31.6842916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6843273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6843601Z return mod(**inputs) 2025-08-14T21:43:31.6844017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6844451Z outputs = self.model( 2025-08-14T21:43:31.6844891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6845324Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6845768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6846196Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6846544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6846900Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6847334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6847779Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6848228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6848316Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6848320Z 2025-08-14T21:43:31.6848412Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6848493Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6848579Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6848655Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6848760Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6848967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6849032Z return mod(**inputs) 2025-08-14T21:43:31.6849334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6849409Z outputs = self.model( 2025-08-14T21:43:31.6849711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6849791Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6850090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6850163Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6850388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6850472Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6850773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6850893Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6851187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6851293Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6851580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6851712Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6851716Z 2025-08-14T21:43:31.6851828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6852045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6852117Z return mod(**inputs) 2025-08-14T21:43:31.6852422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6852492Z outputs = self.model( 2025-08-14T21:43:31.6852817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6852895Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6853222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6853304Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6853521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6853606Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6853909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6853997Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6854303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6854399Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6854689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6854800Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6854804Z 2025-08-14T21:43:31.6854905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6855111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6855176Z return mod(**inputs) 2025-08-14T21:43:31.6855482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6855551Z outputs = self.model( 2025-08-14T21:43:31.6855853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6855933Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6856229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6856301Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6856526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6856608Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6856911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:43:31.6857001Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:43:31.6857317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6857405Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6857409Z 2025-08-14T21:43:31.6857523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6857715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6857779Z return mod(**inputs) 2025-08-14T21:43:31.6858068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6858157Z outputs = self.model( 2025-08-14T21:43:31.6858457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6858527Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6858834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6858904Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6859144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6859220Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6859523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6859648Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6859653Z 2025-08-14T21:43:31.6859750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6859945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6860008Z return mod(**inputs) 2025-08-14T21:43:31.6860298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6860372Z outputs = self.model( 2025-08-14T21:43:31.6860663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6860733Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6861033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6861102Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6861318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6861391Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6861678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:43:31.6861803Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6862011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6862088Z return self.act(input) 2025-08-14T21:43:31.6862092Z 2025-08-14T21:43:31.6862193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6862395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6862467Z return mod(**inputs) 2025-08-14T21:43:31.6862758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6862823Z outputs = self.model( 2025-08-14T21:43:31.6863117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:43:31.6863210Z encoder_outputs = self.encoder( 2025-08-14T21:43:31.6863509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:43:31.6863578Z layer_outputs = encoder_layer( 2025-08-14T21:43:31.6863795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6863881Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6864180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:43:31.6864289Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6864293Z 2025-08-14T21:43:31.6864395Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6864590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6864666Z return mod(**inputs) 2025-08-14T21:43:31.6864964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6865054Z outputs = self.model( 2025-08-14T21:43:31.6865381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6865479Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6865804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6865880Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6866108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6866198Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6866514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6866628Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6866951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6867103Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6867107Z 2025-08-14T21:43:31.6867214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6867410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6867481Z return mod(**inputs) 2025-08-14T21:43:31.6867778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6867845Z outputs = self.model( 2025-08-14T21:43:31.6868152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6868224Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6868524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6868603Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6868826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6868910Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6869196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6869294Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6869618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6869698Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6869701Z 2025-08-14T21:43:31.6869812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6870005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6870072Z return mod(**inputs) 2025-08-14T21:43:31.6870385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6870469Z outputs = self.model( 2025-08-14T21:43:31.6870760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6870838Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6871129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6871203Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6871439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6871521Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6871859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6871964Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6872286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6872375Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6872379Z 2025-08-14T21:43:31.6872461Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6872553Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6872633Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6872712Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6872827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6873031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6873107Z return mod(**inputs) 2025-08-14T21:43:31.6873429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6873501Z outputs = self.model( 2025-08-14T21:43:31.6873834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6873907Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6874205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6874285Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6874510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6874597Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6874913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6875016Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6875339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6875441Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6875747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6875992Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6875999Z 2025-08-14T21:43:31.6876108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6876323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6876392Z return mod(**inputs) 2025-08-14T21:43:31.6876710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6876790Z outputs = self.model( 2025-08-14T21:43:31.6877135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6877220Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6877539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6877616Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6877888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6877974Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6878323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6878428Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6878746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6878856Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6879163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6879287Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6879291Z 2025-08-14T21:43:31.6879399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6879608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6879685Z return mod(**inputs) 2025-08-14T21:43:31.6880006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6880078Z outputs = self.model( 2025-08-14T21:43:31.6880411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6880488Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6880815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6880892Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6881123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6881215Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6881537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6881647Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6881964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6882051Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6882055Z 2025-08-14T21:43:31.6882168Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6882374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6882466Z return mod(**inputs) 2025-08-14T21:43:31.6882795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6882865Z outputs = self.model( 2025-08-14T21:43:31.6883185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6883264Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6883578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6883678Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6883914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6884001Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6884329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6884447Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6884796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6884974Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6884978Z 2025-08-14T21:43:31.6885093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6885297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6885366Z return mod(**inputs) 2025-08-14T21:43:31.6885690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6885762Z outputs = self.model( 2025-08-14T21:43:31.6886083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6886167Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6886484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6886565Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6886795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6886878Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6887210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6887314Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6887613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6887692Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6887696Z 2025-08-14T21:43:31.6887796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6887996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6888064Z return mod(**inputs) 2025-08-14T21:43:31.6888364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6888440Z outputs = self.model( 2025-08-14T21:43:31.6888759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6888842Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6889175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6889249Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6889490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6889571Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6889898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6890028Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6890343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6890439Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6890443Z 2025-08-14T21:43:31.6890529Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6890611Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6890698Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6890777Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6890905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6891112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6891199Z return mod(**inputs) 2025-08-14T21:43:31.6891526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6891599Z outputs = self.model( 2025-08-14T21:43:31.6891915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6891997Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6892312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6892392Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6892630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6892711Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6893039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6893145Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6893451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6893548Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6893830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6893969Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6893973Z 2025-08-14T21:43:31.6894076Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6894271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6894344Z return mod(**inputs) 2025-08-14T21:43:31.6894646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6894721Z outputs = self.model( 2025-08-14T21:43:31.6895020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6895092Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6895398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6895492Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6895725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6895806Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6896118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6896230Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6896534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6896633Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6896906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6897013Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6897017Z 2025-08-14T21:43:31.6897123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6897330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6897399Z return mod(**inputs) 2025-08-14T21:43:31.6897717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6897783Z outputs = self.model( 2025-08-14T21:43:31.6898084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6898152Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6898445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6898525Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6898743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6898828Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6899135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6899247Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6899576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6899663Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6899667Z 2025-08-14T21:43:31.6899774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6899986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6900054Z return mod(**inputs) 2025-08-14T21:43:31.6900388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6900458Z outputs = self.model( 2025-08-14T21:43:31.6900785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6900869Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6901198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6901281Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6901519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6901630Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6901933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.6902056Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6902060Z 2025-08-14T21:43:31.6902160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6902361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6902427Z return mod(**inputs) 2025-08-14T21:43:31.6902733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6902829Z outputs = self.model( 2025-08-14T21:43:31.6903160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6903247Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6903583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6903684Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6903919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6904018Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6904339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.6904457Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6904667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6904743Z return self.act(input) 2025-08-14T21:43:31.6904748Z 2025-08-14T21:43:31.6904850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6905050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6905115Z return mod(**inputs) 2025-08-14T21:43:31.6905418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6905493Z outputs = self.model( 2025-08-14T21:43:31.6905796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6905875Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6906199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6906273Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6906508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6906591Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6906907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:31.6907000Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6907004Z 2025-08-14T21:43:31.6907110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6907321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6907391Z return mod(**inputs) 2025-08-14T21:43:31.6907716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6907793Z outputs = self.model( 2025-08-14T21:43:31.6908118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6908220Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6908549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6908755Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6909017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6909100Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6909479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6909594Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6909917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6910091Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6910095Z 2025-08-14T21:43:31.6910206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6910497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6910578Z return mod(**inputs) 2025-08-14T21:43:31.6910931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6911011Z outputs = self.model( 2025-08-14T21:43:31.6911336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6911412Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6911740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6911817Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6912061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6912143Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6912474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6912593Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6912924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6913014Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6913018Z 2025-08-14T21:43:31.6913134Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6913349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6913428Z return mod(**inputs) 2025-08-14T21:43:31.6913766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6913838Z outputs = self.model( 2025-08-14T21:43:31.6914183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6914262Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6914604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6914684Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6914927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6915046Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6915369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6915476Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6915863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6915968Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6915973Z 2025-08-14T21:43:31.6916067Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6916178Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6916261Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6916352Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6916463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6916673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6916758Z return mod(**inputs) 2025-08-14T21:43:31.6917106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6917189Z outputs = self.model( 2025-08-14T21:43:31.6917535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6917617Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6917961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6918039Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6918282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6918365Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6918692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6918803Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6919122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6919226Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6919535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6919675Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6919679Z 2025-08-14T21:43:31.6919791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6919998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6920079Z return mod(**inputs) 2025-08-14T21:43:31.6920389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6920457Z outputs = self.model( 2025-08-14T21:43:31.6920765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6920838Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6921139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6921220Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6921439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6921527Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6921843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6921938Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6922237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6922330Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6922606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6922736Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6922740Z 2025-08-14T21:43:31.6922837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6923034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6923098Z return mod(**inputs) 2025-08-14T21:43:31.6923394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6923466Z outputs = self.model( 2025-08-14T21:43:31.6923783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6923861Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6924188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6924265Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6924501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6924583Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6924897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6925007Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6925323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6925412Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6925416Z 2025-08-14T21:43:31.6925524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6925727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6925811Z return mod(**inputs) 2025-08-14T21:43:31.6926102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6926173Z outputs = self.model( 2025-08-14T21:43:31.6926467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6926539Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6926844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6926914Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6927130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6927213Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6927512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6927627Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6927921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6928099Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6928103Z 2025-08-14T21:43:31.6928214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6928407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6928479Z return mod(**inputs) 2025-08-14T21:43:31.6928782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6928869Z outputs = self.model( 2025-08-14T21:43:31.6929186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6929257Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6929574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6929647Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6929889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6929976Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6930290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6930400Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6930706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6930786Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6930790Z 2025-08-14T21:43:31.6930898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6931092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6931159Z return mod(**inputs) 2025-08-14T21:43:31.6931476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6931541Z outputs = self.model( 2025-08-14T21:43:31.6931841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6931911Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6932205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6932280Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6932492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6932569Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6932866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6932970Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6933273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6933361Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6933365Z 2025-08-14T21:43:31.6933449Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6933540Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6933620Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6933699Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6933814Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6934041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6934116Z return mod(**inputs) 2025-08-14T21:43:31.6934450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6934521Z outputs = self.model( 2025-08-14T21:43:31.6934845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6934921Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6935247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6935339Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6935571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6935662Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6935981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6936129Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6936458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6936567Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6936849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6936979Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6936982Z 2025-08-14T21:43:31.6937080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6937275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6937341Z return mod(**inputs) 2025-08-14T21:43:31.6937654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6937725Z outputs = self.model( 2025-08-14T21:43:31.6938041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6938128Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6938442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6938518Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6938754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6939000Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6939336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6939450Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6939765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6939876Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6940174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6940297Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6940301Z 2025-08-14T21:43:31.6940408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6940612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6940717Z return mod(**inputs) 2025-08-14T21:43:31.6941035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6941118Z outputs = self.model( 2025-08-14T21:43:31.6941442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6941519Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6941850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6941947Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6942189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6942281Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6942601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6942720Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6943054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6943142Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6943160Z 2025-08-14T21:43:31.6943279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6943484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6943562Z return mod(**inputs) 2025-08-14T21:43:31.6943878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6943947Z outputs = self.model( 2025-08-14T21:43:31.6944276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6944352Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6944678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6944760Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6944999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6945089Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6945404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.6945529Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6945533Z 2025-08-14T21:43:31.6945649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6945853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6945928Z return mod(**inputs) 2025-08-14T21:43:31.6946260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6946330Z outputs = self.model( 2025-08-14T21:43:31.6946663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6946741Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6947133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6947216Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6947454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6948116Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6948445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.6948573Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6948811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6948885Z return self.act(input) 2025-08-14T21:43:31.6948889Z 2025-08-14T21:43:31.6949027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6949245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6949316Z return mod(**inputs) 2025-08-14T21:43:31.6949649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6949721Z outputs = self.model( 2025-08-14T21:43:31.6950072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6950166Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6950498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6950580Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6950802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6950881Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6951191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:31.6951273Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6951278Z 2025-08-14T21:43:31.6951383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6951584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6951648Z return mod(**inputs) 2025-08-14T21:43:31.6951958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6952024Z outputs = self.model( 2025-08-14T21:43:31.6952329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6952409Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6952706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6952786Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6953000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6953078Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6953384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6953484Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6953789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6953951Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6953955Z 2025-08-14T21:43:31.6954062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6954277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6954372Z return mod(**inputs) 2025-08-14T21:43:31.6954704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6954775Z outputs = self.model( 2025-08-14T21:43:31.6955099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6955183Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6955510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6955605Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6955900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6955989Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6956313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6956418Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6956756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6956848Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6956879Z 2025-08-14T21:43:31.6956980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6957185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6957252Z return mod(**inputs) 2025-08-14T21:43:31.6957555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6957628Z outputs = self.model( 2025-08-14T21:43:31.6957930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6958002Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6958315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6958384Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6958608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6958685Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6958984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6959087Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6959384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6959476Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6959480Z 2025-08-14T21:43:31.6959560Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6959637Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6959718Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6959794Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6959896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6960098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6960164Z return mod(**inputs) 2025-08-14T21:43:31.6960472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6960539Z outputs = self.model( 2025-08-14T21:43:31.6960856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6960936Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6961237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6961306Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6961532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6961608Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6961934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6962030Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6962326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6962431Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6962733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6962872Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6962875Z 2025-08-14T21:43:31.6962993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6963194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6963272Z return mod(**inputs) 2025-08-14T21:43:31.6963592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6963664Z outputs = self.model( 2025-08-14T21:43:31.6963990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6964067Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6964393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6964468Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6964700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6964790Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6965109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6965213Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6965510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6965610Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6965906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6966014Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6966017Z 2025-08-14T21:43:31.6966125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6966321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6966386Z return mod(**inputs) 2025-08-14T21:43:31.6966695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6966763Z outputs = self.model( 2025-08-14T21:43:31.6967061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6967163Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6967465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6967546Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6967779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6967860Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6968184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6968306Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6968627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6968714Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6968717Z 2025-08-14T21:43:31.6968823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6969059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6969129Z return mod(**inputs) 2025-08-14T21:43:31.6969463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6969546Z outputs = self.model( 2025-08-14T21:43:31.6969867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6969953Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6970273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6970352Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6970593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6970678Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6971004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6971122Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6971442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6971611Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6971615Z 2025-08-14T21:43:31.6971725Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6971943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6972016Z return mod(**inputs) 2025-08-14T21:43:31.6972336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6972415Z outputs = self.model( 2025-08-14T21:43:31.6972736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6972816Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6973144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6973223Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6973463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6973547Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6973882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6974013Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6974308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6974393Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6974397Z 2025-08-14T21:43:31.6974497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6974708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6974779Z return mod(**inputs) 2025-08-14T21:43:31.6975082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6975149Z outputs = self.model( 2025-08-14T21:43:31.6975447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6975538Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6975835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6975921Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6976135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6976219Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6976515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6976624Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6976920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.6977002Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.6977005Z 2025-08-14T21:43:31.6977091Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6977169Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6977242Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6977326Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.6977424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6977624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6977689Z return mod(**inputs) 2025-08-14T21:43:31.6977994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6978068Z outputs = self.model( 2025-08-14T21:43:31.6978380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6978453Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6978772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6978845Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6979079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6979163Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6979494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6979613Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6979939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6980066Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6980348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.6980477Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.6980480Z 2025-08-14T21:43:31.6980590Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6980783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6980869Z return mod(**inputs) 2025-08-14T21:43:31.6981180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6981249Z outputs = self.model( 2025-08-14T21:43:31.6981557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6981654Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6981966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6982048Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6982279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6982365Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6982666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6982771Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6983078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.6983175Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.6983462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.6983572Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.6983576Z 2025-08-14T21:43:31.6983683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6983905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6983975Z return mod(**inputs) 2025-08-14T21:43:31.6984296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6984376Z outputs = self.model( 2025-08-14T21:43:31.6984694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6984785Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6985086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6985157Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6985382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6985458Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6985766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.6985871Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.6986166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.6986276Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.6986279Z 2025-08-14T21:43:31.6986379Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6986575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6986647Z return mod(**inputs) 2025-08-14T21:43:31.6986944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6987019Z outputs = self.model( 2025-08-14T21:43:31.6987337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6987412Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6987736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6987811Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6988058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6988152Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6988462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.6988589Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6988593Z 2025-08-14T21:43:31.6988694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6988884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6988955Z return mod(**inputs) 2025-08-14T21:43:31.6989259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6989336Z outputs = self.model( 2025-08-14T21:43:31.6989642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6989717Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6990044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6990117Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6990355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6990437Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6990758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.6990886Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.6991113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.6991186Z return self.act(input) 2025-08-14T21:43:31.6991199Z 2025-08-14T21:43:31.6991305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6991515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6991594Z return mod(**inputs) 2025-08-14T21:43:31.6991929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6992002Z outputs = self.model( 2025-08-14T21:43:31.6992342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6992419Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6992770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6992845Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6993087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6993176Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6993488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:31.6993572Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.6993617Z 2025-08-14T21:43:31.6993723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6993924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6993999Z return mod(**inputs) 2025-08-14T21:43:31.6994317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6994387Z outputs = self.model( 2025-08-14T21:43:31.6994726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6994803Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6995148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6995224Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6995456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6995545Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6995934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6996047Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6996375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.6996532Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.6996537Z 2025-08-14T21:43:31.6996652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6996854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6996926Z return mod(**inputs) 2025-08-14T21:43:31.6997249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.6997323Z outputs = self.model( 2025-08-14T21:43:31.6997648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.6997727Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.6998043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.6998127Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.6998356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.6998432Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.6998738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.6998838Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.6999142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.6999244Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.6999248Z 2025-08-14T21:43:31.6999348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.6999554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.6999618Z return mod(**inputs) 2025-08-14T21:43:31.6999938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7000009Z outputs = self.model( 2025-08-14T21:43:31.7000353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7000436Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7000752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7000832Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7001063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7001162Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7001486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7001605Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7001930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7002023Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7002027Z 2025-08-14T21:43:31.7002107Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7002191Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7002270Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7002345Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7002454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7002649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7002714Z return mod(**inputs) 2025-08-14T21:43:31.7003020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7003085Z outputs = self.model( 2025-08-14T21:43:31.7003391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7003464Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7003766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7003850Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7004075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7004166Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7004481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7004583Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7004903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7005007Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7005303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7005448Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7005470Z 2025-08-14T21:43:31.7005586Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7005784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7005848Z return mod(**inputs) 2025-08-14T21:43:31.7006149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7006224Z outputs = self.model( 2025-08-14T21:43:31.7006524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7006623Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7006946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7007022Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7007260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7007341Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7007674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7007798Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7008113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7008220Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7008516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7008819Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7008828Z 2025-08-14T21:43:31.7008954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7009160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7009237Z return mod(**inputs) 2025-08-14T21:43:31.7009562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7009634Z outputs = self.model( 2025-08-14T21:43:31.7009957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7010035Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7010368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7010442Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7010687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7010778Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7011094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7011196Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7011521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7011610Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7011614Z 2025-08-14T21:43:31.7011729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7011933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7012003Z return mod(**inputs) 2025-08-14T21:43:31.7012377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7012448Z outputs = self.model( 2025-08-14T21:43:31.7012780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7012855Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7013184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7013332Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7013573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7013656Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7013978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7014095Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7014446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.7014606Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.7014609Z 2025-08-14T21:43:31.7014740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7014955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7015027Z return mod(**inputs) 2025-08-14T21:43:31.7015354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7015424Z outputs = self.model( 2025-08-14T21:43:31.7015742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7015824Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7016151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7016224Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7016460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7016541Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7016866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7016978Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7017293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.7017387Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.7017391Z 2025-08-14T21:43:31.7017497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7017711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7017780Z return mod(**inputs) 2025-08-14T21:43:31.7018101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7018182Z outputs = self.model( 2025-08-14T21:43:31.7018501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7018582Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7018908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7019005Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7019255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7019331Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7019636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7019750Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7020064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7020182Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7020186Z 2025-08-14T21:43:31.7020269Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7020350Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7020443Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7020522Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7020627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7020855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7020925Z return mod(**inputs) 2025-08-14T21:43:31.7021269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7021342Z outputs = self.model( 2025-08-14T21:43:31.7021668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7021753Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7022084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7022166Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7022396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7022480Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7022804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7022919Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7023239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7023350Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7023654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7023812Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7023816Z 2025-08-14T21:43:31.7023917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7024115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7024188Z return mod(**inputs) 2025-08-14T21:43:31.7024492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7024567Z outputs = self.model( 2025-08-14T21:43:31.7024873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7024945Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7025251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7025370Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7025586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7025675Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7025989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7026111Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7026426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7026547Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7026854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7026970Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7026974Z 2025-08-14T21:43:31.7027091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7027317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7027388Z return mod(**inputs) 2025-08-14T21:43:31.7027730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7027802Z outputs = self.model( 2025-08-14T21:43:31.7028124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7028203Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7028517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7028599Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7028829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7028911Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7029245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7029357Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7029685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7029774Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7029778Z 2025-08-14T21:43:31.7029884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7030097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7030170Z return mod(**inputs) 2025-08-14T21:43:31.7030492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7030565Z outputs = self.model( 2025-08-14T21:43:31.7030882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7030969Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7031296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7031376Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7031626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7031708Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7032039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7032189Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7032194Z 2025-08-14T21:43:31.7032307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7032525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7032597Z return mod(**inputs) 2025-08-14T21:43:31.7032935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7033027Z outputs = self.model( 2025-08-14T21:43:31.7033355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7033441Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7033768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7033844Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7034108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7034193Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7034551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7034682Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7034918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.7035001Z return self.act(input) 2025-08-14T21:43:31.7035005Z 2025-08-14T21:43:31.7035115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7035331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7035402Z return mod(**inputs) 2025-08-14T21:43:31.7035744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7036041Z outputs = self.model( 2025-08-14T21:43:31.7036392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7036471Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7036814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7036892Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7037141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7037231Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7037562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:31.7037663Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.7037667Z 2025-08-14T21:43:31.7037779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7037998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7038069Z return mod(**inputs) 2025-08-14T21:43:31.7038406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7038489Z outputs = self.model( 2025-08-14T21:43:31.7038820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7038924Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7039268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7039344Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7039596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7039682Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7040021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7040162Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7040490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.7040659Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.7040665Z 2025-08-14T21:43:31.7040774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7041010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7041092Z return mod(**inputs) 2025-08-14T21:43:31.7041448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7041530Z outputs = self.model( 2025-08-14T21:43:31.7041856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7041935Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7042268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7042346Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7042578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7042667Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7042991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7043106Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7043433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.7043521Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.7043525Z 2025-08-14T21:43:31.7043641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7043853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7043930Z return mod(**inputs) 2025-08-14T21:43:31.7044242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7044310Z outputs = self.model( 2025-08-14T21:43:31.7044615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7044689Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7045000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7045082Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7045309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7045399Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7045713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7045836Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7046160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7046248Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7046252Z 2025-08-14T21:43:31.7046346Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7046431Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7046533Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7046622Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7046729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7046932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7047009Z return mod(**inputs) 2025-08-14T21:43:31.7047331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7047409Z outputs = self.model( 2025-08-14T21:43:31.7047749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7047830Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7048178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7048258Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7048488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7048580Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7048896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7048999Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7049304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7049401Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7049698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7049831Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7049836Z 2025-08-14T21:43:31.7049943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7050139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7050204Z return mod(**inputs) 2025-08-14T21:43:31.7050518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7050585Z outputs = self.model( 2025-08-14T21:43:31.7050891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7050972Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7051277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7051355Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7051576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7051654Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7051974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7052094Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7052403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7052498Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7052784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7052899Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7052921Z 2025-08-14T21:43:31.7053023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7053223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7053289Z return mod(**inputs) 2025-08-14T21:43:31.7053589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7053666Z outputs = self.model( 2025-08-14T21:43:31.7053981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7054055Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7054379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7054452Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7054678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7054757Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7055075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7055187Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7055504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7055599Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7055603Z 2025-08-14T21:43:31.7055710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7055920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7055995Z return mod(**inputs) 2025-08-14T21:43:31.7056318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7056388Z outputs = self.model( 2025-08-14T21:43:31.7056732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7056804Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7057113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7057182Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7057416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7057507Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7057823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7057943Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7058260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.7058440Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.7058444Z 2025-08-14T21:43:31.7058558Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7058763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7058831Z return mod(**inputs) 2025-08-14T21:43:31.7059156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7059226Z outputs = self.model( 2025-08-14T21:43:31.7059546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7059645Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7059962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7060041Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7060256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7060355Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7060653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7060777Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7061083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.7061164Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.7061167Z 2025-08-14T21:43:31.7061274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7061493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7061563Z return mod(**inputs) 2025-08-14T21:43:31.7061887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7061960Z outputs = self.model( 2025-08-14T21:43:31.7062278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7062361Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7062679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7062763Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7062995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7063077Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7063401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7063512Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7063832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7063917Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7063922Z 2025-08-14T21:43:31.7064001Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7064087Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7064171Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7064250Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7064366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7064571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7064670Z return mod(**inputs) 2025-08-14T21:43:31.7064986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7065058Z outputs = self.model( 2025-08-14T21:43:31.7065384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7065461Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7065775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7065877Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7066112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7066198Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7066514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7066628Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7066975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7067079Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7067400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7067544Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7067548Z 2025-08-14T21:43:31.7067653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7067867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7067938Z return mod(**inputs) 2025-08-14T21:43:31.7068256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7068334Z outputs = self.model( 2025-08-14T21:43:31.7068654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7068740Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7069060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7069136Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7069372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7069454Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7069779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7069891Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7070211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7070319Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7070621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7070741Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7070746Z 2025-08-14T21:43:31.7070853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7071060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7071135Z return mod(**inputs) 2025-08-14T21:43:31.7071475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7071545Z outputs = self.model( 2025-08-14T21:43:31.7071873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7071949Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7072274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7072368Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7072599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7072689Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7073003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7073121Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7073459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7073547Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7073550Z 2025-08-14T21:43:31.7073686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7073891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7073962Z return mod(**inputs) 2025-08-14T21:43:31.7074287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7074356Z outputs = self.model( 2025-08-14T21:43:31.7074676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7074755Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7075075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7075156Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7075385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7075473Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7075867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7076012Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7076017Z 2025-08-14T21:43:31.7076135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7076350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7076422Z return mod(**inputs) 2025-08-14T21:43:31.7076762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7076833Z outputs = self.model( 2025-08-14T21:43:31.7077170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7077249Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7077578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7077664Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7077896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7078014Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7078332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7078459Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7078691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.7078767Z return self.act(input) 2025-08-14T21:43:31.7078771Z 2025-08-14T21:43:31.7078879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7079111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7079181Z return mod(**inputs) 2025-08-14T21:43:31.7079519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7079594Z outputs = self.model( 2025-08-14T21:43:31.7079919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7080006Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7080360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7080459Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7080688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7080771Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7081089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:31.7081174Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.7081178Z 2025-08-14T21:43:31.7081287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7081496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7081564Z return mod(**inputs) 2025-08-14T21:43:31.7081894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7081966Z outputs = self.model( 2025-08-14T21:43:31.7082284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7082370Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7082682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7082763Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7082989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7083074Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7083394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7083498Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7083810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.7083976Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.7083981Z 2025-08-14T21:43:31.7084088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7084297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7084365Z return mod(**inputs) 2025-08-14T21:43:31.7084707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7084786Z outputs = self.model( 2025-08-14T21:43:31.7085102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7085184Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7085500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7085597Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7085830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7085910Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7086227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7086340Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7086670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.7086763Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.7086766Z 2025-08-14T21:43:31.7086889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7087094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7087170Z return mod(**inputs) 2025-08-14T21:43:31.7087497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7087575Z outputs = self.model( 2025-08-14T21:43:31.7087907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7087984Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7088311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7088386Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7088633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7088715Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7089031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7089143Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7089458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7089549Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7089560Z 2025-08-14T21:43:31.7089644Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7089726Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7089816Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7089895Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7090003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7090216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7090285Z return mod(**inputs) 2025-08-14T21:43:31.7090616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7090697Z outputs = self.model( 2025-08-14T21:43:31.7091028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7091134Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7091460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7091534Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7091773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7091856Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7092178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7092300Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7092614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7092725Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7093024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7093179Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7093191Z 2025-08-14T21:43:31.7093299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7093522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7093598Z return mod(**inputs) 2025-08-14T21:43:31.7093931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7094004Z outputs = self.model( 2025-08-14T21:43:31.7094344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7094420Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7094751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7094827Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7095056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7095145Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7095459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7095564Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7095881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7095982Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7096287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7096402Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7096405Z 2025-08-14T21:43:31.7096510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7096727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7096800Z return mod(**inputs) 2025-08-14T21:43:31.7097109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7097178Z outputs = self.model( 2025-08-14T21:43:31.7097482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7097593Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7097911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7097996Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7098227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7098308Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7099907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7100422Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7100815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7100930Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7100945Z 2025-08-14T21:43:31.7101073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7101364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7101480Z return mod(**inputs) 2025-08-14T21:43:31.7101827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7101915Z outputs = self.model( 2025-08-14T21:43:31.7102299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7102397Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7102742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7102827Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7103096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7103188Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7103530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7103661Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7104003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.7104192Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.7104199Z 2025-08-14T21:43:31.7104317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7104532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7104612Z return mod(**inputs) 2025-08-14T21:43:31.7104952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7105034Z outputs = self.model( 2025-08-14T21:43:31.7105367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7105448Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7105787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7105868Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7106141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7106266Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7106768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7106994Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7107391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.7107486Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.7107491Z 2025-08-14T21:43:31.7107616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7107835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7107938Z return mod(**inputs) 2025-08-14T21:43:31.7108276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7108352Z outputs = self.model( 2025-08-14T21:43:31.7108898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7109034Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7109483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7109566Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7109859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7109954Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7110290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7110406Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7110751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7110845Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7110850Z 2025-08-14T21:43:31.7110948Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7111037Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7111122Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7111212Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7111328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7111544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7111627Z return mod(**inputs) 2025-08-14T21:43:31.7111965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7112048Z outputs = self.model( 2025-08-14T21:43:31.7112385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7112467Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7112822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7112900Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7113157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7113244Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7113688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7113821Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7114167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7114320Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7114642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7114795Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7114799Z 2025-08-14T21:43:31.7114958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7115188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7115261Z return mod(**inputs) 2025-08-14T21:43:31.7115636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7115708Z outputs = self.model( 2025-08-14T21:43:31.7116119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7116231Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7116610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7116698Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7116943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7117046Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7117384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7117502Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7117836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7117951Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7118266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7118395Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7118400Z 2025-08-14T21:43:31.7118514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7118739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7118811Z return mod(**inputs) 2025-08-14T21:43:31.7119153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7119241Z outputs = self.model( 2025-08-14T21:43:31.7119581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7119668Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7120006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7120087Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7120337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7120424Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7120820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7120935Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7121231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7121322Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7121387Z 2025-08-14T21:43:31.7121489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7121684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7121761Z return mod(**inputs) 2025-08-14T21:43:31.7122065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7122142Z outputs = self.model( 2025-08-14T21:43:31.7122447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7122544Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7122858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7122928Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7123155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7123245Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7123573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7123725Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7123744Z 2025-08-14T21:43:31.7123845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7124040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7124116Z return mod(**inputs) 2025-08-14T21:43:31.7124465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7124540Z outputs = self.model( 2025-08-14T21:43:31.7124849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7124921Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7125248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7125325Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7125565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7125662Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7125992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7126121Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7126336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.7126408Z return self.act(input) 2025-08-14T21:43:31.7126412Z 2025-08-14T21:43:31.7126522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7126721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7126794Z return mod(**inputs) 2025-08-14T21:43:31.7127110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7127182Z outputs = self.model( 2025-08-14T21:43:31.7127503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7127579Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7127892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7128009Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7128225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7128313Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7128614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:31.7128696Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.7128700Z 2025-08-14T21:43:31.7128836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7129032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7129109Z return mod(**inputs) 2025-08-14T21:43:31.7129412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7129496Z outputs = self.model( 2025-08-14T21:43:31.7129820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7129898Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7130229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7130312Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7130530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7130621Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7130918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7131019Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7131327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.7131489Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.7131494Z 2025-08-14T21:43:31.7131616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7131820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7131887Z return mod(**inputs) 2025-08-14T21:43:31.7132195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7132266Z outputs = self.model( 2025-08-14T21:43:31.7132564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7132645Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7132943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7133021Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7133239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7133322Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7133630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7133733Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7134045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.7134128Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.7134161Z 2025-08-14T21:43:31.7134265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7134482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7134558Z return mod(**inputs) 2025-08-14T21:43:31.7134895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7134969Z outputs = self.model( 2025-08-14T21:43:31.7135296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7135411Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7135746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7135822Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7136070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7136156Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7136514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7136616Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7136947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7137041Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7137047Z 2025-08-14T21:43:31.7137126Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7137212Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7137287Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7137362Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7137473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7137669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7137738Z return mod(**inputs) 2025-08-14T21:43:31.7138047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7138116Z outputs = self.model( 2025-08-14T21:43:31.7138424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7138498Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7138805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7138886Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7139106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7139182Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7139510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7139614Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7139942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7140056Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7140340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7140481Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7140486Z 2025-08-14T21:43:31.7140618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7140818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7140886Z return mod(**inputs) 2025-08-14T21:43:31.7141186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7141261Z outputs = self.model( 2025-08-14T21:43:31.7141563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7141659Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7141980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7142054Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7142290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7142372Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7142697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7142807Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7143124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7143233Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7143520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7143630Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7143634Z 2025-08-14T21:43:31.7143742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7143938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7144011Z return mod(**inputs) 2025-08-14T21:43:31.7144317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7144387Z outputs = self.model( 2025-08-14T21:43:31.7144701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7144775Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7145080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7145159Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7145378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7145465Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7145766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7145863Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7146171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7146254Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7146259Z 2025-08-14T21:43:31.7146371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7146564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7146630Z return mod(**inputs) 2025-08-14T21:43:31.7146937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7147030Z outputs = self.model( 2025-08-14T21:43:31.7147334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7147414Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7147717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7147795Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7148019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7148116Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7148424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7148532Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7148839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.7149026Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.7149031Z 2025-08-14T21:43:31.7149128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7149340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7149405Z return mod(**inputs) 2025-08-14T21:43:31.7149704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7149778Z outputs = self.model( 2025-08-14T21:43:31.7150072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7150155Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7150458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7150528Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7150754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7150835Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7151147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7151254Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7151556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.7151647Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.7151650Z 2025-08-14T21:43:31.7151751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7151953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7152022Z return mod(**inputs) 2025-08-14T21:43:31.7152328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7152401Z outputs = self.model( 2025-08-14T21:43:31.7152704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7152778Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7153091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7153192Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7153416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7153499Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7153800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7153916Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7154215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7154330Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7154334Z 2025-08-14T21:43:31.7154414Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7154495Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7154577Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7154657Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7154759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7154961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7155043Z return mod(**inputs) 2025-08-14T21:43:31.7155350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7155438Z outputs = self.model( 2025-08-14T21:43:31.7155755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7156119Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7156486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7156565Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7156817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7156897Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7157225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7157341Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7157665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7157769Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7158051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7158192Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7158198Z 2025-08-14T21:43:31.7158300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7158497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7158572Z return mod(**inputs) 2025-08-14T21:43:31.7158873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7158942Z outputs = self.model( 2025-08-14T21:43:31.7159248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7159324Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7159631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7159702Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7159960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7160046Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7160349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7160468Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7160772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7160890Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7161180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7161288Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7161292Z 2025-08-14T21:43:31.7161404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7161601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7161666Z return mod(**inputs) 2025-08-14T21:43:31.7161999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7162072Z outputs = self.model( 2025-08-14T21:43:31.7162399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7162481Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7162785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7162874Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7163087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7163174Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7163496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7163602Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7163900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7163978Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7163983Z 2025-08-14T21:43:31.7164083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7164280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7164344Z return mod(**inputs) 2025-08-14T21:43:31.7164644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7164718Z outputs = self.model( 2025-08-14T21:43:31.7165021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7165098Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7165399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7165470Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7165697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7165772Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7166078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7166222Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7166226Z 2025-08-14T21:43:31.7166327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7166530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7166597Z return mod(**inputs) 2025-08-14T21:43:31.7166902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7166980Z outputs = self.model( 2025-08-14T21:43:31.7167297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7167375Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7167677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7167752Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7167990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7168090Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7168446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7168566Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7168777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.7168860Z return self.act(input) 2025-08-14T21:43:31.7168864Z 2025-08-14T21:43:31.7168972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7169180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7169258Z return mod(**inputs) 2025-08-14T21:43:31.7169583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7169664Z outputs = self.model( 2025-08-14T21:43:31.7169982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7170063Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7170393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7170465Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7170680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7170757Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7171054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:31.7171143Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.7171147Z 2025-08-14T21:43:31.7171248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7171437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7171512Z return mod(**inputs) 2025-08-14T21:43:31.7171804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7171879Z outputs = self.model( 2025-08-14T21:43:31.7172171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7172241Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7172568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7172636Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7172855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7172932Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7173225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7173330Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7173651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.7173801Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.7173811Z 2025-08-14T21:43:31.7173914Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7174106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7174176Z return mod(**inputs) 2025-08-14T21:43:31.7174502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7174573Z outputs = self.model( 2025-08-14T21:43:31.7174896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7174971Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7175277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7175348Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7175566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7175654Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7175961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7176067Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7176370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.7176450Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.7176455Z 2025-08-14T21:43:31.7176563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7176758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7176824Z return mod(**inputs) 2025-08-14T21:43:31.7177135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7177204Z outputs = self.model( 2025-08-14T21:43:31.7177600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7177720Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7178031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7178116Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7178336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7178422Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7178721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7178848Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7179161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7179249Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7179253Z 2025-08-14T21:43:31.7179341Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7179435Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7179516Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7179699Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7179830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7180047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7180115Z return mod(**inputs) 2025-08-14T21:43:31.7180429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7180498Z outputs = self.model( 2025-08-14T21:43:31.7180828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7180919Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7181253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7181332Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7181553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7181634Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7181951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7182055Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7182389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7182501Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7182807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7182956Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7182960Z 2025-08-14T21:43:31.7183071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7183272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7183348Z return mod(**inputs) 2025-08-14T21:43:31.7183677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7183761Z outputs = self.model( 2025-08-14T21:43:31.7184089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7184166Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7184501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7184577Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7184803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7184897Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7185213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7185346Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7185662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7185765Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7186071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7186189Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7186193Z 2025-08-14T21:43:31.7186308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7186535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7186605Z return mod(**inputs) 2025-08-14T21:43:31.7186932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7187008Z outputs = self.model( 2025-08-14T21:43:31.7187330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7187422Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7187743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7187843Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7188076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7188162Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7188489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:43:31.7188590Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:43:31.7188917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7189003Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7189009Z 2025-08-14T21:43:31.7189117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7189330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7189401Z return mod(**inputs) 2025-08-14T21:43:31.7189746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7189822Z outputs = self.model( 2025-08-14T21:43:31.7190142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7190226Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7190550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7190625Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7190864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7190945Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7191269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7191386Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7191704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:43:31.7191871Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:43:31.7191897Z 2025-08-14T21:43:31.7192005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7192215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7192288Z return mod(**inputs) 2025-08-14T21:43:31.7192609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7192689Z outputs = self.model( 2025-08-14T21:43:31.7193008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7193114Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7193442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7193518Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7193769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7193852Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7194204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7194329Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7194672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:43:31.7194772Z key_states = self.k_proj(current_states) 2025-08-14T21:43:31.7194776Z 2025-08-14T21:43:31.7194896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7195100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7195178Z return mod(**inputs) 2025-08-14T21:43:31.7195500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7195580Z outputs = self.model( 2025-08-14T21:43:31.7196006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7196096Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7196437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7196515Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7196751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7196849Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7197175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7197310Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7197624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:43:31.7197713Z value_states = self.v_proj(current_states) 2025-08-14T21:43:31.7197717Z 2025-08-14T21:43:31.7197808Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7197892Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7197981Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7198066Z cudagraph partition due to non gpu ops 2025-08-14T21:43:31.7198173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7198387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7198456Z return mod(**inputs) 2025-08-14T21:43:31.7198771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7198877Z outputs = self.model( 2025-08-14T21:43:31.7199197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7199282Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7199621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7199697Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7199971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7200055Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7200382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7200508Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7200853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7200968Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7201518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:43:31.7201666Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:31.7201673Z 2025-08-14T21:43:31.7201794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7202005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7202087Z return mod(**inputs) 2025-08-14T21:43:31.7202421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7202495Z outputs = self.model( 2025-08-14T21:43:31.7202838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7202919Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7203247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7203334Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7203572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7203663Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7203986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7204103Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7204436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:43:31.7204541Z attn_output, attn_weights = attention_interface( 2025-08-14T21:43:31.7204856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:43:31.7204972Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:43:31.7204976Z 2025-08-14T21:43:31.7205089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7205308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7205378Z return mod(**inputs) 2025-08-14T21:43:31.7205708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7205805Z outputs = self.model( 2025-08-14T21:43:31.7206136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7206224Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7206558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7206636Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7206885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7206990Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7207322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:43:31.7207438Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:43:31.7207765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:43:31.7207876Z attn_output = self.out_proj(attn_output) 2025-08-14T21:43:31.7207881Z 2025-08-14T21:43:31.7207990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7208224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7208297Z return mod(**inputs) 2025-08-14T21:43:31.7208814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7208958Z outputs = self.model( 2025-08-14T21:43:31.7209448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7209537Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7209875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7209955Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7210201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7210288Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7210615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7210755Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7210761Z 2025-08-14T21:43:31.7210873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7211092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7211167Z return mod(**inputs) 2025-08-14T21:43:31.7211494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7211589Z outputs = self.model( 2025-08-14T21:43:31.7211909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7211989Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7212312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7212388Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7212623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7212705Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7213017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:43:31.7213224Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:43:31.7213450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:31.7213533Z return self.act(input) 2025-08-14T21:43:31.7213537Z 2025-08-14T21:43:31.7213646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7213851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7213964Z return mod(**inputs) 2025-08-14T21:43:31.7214293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:43:31.7214365Z outputs = self.model( 2025-08-14T21:43:31.7214692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:43:31.7214771Z decoder_outputs = self.decoder( 2025-08-14T21:43:31.7215125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:43:31.7215205Z layer_outputs = decoder_layer( 2025-08-14T21:43:31.7215458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:31.7215549Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:31.7215865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:43:31.7215962Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:43:31.7215966Z 2025-08-14T21:43:31.7216073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7216282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7216356Z return mod(**inputs) 2025-08-14T21:43:31.7216679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1393, in forward 2025-08-14T21:43:31.7216808Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-08-14T21:43:31.7216818Z 2025-08-14T21:43:31.7216925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:31.7217132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:31.7217210Z return mod(**inputs) 2025-08-14T21:43:31.7217531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1398, in forward 2025-08-14T21:43:31.7217711Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:43:31.7217716Z 2025-08-14T21:43:41.1082579Z Compilation time (from dynamo_timed): 20.221350911 2025-08-14T21:43:41.1103046Z pass 2025-08-14T21:43:41.1106814Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:41.1107674Z TIMING: _recursive_pre_grad_passes:0.01048 _recursive_joint_graph_passes:0.58011 _recursive_post_grad_passes:0.12281 async_compile.wait:0.7435 code_gen:8.7472 inductor_compile:11.14212 backend_compile:16.36435 gc:0.00017 entire_frame_compile:20.22135 total_wall_time:20.22135 2025-08-14T21:43:41.1108947Z STATS: call_* op count: 652 | FakeTensorMode.__torch_dispatch__:22579 | FakeTensor.__torch_dispatch__:8019 | ProxyTorchDispatchMode.__torch_dispatch__:8304 2025-08-14T21:43:41.1109477Z Dynamo produced 1 graphs covering 652 ops with 0 graph breaks (0 unique) 2025-08-14T21:43:46.5580007Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:46.5581410Z from pkg_resources import resource_filename 2025-08-14T21:43:47.1714710Z 2025-08-14T21:43:48.6161992Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:43:48.6162277Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:43:48.6180832Z cpu eval CamemBert 2025-08-14T21:43:49.1715145Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:49.4161801Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:49.7032349Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:57.2661475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2665656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2666102Z return mod(**inputs) 2025-08-14T21:43:57.2666904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2667376Z outputs = self.roberta( 2025-08-14T21:43:57.2667774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-08-14T21:43:57.2668272Z embedding_output = self.embeddings( 2025-08-14T21:43:57.2668682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-08-14T21:43:57.2669215Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:43:57.2669829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1590, in create_position_ids_from_input_ids 2025-08-14T21:43:57.2670325Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:43:57.2670479Z 2025-08-14T21:43:57.2670574Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2670805Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2671043Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2671268Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2671485Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2673163Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2675028Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2675358Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2676137Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2676443Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2680770Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2685945Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2691007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2693860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2699915Z return mod(**inputs) 2025-08-14T21:43:57.2700483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2700957Z outputs = self.roberta( 2025-08-14T21:43:57.2701409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-08-14T21:43:57.2701875Z embedding_output = self.embeddings( 2025-08-14T21:43:57.2702338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-08-14T21:43:57.2702927Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:43:57.2703587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1591, in create_position_ids_from_input_ids 2025-08-14T21:43:57.2704404Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:43:57.2704674Z 2025-08-14T21:43:57.2704804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2705262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2705632Z return mod(**inputs) 2025-08-14T21:43:57.2706062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2706571Z outputs = self.roberta( 2025-08-14T21:43:57.2706988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-08-14T21:43:57.2707443Z embedding_output = self.embeddings( 2025-08-14T21:43:57.2707879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-08-14T21:43:57.2708465Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:43:57.2710106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1591, in create_position_ids_from_input_ids 2025-08-14T21:43:57.2710871Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:43:57.2711150Z 2025-08-14T21:43:57.2711276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2711687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2712047Z return mod(**inputs) 2025-08-14T21:43:57.2712486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2712948Z outputs = self.roberta( 2025-08-14T21:43:57.2713426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2713888Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2714670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2715126Z layer_outputs = layer_module( 2025-08-14T21:43:57.2715517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2716134Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2716600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2717066Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2717492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2717920Z return func(*args, **kwargs) 2025-08-14T21:43:57.2718343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2718778Z self_outputs = self.self( 2025-08-14T21:43:57.2719168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2719567Z return func(*args, **kwargs) 2025-08-14T21:43:57.2719980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.2720568Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.2720876Z 2025-08-14T21:43:57.2720992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2721462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2721857Z return mod(**inputs) 2025-08-14T21:43:57.2722298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2722744Z outputs = self.roberta( 2025-08-14T21:43:57.2723158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2723559Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2723957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2724416Z layer_outputs = layer_module( 2025-08-14T21:43:57.2724765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2725122Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2725576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2726925Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2727323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2727689Z return func(*args, **kwargs) 2025-08-14T21:43:57.2728107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2728518Z self_outputs = self.self( 2025-08-14T21:43:57.2728871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2729299Z return func(*args, **kwargs) 2025-08-14T21:43:57.2729689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.2730093Z self.key(current_states) 2025-08-14T21:43:57.2730208Z 2025-08-14T21:43:57.2730315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2730680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2731008Z return mod(**inputs) 2025-08-14T21:43:57.2731386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2731791Z outputs = self.roberta( 2025-08-14T21:43:57.2732179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2732583Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2732977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2733377Z layer_outputs = layer_module( 2025-08-14T21:43:57.2733722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2734087Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2734488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2734898Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2735275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2735643Z return func(*args, **kwargs) 2025-08-14T21:43:57.2736032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2736431Z self_outputs = self.self( 2025-08-14T21:43:57.2736792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2737167Z return func(*args, **kwargs) 2025-08-14T21:43:57.2737547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.2737938Z self.value(current_states) 2025-08-14T21:43:57.2738051Z 2025-08-14T21:43:57.2738137Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2738367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2738715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2739063Z return mod(**inputs) 2025-08-14T21:43:57.2739439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2739845Z outputs = self.roberta( 2025-08-14T21:43:57.2740236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2740644Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2741058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2741463Z layer_outputs = layer_module( 2025-08-14T21:43:57.2741823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2742182Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2742578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2742986Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2743356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2743708Z return func(*args, **kwargs) 2025-08-14T21:43:57.2744089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2744479Z self_outputs = self.self( 2025-08-14T21:43:57.2744821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2745182Z return func(*args, **kwargs) 2025-08-14T21:43:57.2745560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.2746010Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.2746191Z 2025-08-14T21:43:57.2746295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2746646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2746974Z return mod(**inputs) 2025-08-14T21:43:57.2747382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2747801Z outputs = self.roberta( 2025-08-14T21:43:57.2748213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2748637Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2749039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2749426Z layer_outputs = layer_module( 2025-08-14T21:43:57.2749762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2750112Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2750496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2750933Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2751313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2751677Z return func(*args, **kwargs) 2025-08-14T21:43:57.2752064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.2752523Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.2752983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.2753422Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2753577Z 2025-08-14T21:43:57.2753689Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2754067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2754418Z return mod(**inputs) 2025-08-14T21:43:57.2754820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2755273Z outputs = self.roberta( 2025-08-14T21:43:57.2755687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2756229Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2756695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2757138Z layer_outputs = layer_module( 2025-08-14T21:43:57.2757525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2757881Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2758285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2758702Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2759110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2759504Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2759939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.2760420Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.2760859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.2761274Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2761420Z 2025-08-14T21:43:57.2761524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2761880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2762197Z return mod(**inputs) 2025-08-14T21:43:57.2762578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2762980Z outputs = self.roberta( 2025-08-14T21:43:57.2763366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2763781Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2764203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2764626Z layer_outputs = layer_module( 2025-08-14T21:43:57.2764984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2765363Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2765842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2766286Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2766713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2767111Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2767551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.2768053Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.2768488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.2768928Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.2769313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.2769638Z return self.act(input) 2025-08-14T21:43:57.2769758Z 2025-08-14T21:43:57.2769929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2770307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2770623Z return mod(**inputs) 2025-08-14T21:43:57.2771004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2771398Z outputs = self.roberta( 2025-08-14T21:43:57.2771771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2772160Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2772565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2772971Z layer_outputs = layer_module( 2025-08-14T21:43:57.2773319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2773677Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2774083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2774501Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2774902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2775291Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2775724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.2776215Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.2776668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.2777087Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2777241Z 2025-08-14T21:43:57.2777344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2777695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2778003Z return mod(**inputs) 2025-08-14T21:43:57.2778376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2778770Z outputs = self.roberta( 2025-08-14T21:43:57.2779154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2779546Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2779998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2780405Z layer_outputs = layer_module( 2025-08-14T21:43:57.2780748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2781112Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2781521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2781939Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2782359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2782754Z return func(*args, **kwargs) 2025-08-14T21:43:57.2783170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2783592Z self_outputs = self.self( 2025-08-14T21:43:57.2783978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2784394Z return func(*args, **kwargs) 2025-08-14T21:43:57.2784811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.2785391Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.2785677Z 2025-08-14T21:43:57.2785790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2786173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2786512Z return mod(**inputs) 2025-08-14T21:43:57.2786908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2787335Z outputs = self.roberta( 2025-08-14T21:43:57.2787744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2788164Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2788583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2789008Z layer_outputs = layer_module( 2025-08-14T21:43:57.2789376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2789754Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2790272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2790762Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2791179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2791572Z return func(*args, **kwargs) 2025-08-14T21:43:57.2792001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2792431Z self_outputs = self.self( 2025-08-14T21:43:57.2792862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2793280Z return func(*args, **kwargs) 2025-08-14T21:43:57.2793724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.2794177Z self.key(current_states) 2025-08-14T21:43:57.2794305Z 2025-08-14T21:43:57.2794420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2794819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2795203Z return mod(**inputs) 2025-08-14T21:43:57.2795615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2796132Z outputs = self.roberta( 2025-08-14T21:43:57.2796570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2797019Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2811192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2811810Z layer_outputs = layer_module( 2025-08-14T21:43:57.2812176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2812537Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2812962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2813374Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2813801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2814171Z return func(*args, **kwargs) 2025-08-14T21:43:57.2814602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2815010Z self_outputs = self.self( 2025-08-14T21:43:57.2815373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2815754Z return func(*args, **kwargs) 2025-08-14T21:43:57.2816160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.2816569Z self.value(current_states) 2025-08-14T21:43:57.2816691Z 2025-08-14T21:43:57.2816780Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2817034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2817487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2817812Z return mod(**inputs) 2025-08-14T21:43:57.2818195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2818603Z outputs = self.roberta( 2025-08-14T21:43:57.2819001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2819399Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2819801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2820203Z layer_outputs = layer_module( 2025-08-14T21:43:57.2820570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2820960Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2821372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2821790Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2822167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2822538Z return func(*args, **kwargs) 2025-08-14T21:43:57.2823019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2823429Z self_outputs = self.self( 2025-08-14T21:43:57.2823784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2824203Z return func(*args, **kwargs) 2025-08-14T21:43:57.2824596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.2825058Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.2825246Z 2025-08-14T21:43:57.2825355Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2825721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2826076Z return mod(**inputs) 2025-08-14T21:43:57.2826457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2826865Z outputs = self.roberta( 2025-08-14T21:43:57.2827257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2827664Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2828079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2828485Z layer_outputs = layer_module( 2025-08-14T21:43:57.2828854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2829207Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2829615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2830030Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2830410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2830777Z return func(*args, **kwargs) 2025-08-14T21:43:57.2831185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.2831676Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.2832163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.2832600Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2832756Z 2025-08-14T21:43:57.2832869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2833253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2833595Z return mod(**inputs) 2025-08-14T21:43:57.2834006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2834436Z outputs = self.roberta( 2025-08-14T21:43:57.2834849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2835274Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2835725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2836257Z layer_outputs = layer_module( 2025-08-14T21:43:57.2836650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2837056Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2837510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2837976Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2838418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2838869Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2839332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.2839844Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.2840318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.2840759Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2840906Z 2025-08-14T21:43:57.2841047Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2841429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2841768Z return mod(**inputs) 2025-08-14T21:43:57.2842176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2842607Z outputs = self.roberta( 2025-08-14T21:43:57.2843009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2843460Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2843881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2844326Z layer_outputs = layer_module( 2025-08-14T21:43:57.2844686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2845068Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2845479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2845889Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2846295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2846696Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2847130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.2847602Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.2848056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.2848500Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.2848883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.2849217Z return self.act(input) 2025-08-14T21:43:57.2849338Z 2025-08-14T21:43:57.2849445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2849812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2850128Z return mod(**inputs) 2025-08-14T21:43:57.2850514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2850920Z outputs = self.roberta( 2025-08-14T21:43:57.2851308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2851705Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2852108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2852511Z layer_outputs = layer_module( 2025-08-14T21:43:57.2852858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2853232Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2853643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2854059Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2854452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2854846Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2855345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.2855869Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.2856326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.2856744Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2856883Z 2025-08-14T21:43:57.2856998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2857355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2857750Z return mod(**inputs) 2025-08-14T21:43:57.2858122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2858532Z outputs = self.roberta( 2025-08-14T21:43:57.2858911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2859311Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2859700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2860093Z layer_outputs = layer_module( 2025-08-14T21:43:57.2860436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2860794Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2861206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2861621Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2862011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2862378Z return func(*args, **kwargs) 2025-08-14T21:43:57.2862770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2863177Z self_outputs = self.self( 2025-08-14T21:43:57.2863533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2863905Z return func(*args, **kwargs) 2025-08-14T21:43:57.2864318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.2864889Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.2865151Z 2025-08-14T21:43:57.2865255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2865616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2865939Z return mod(**inputs) 2025-08-14T21:43:57.2866323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2866729Z outputs = self.roberta( 2025-08-14T21:43:57.2867136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2867613Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2868031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2868457Z layer_outputs = layer_module( 2025-08-14T21:43:57.2868822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2869201Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2869626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2870075Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2870472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2870864Z return func(*args, **kwargs) 2025-08-14T21:43:57.2871267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2871688Z self_outputs = self.self( 2025-08-14T21:43:57.2872085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2872465Z return func(*args, **kwargs) 2025-08-14T21:43:57.2872907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.2873332Z self.key(current_states) 2025-08-14T21:43:57.2873453Z 2025-08-14T21:43:57.2873569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2873942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2874287Z return mod(**inputs) 2025-08-14T21:43:57.2874694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2875114Z outputs = self.roberta( 2025-08-14T21:43:57.2875524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2876057Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2876498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2876935Z layer_outputs = layer_module( 2025-08-14T21:43:57.2877317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2877704Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2878134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2878565Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2878971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2879368Z return func(*args, **kwargs) 2025-08-14T21:43:57.2879780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2880204Z self_outputs = self.self( 2025-08-14T21:43:57.2880583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2880971Z return func(*args, **kwargs) 2025-08-14T21:43:57.2881373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.2881803Z self.value(current_states) 2025-08-14T21:43:57.2881932Z 2025-08-14T21:43:57.2882026Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2882272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2882672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2883014Z return mod(**inputs) 2025-08-14T21:43:57.2883421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2883836Z outputs = self.roberta( 2025-08-14T21:43:57.2884243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2884669Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2885081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2885529Z layer_outputs = layer_module( 2025-08-14T21:43:57.2885872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2886237Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2886662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2887102Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2887523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2887918Z return func(*args, **kwargs) 2025-08-14T21:43:57.2888343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2888747Z self_outputs = self.self( 2025-08-14T21:43:57.2889106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2889470Z return func(*args, **kwargs) 2025-08-14T21:43:57.2889862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.2890331Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.2890517Z 2025-08-14T21:43:57.2890627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2890977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2891298Z return mod(**inputs) 2025-08-14T21:43:57.2891686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2892090Z outputs = self.roberta( 2025-08-14T21:43:57.2892468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2892873Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2893275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2893675Z layer_outputs = layer_module( 2025-08-14T21:43:57.2894025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2894391Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2894801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2895209Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2895590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2895968Z return func(*args, **kwargs) 2025-08-14T21:43:57.2896354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.2896819Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.2897288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.2897702Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2897839Z 2025-08-14T21:43:57.2897945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2898314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2898640Z return mod(**inputs) 2025-08-14T21:43:57.2899025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2899438Z outputs = self.roberta( 2025-08-14T21:43:57.2899818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2900221Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2900612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2901015Z layer_outputs = layer_module( 2025-08-14T21:43:57.2901383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2901742Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2902163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2902584Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2902988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2903385Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2903812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.2904298Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.2904757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.2905179Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2905328Z 2025-08-14T21:43:57.2905434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2905799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2906133Z return mod(**inputs) 2025-08-14T21:43:57.2906522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2906944Z outputs = self.roberta( 2025-08-14T21:43:57.2907340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2907763Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2908158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2908560Z layer_outputs = layer_module( 2025-08-14T21:43:57.2909060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2909417Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2909828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2910244Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2910646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2911033Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2911466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.2911989Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.2912432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.2912874Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.2913255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.2913614Z return self.act(input) 2025-08-14T21:43:57.2913763Z 2025-08-14T21:43:57.2913878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2914267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2914620Z return mod(**inputs) 2025-08-14T21:43:57.2915037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2915467Z outputs = self.roberta( 2025-08-14T21:43:57.2915969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2916412Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2916862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2917301Z layer_outputs = layer_module( 2025-08-14T21:43:57.2917652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2918018Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2918423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2918855Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2919257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2919658Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2920099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.2920592Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.2921045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.2921467Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2921608Z 2025-08-14T21:43:57.2921720Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2922072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2922405Z return mod(**inputs) 2025-08-14T21:43:57.2922793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2923197Z outputs = self.roberta( 2025-08-14T21:43:57.2923577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2923981Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2924386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2924792Z layer_outputs = layer_module( 2025-08-14T21:43:57.2925132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2925485Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2925882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2926305Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2926680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2927042Z return func(*args, **kwargs) 2025-08-14T21:43:57.2927429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2927813Z self_outputs = self.self( 2025-08-14T21:43:57.2928168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2928575Z return func(*args, **kwargs) 2025-08-14T21:43:57.2928984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.2929526Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.2929794Z 2025-08-14T21:43:57.2929900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2930287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2930601Z return mod(**inputs) 2025-08-14T21:43:57.2931001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2931391Z outputs = self.roberta( 2025-08-14T21:43:57.2931762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2932148Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2932544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2932941Z layer_outputs = layer_module( 2025-08-14T21:43:57.2933275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2933639Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2934041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2934447Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2934824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2935187Z return func(*args, **kwargs) 2025-08-14T21:43:57.2935567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2935947Z self_outputs = self.self( 2025-08-14T21:43:57.2936297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2936658Z return func(*args, **kwargs) 2025-08-14T21:43:57.2937038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.2937421Z self.key(current_states) 2025-08-14T21:43:57.2937544Z 2025-08-14T21:43:57.2937648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2938009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2938334Z return mod(**inputs) 2025-08-14T21:43:57.2938708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2939108Z outputs = self.roberta( 2025-08-14T21:43:57.2939495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2939912Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2940364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2940789Z layer_outputs = layer_module( 2025-08-14T21:43:57.2941155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2941530Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2941964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2942403Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2942816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2943206Z return func(*args, **kwargs) 2025-08-14T21:43:57.2943618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2944041Z self_outputs = self.self( 2025-08-14T21:43:57.2944411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2944821Z return func(*args, **kwargs) 2025-08-14T21:43:57.2945250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.2945696Z self.value(current_states) 2025-08-14T21:43:57.2945821Z 2025-08-14T21:43:57.2945908Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.2946162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2946536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2946867Z return mod(**inputs) 2025-08-14T21:43:57.2947272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2947692Z outputs = self.roberta( 2025-08-14T21:43:57.2948100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2948521Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2948938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2949370Z layer_outputs = layer_module( 2025-08-14T21:43:57.2949733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2950110Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2950542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2950979Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2951378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2951763Z return func(*args, **kwargs) 2025-08-14T21:43:57.2952175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2952598Z self_outputs = self.self( 2025-08-14T21:43:57.2952977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2953375Z return func(*args, **kwargs) 2025-08-14T21:43:57.2953808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.2954320Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.2954525Z 2025-08-14T21:43:57.2954643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2955129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2955528Z return mod(**inputs) 2025-08-14T21:43:57.2956039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2956485Z outputs = self.roberta( 2025-08-14T21:43:57.2956919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2957350Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2957770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2958222Z layer_outputs = layer_module( 2025-08-14T21:43:57.2958573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2958940Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2959328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2959729Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2960121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2960498Z return func(*args, **kwargs) 2025-08-14T21:43:57.2960909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.2961377Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.2961846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.2962270Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2962413Z 2025-08-14T21:43:57.2962531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2962885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2963207Z return mod(**inputs) 2025-08-14T21:43:57.2963583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2963999Z outputs = self.roberta( 2025-08-14T21:43:57.2964394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2964809Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2965213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2965627Z layer_outputs = layer_module( 2025-08-14T21:43:57.2965979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2966343Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2966765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2967217Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2967650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2968049Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2968494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.2968990Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.2969455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.2969883Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2970031Z 2025-08-14T21:43:57.2970133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2970492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2970811Z return mod(**inputs) 2025-08-14T21:43:57.2971202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2971627Z outputs = self.roberta( 2025-08-14T21:43:57.2972012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2972424Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2972828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2973230Z layer_outputs = layer_module( 2025-08-14T21:43:57.2973573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2973928Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2974367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2974790Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2975207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2975599Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2976054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.2976564Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.2977036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.2977508Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.2977920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.2978283Z return self.act(input) 2025-08-14T21:43:57.2978404Z 2025-08-14T21:43:57.2978516Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2978897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2979241Z return mod(**inputs) 2025-08-14T21:43:57.2979643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2980069Z outputs = self.roberta( 2025-08-14T21:43:57.2980489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2980939Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2981375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2981810Z layer_outputs = layer_module( 2025-08-14T21:43:57.2982178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2982568Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2982993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.2983434Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.2983858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.2984267Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.2984750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.2985266Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.2985756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.2986190Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.2986342Z 2025-08-14T21:43:57.2986453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2986835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2987192Z return mod(**inputs) 2025-08-14T21:43:57.2987604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2988037Z outputs = self.roberta( 2025-08-14T21:43:57.2988451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2988872Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2989312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2989736Z layer_outputs = layer_module( 2025-08-14T21:43:57.2990126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2990516Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2990961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.2991420Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.2991840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2992233Z return func(*args, **kwargs) 2025-08-14T21:43:57.2992649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.2993077Z self_outputs = self.self( 2025-08-14T21:43:57.2993464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.2993871Z return func(*args, **kwargs) 2025-08-14T21:43:57.2994305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.2994895Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.2995182Z 2025-08-14T21:43:57.2995296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.2995688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.2996138Z return mod(**inputs) 2025-08-14T21:43:57.2996562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.2997019Z outputs = self.roberta( 2025-08-14T21:43:57.2997427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.2997848Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.2998241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.2998647Z layer_outputs = layer_module( 2025-08-14T21:43:57.2999014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.2999391Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.2999829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3000299Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3000706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3001097Z return func(*args, **kwargs) 2025-08-14T21:43:57.3001516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3001940Z self_outputs = self.self( 2025-08-14T21:43:57.3002325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3002730Z return func(*args, **kwargs) 2025-08-14T21:43:57.3003142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.3003565Z self.key(current_states) 2025-08-14T21:43:57.3003689Z 2025-08-14T21:43:57.3003800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3004187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3004555Z return mod(**inputs) 2025-08-14T21:43:57.3004963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3005400Z outputs = self.roberta( 2025-08-14T21:43:57.3005813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3006246Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3006663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3007085Z layer_outputs = layer_module( 2025-08-14T21:43:57.3007446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3007828Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3008251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3008864Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3009288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3009685Z return func(*args, **kwargs) 2025-08-14T21:43:57.3010093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3010522Z self_outputs = self.self( 2025-08-14T21:43:57.3010903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3011287Z return func(*args, **kwargs) 2025-08-14T21:43:57.3011701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.3012126Z self.value(current_states) 2025-08-14T21:43:57.3012254Z 2025-08-14T21:43:57.3012347Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.3012596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3012984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3013305Z return mod(**inputs) 2025-08-14T21:43:57.3013680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3014081Z outputs = self.roberta( 2025-08-14T21:43:57.3014455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3014905Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3015293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3015692Z layer_outputs = layer_module( 2025-08-14T21:43:57.3016038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3016395Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3016791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3017227Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3017597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3017954Z return func(*args, **kwargs) 2025-08-14T21:43:57.3018338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3018733Z self_outputs = self.self( 2025-08-14T21:43:57.3019136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3019495Z return func(*args, **kwargs) 2025-08-14T21:43:57.3019902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.3020355Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.3020533Z 2025-08-14T21:43:57.3020637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3020989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3021306Z return mod(**inputs) 2025-08-14T21:43:57.3021678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3022068Z outputs = self.roberta( 2025-08-14T21:43:57.3022445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3022841Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3023234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3023622Z layer_outputs = layer_module( 2025-08-14T21:43:57.3023959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3024317Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3024709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3025116Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3025490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3025851Z return func(*args, **kwargs) 2025-08-14T21:43:57.3026226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.3026680Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.3027128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.3027537Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3027672Z 2025-08-14T21:43:57.3027773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3028122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3028436Z return mod(**inputs) 2025-08-14T21:43:57.3028802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3029219Z outputs = self.roberta( 2025-08-14T21:43:57.3029595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3029986Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3030424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3030812Z layer_outputs = layer_module( 2025-08-14T21:43:57.3031196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3031550Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3031955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3032374Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3032778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3033182Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3033620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3034122Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3034605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.3035040Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3035196Z 2025-08-14T21:43:57.3035305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3035684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3036095Z return mod(**inputs) 2025-08-14T21:43:57.3036505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3036958Z outputs = self.roberta( 2025-08-14T21:43:57.3037393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3037786Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3038192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3038578Z layer_outputs = layer_module( 2025-08-14T21:43:57.3038907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3039244Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3039637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3040043Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3040436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3040818Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3041240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3041702Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3042132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.3042571Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.3042959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.3043321Z return self.act(input) 2025-08-14T21:43:57.3043432Z 2025-08-14T21:43:57.3043536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3043899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3044230Z return mod(**inputs) 2025-08-14T21:43:57.3044587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3044968Z outputs = self.roberta( 2025-08-14T21:43:57.3045336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3045749Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3046133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3046518Z layer_outputs = layer_module( 2025-08-14T21:43:57.3046846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3047175Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3047578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3047972Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3048372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3048741Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3049152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.3049626Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.3050075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.3050476Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3050614Z 2025-08-14T21:43:57.3050715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3051055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3051358Z return mod(**inputs) 2025-08-14T21:43:57.3051729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3052119Z outputs = self.roberta( 2025-08-14T21:43:57.3052487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3052872Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3053261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3053745Z layer_outputs = layer_module( 2025-08-14T21:43:57.3054090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3054439Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3054840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3055247Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3055615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3055986Z return func(*args, **kwargs) 2025-08-14T21:43:57.3056371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3056762Z self_outputs = self.self( 2025-08-14T21:43:57.3057138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3057497Z return func(*args, **kwargs) 2025-08-14T21:43:57.3057879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.3058403Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.3058665Z 2025-08-14T21:43:57.3058770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3059141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3059453Z return mod(**inputs) 2025-08-14T21:43:57.3059818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3060208Z outputs = self.roberta( 2025-08-14T21:43:57.3060584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3060974Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3061371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3061763Z layer_outputs = layer_module( 2025-08-14T21:43:57.3062117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3062460Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3062867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3063266Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3063639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3063995Z return func(*args, **kwargs) 2025-08-14T21:43:57.3064373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3064766Z self_outputs = self.self( 2025-08-14T21:43:57.3065119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3065480Z return func(*args, **kwargs) 2025-08-14T21:43:57.3065901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.3066293Z self.key(current_states) 2025-08-14T21:43:57.3066404Z 2025-08-14T21:43:57.3066506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3066856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3067171Z return mod(**inputs) 2025-08-14T21:43:57.3067545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3067929Z outputs = self.roberta( 2025-08-14T21:43:57.3068306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3068701Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3069084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3069478Z layer_outputs = layer_module( 2025-08-14T21:43:57.3069813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3070179Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3070579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3071018Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3071402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3071772Z return func(*args, **kwargs) 2025-08-14T21:43:57.3072157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3072561Z self_outputs = self.self( 2025-08-14T21:43:57.3072925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3073308Z return func(*args, **kwargs) 2025-08-14T21:43:57.3073701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.3074104Z self.value(current_states) 2025-08-14T21:43:57.3074231Z 2025-08-14T21:43:57.3074326Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.3074573Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3074972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3075316Z return mod(**inputs) 2025-08-14T21:43:57.3075713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3076258Z outputs = self.roberta( 2025-08-14T21:43:57.3076684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3077124Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3077554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3077960Z layer_outputs = layer_module( 2025-08-14T21:43:57.3078313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3078671Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3079080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3079489Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3079871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3080228Z return func(*args, **kwargs) 2025-08-14T21:43:57.3080613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3081007Z self_outputs = self.self( 2025-08-14T21:43:57.3081357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3081718Z return func(*args, **kwargs) 2025-08-14T21:43:57.3082125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.3082610Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.3082792Z 2025-08-14T21:43:57.3082893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3083248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3083574Z return mod(**inputs) 2025-08-14T21:43:57.3083957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3084349Z outputs = self.roberta( 2025-08-14T21:43:57.3084730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3085170Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3085566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3085973Z layer_outputs = layer_module( 2025-08-14T21:43:57.3086328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3086695Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3087105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3087558Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3087968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3088336Z return func(*args, **kwargs) 2025-08-14T21:43:57.3088720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.3089180Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.3089652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.3090056Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3090201Z 2025-08-14T21:43:57.3090325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3090684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3091018Z return mod(**inputs) 2025-08-14T21:43:57.3091395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3091797Z outputs = self.roberta( 2025-08-14T21:43:57.3092185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3092585Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3092975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3093378Z layer_outputs = layer_module( 2025-08-14T21:43:57.3093721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3094092Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3094527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3094952Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3095372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3095793Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3096232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3096712Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3097161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.3097571Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3097716Z 2025-08-14T21:43:57.3097822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3098182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3098506Z return mod(**inputs) 2025-08-14T21:43:57.3098903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3099324Z outputs = self.roberta( 2025-08-14T21:43:57.3099757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3100182Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3100607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3101031Z layer_outputs = layer_module( 2025-08-14T21:43:57.3101374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3101735Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3102167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3102594Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3103002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3103427Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3103915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3104423Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3104914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.3105372Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.3105757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.3106098Z return self.act(input) 2025-08-14T21:43:57.3106215Z 2025-08-14T21:43:57.3106325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3106707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3107059Z return mod(**inputs) 2025-08-14T21:43:57.3107437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3107855Z outputs = self.roberta( 2025-08-14T21:43:57.3108270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3108799Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3109202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3109614Z layer_outputs = layer_module( 2025-08-14T21:43:57.3109980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3110354Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3110791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3111249Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3111694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3112111Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3112585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.3113118Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.3113623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.3114064Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3114223Z 2025-08-14T21:43:57.3114336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3114787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3115131Z return mod(**inputs) 2025-08-14T21:43:57.3115562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3116076Z outputs = self.roberta( 2025-08-14T21:43:57.3116527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3116975Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3117446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3117877Z layer_outputs = layer_module( 2025-08-14T21:43:57.3118246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3118605Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3119011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3119452Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3119836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3120248Z return func(*args, **kwargs) 2025-08-14T21:43:57.3120647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3121051Z self_outputs = self.self( 2025-08-14T21:43:57.3121408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3121786Z return func(*args, **kwargs) 2025-08-14T21:43:57.3122177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.3122711Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.3122979Z 2025-08-14T21:43:57.3123088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3123453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3123780Z return mod(**inputs) 2025-08-14T21:43:57.3124161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3124564Z outputs = self.roberta( 2025-08-14T21:43:57.3124950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3125372Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3125755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3126150Z layer_outputs = layer_module( 2025-08-14T21:43:57.3126489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3126834Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3127231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3127633Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3128007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3128374Z return func(*args, **kwargs) 2025-08-14T21:43:57.3128766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3129182Z self_outputs = self.self( 2025-08-14T21:43:57.3129522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3129880Z return func(*args, **kwargs) 2025-08-14T21:43:57.3130259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.3130645Z self.key(current_states) 2025-08-14T21:43:57.3130757Z 2025-08-14T21:43:57.3130859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3131212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3131551Z return mod(**inputs) 2025-08-14T21:43:57.3131925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3132313Z outputs = self.roberta( 2025-08-14T21:43:57.3132690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3133096Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3133499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3133895Z layer_outputs = layer_module( 2025-08-14T21:43:57.3134271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3134626Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3135018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3135423Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3135794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3136168Z return func(*args, **kwargs) 2025-08-14T21:43:57.3136544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3136934Z self_outputs = self.self( 2025-08-14T21:43:57.3137286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3137639Z return func(*args, **kwargs) 2025-08-14T21:43:57.3138025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.3138422Z self.value(current_states) 2025-08-14T21:43:57.3138538Z 2025-08-14T21:43:57.3138624Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.3138851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3139205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3139525Z return mod(**inputs) 2025-08-14T21:43:57.3139892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3140292Z outputs = self.roberta( 2025-08-14T21:43:57.3140673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3141067Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3141452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3141852Z layer_outputs = layer_module( 2025-08-14T21:43:57.3142192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3142542Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3142942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3143367Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3143739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3144097Z return func(*args, **kwargs) 2025-08-14T21:43:57.3144473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3144862Z self_outputs = self.self( 2025-08-14T21:43:57.3145218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3145621Z return func(*args, **kwargs) 2025-08-14T21:43:57.3146008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.3146470Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.3146656Z 2025-08-14T21:43:57.3146761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3147145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3147474Z return mod(**inputs) 2025-08-14T21:43:57.3147862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3148363Z outputs = self.roberta( 2025-08-14T21:43:57.3148749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3149162Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3149553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3149955Z layer_outputs = layer_module( 2025-08-14T21:43:57.3150300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3150674Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3151076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3151493Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3151891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3152280Z return func(*args, **kwargs) 2025-08-14T21:43:57.3152690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.3153227Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.3153725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.3154184Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3154340Z 2025-08-14T21:43:57.3154455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3154858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3155223Z return mod(**inputs) 2025-08-14T21:43:57.3155643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3156166Z outputs = self.roberta( 2025-08-14T21:43:57.3156604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3157048Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3157444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3157875Z layer_outputs = layer_module( 2025-08-14T21:43:57.3158226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3158586Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3158997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3159419Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3159823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3160231Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3160668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3161157Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3161612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.3162026Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3162173Z 2025-08-14T21:43:57.3162298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3162664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3163025Z return mod(**inputs) 2025-08-14T21:43:57.3163436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3163860Z outputs = self.roberta( 2025-08-14T21:43:57.3164268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3164690Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3165086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3165521Z layer_outputs = layer_module( 2025-08-14T21:43:57.3165878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3166258Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3166692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3167136Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3167555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3167977Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3168437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3168946Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3169409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.3169878Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.3170280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.3170631Z return self.act(input) 2025-08-14T21:43:57.3170757Z 2025-08-14T21:43:57.3170868Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3171252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3171594Z return mod(**inputs) 2025-08-14T21:43:57.3171991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3172440Z outputs = self.roberta( 2025-08-14T21:43:57.3172841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3173262Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3173677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3174103Z layer_outputs = layer_module( 2025-08-14T21:43:57.3174471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3174874Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3175301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3175755Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3176160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3176549Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3177005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.3177500Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.3177986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.3178394Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3178539Z 2025-08-14T21:43:57.3178644Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3178999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3179315Z return mod(**inputs) 2025-08-14T21:43:57.3179719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3180145Z outputs = self.roberta( 2025-08-14T21:43:57.3180530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3180930Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3181344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3181731Z layer_outputs = layer_module( 2025-08-14T21:43:57.3182070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3182432Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3182836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3183249Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3183630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3184004Z return func(*args, **kwargs) 2025-08-14T21:43:57.3184410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3184833Z self_outputs = self.self( 2025-08-14T21:43:57.3185213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3185605Z return func(*args, **kwargs) 2025-08-14T21:43:57.3186026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.3186593Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.3186878Z 2025-08-14T21:43:57.3187017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3187404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3187743Z return mod(**inputs) 2025-08-14T21:43:57.3188124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3188529Z outputs = self.roberta( 2025-08-14T21:43:57.3188915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3189343Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3189737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3190141Z layer_outputs = layer_module( 2025-08-14T21:43:57.3190495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3190874Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3191346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3191785Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3192219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3192624Z return func(*args, **kwargs) 2025-08-14T21:43:57.3193043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3193534Z self_outputs = self.self( 2025-08-14T21:43:57.3193926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3194342Z return func(*args, **kwargs) 2025-08-14T21:43:57.3194781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.3195208Z self.key(current_states) 2025-08-14T21:43:57.3195330Z 2025-08-14T21:43:57.3195442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3195901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3196261Z return mod(**inputs) 2025-08-14T21:43:57.3196682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3197114Z outputs = self.roberta( 2025-08-14T21:43:57.3197546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3197997Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3198428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3198861Z layer_outputs = layer_module( 2025-08-14T21:43:57.3199232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3199623Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3200058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3200500Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3200908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3201298Z return func(*args, **kwargs) 2025-08-14T21:43:57.3201716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3202142Z self_outputs = self.self( 2025-08-14T21:43:57.3202545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3202928Z return func(*args, **kwargs) 2025-08-14T21:43:57.3203340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.3203765Z self.value(current_states) 2025-08-14T21:43:57.3203891Z 2025-08-14T21:43:57.3203987Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.3204236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3204614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3204977Z return mod(**inputs) 2025-08-14T21:43:57.3205385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3205816Z outputs = self.roberta( 2025-08-14T21:43:57.3206227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3206656Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3207093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3207521Z layer_outputs = layer_module( 2025-08-14T21:43:57.3207914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3208267Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3208815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3209252Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3209640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3210011Z return func(*args, **kwargs) 2025-08-14T21:43:57.3210405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3210810Z self_outputs = self.self( 2025-08-14T21:43:57.3211168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3211545Z return func(*args, **kwargs) 2025-08-14T21:43:57.3211942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.3212408Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.3212592Z 2025-08-14T21:43:57.3212700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3213060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3213387Z return mod(**inputs) 2025-08-14T21:43:57.3213773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3214176Z outputs = self.roberta( 2025-08-14T21:43:57.3214567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3214975Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3215372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3215774Z layer_outputs = layer_module( 2025-08-14T21:43:57.3216126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3216487Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3216889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3217370Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3217762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3218135Z return func(*args, **kwargs) 2025-08-14T21:43:57.3218523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.3218985Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.3219471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.3219891Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3220045Z 2025-08-14T21:43:57.3220156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3220519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3220849Z return mod(**inputs) 2025-08-14T21:43:57.3221258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3221665Z outputs = self.roberta( 2025-08-14T21:43:57.3222060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3222443Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3222831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3223225Z layer_outputs = layer_module( 2025-08-14T21:43:57.3223566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3223912Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3224309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3224714Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3225108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3225497Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3225951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3226465Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3226955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.3227371Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3227527Z 2025-08-14T21:43:57.3227631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3227978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3228288Z return mod(**inputs) 2025-08-14T21:43:57.3228666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3229063Z outputs = self.roberta( 2025-08-14T21:43:57.3229444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3229843Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3230250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3230653Z layer_outputs = layer_module( 2025-08-14T21:43:57.3230993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3231384Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3231819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3232260Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3232679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3233101Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3233560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3234086Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3234553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.3235025Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.3235445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.3235874Z return self.act(input) 2025-08-14T21:43:57.3236041Z 2025-08-14T21:43:57.3236163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3236572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3236957Z return mod(**inputs) 2025-08-14T21:43:57.3237361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3237793Z outputs = self.roberta( 2025-08-14T21:43:57.3238193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3238659Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3239113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3239543Z layer_outputs = layer_module( 2025-08-14T21:43:57.3239911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3240289Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3240729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3241166Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3241594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3242005Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3242458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.3242980Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.3243462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.3243897Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3244050Z 2025-08-14T21:43:57.3244159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3244537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3244872Z return mod(**inputs) 2025-08-14T21:43:57.3245281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3245708Z outputs = self.roberta( 2025-08-14T21:43:57.3246142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3246593Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3247027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3247445Z layer_outputs = layer_module( 2025-08-14T21:43:57.3247791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3248182Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3248606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3249070Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3249455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3249854Z return func(*args, **kwargs) 2025-08-14T21:43:57.3250260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3250671Z self_outputs = self.self( 2025-08-14T21:43:57.3251063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3251449Z return func(*args, **kwargs) 2025-08-14T21:43:57.3251887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.3252533Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.3252811Z 2025-08-14T21:43:57.3252927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3253304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3253637Z return mod(**inputs) 2025-08-14T21:43:57.3254030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3254450Z outputs = self.roberta( 2025-08-14T21:43:57.3254848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3255264Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3255672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3256089Z layer_outputs = layer_module( 2025-08-14T21:43:57.3256450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3256818Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3257242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3257650Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3258020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3258378Z return func(*args, **kwargs) 2025-08-14T21:43:57.3258760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3259151Z self_outputs = self.self( 2025-08-14T21:43:57.3259498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3259860Z return func(*args, **kwargs) 2025-08-14T21:43:57.3260241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.3260633Z self.key(current_states) 2025-08-14T21:43:57.3260744Z 2025-08-14T21:43:57.3260845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3261219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3261539Z return mod(**inputs) 2025-08-14T21:43:57.3261919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3262331Z outputs = self.roberta( 2025-08-14T21:43:57.3262734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3263142Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3263545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3263934Z layer_outputs = layer_module( 2025-08-14T21:43:57.3264269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3264627Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3265025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3265473Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3265842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3266217Z return func(*args, **kwargs) 2025-08-14T21:43:57.3266601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3266991Z self_outputs = self.self( 2025-08-14T21:43:57.3267340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3267693Z return func(*args, **kwargs) 2025-08-14T21:43:57.3268074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.3268463Z self.value(current_states) 2025-08-14T21:43:57.3268576Z 2025-08-14T21:43:57.3268662Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.3268893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3269241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3269556Z return mod(**inputs) 2025-08-14T21:43:57.3269925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3270314Z outputs = self.roberta( 2025-08-14T21:43:57.3270691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3271087Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3271476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3271882Z layer_outputs = layer_module( 2025-08-14T21:43:57.3272248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3272627Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3273061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3273498Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3273898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3274288Z return func(*args, **kwargs) 2025-08-14T21:43:57.3274704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3275148Z self_outputs = self.self( 2025-08-14T21:43:57.3275520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3276188Z return func(*args, **kwargs) 2025-08-14T21:43:57.3276629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.3277138Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.3277336Z 2025-08-14T21:43:57.3277448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3277833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3278207Z return mod(**inputs) 2025-08-14T21:43:57.3278626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3279045Z outputs = self.roberta( 2025-08-14T21:43:57.3279452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3279880Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3280313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3280737Z layer_outputs = layer_module( 2025-08-14T21:43:57.3281124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3281508Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3281933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3282366Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3282769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3283154Z return func(*args, **kwargs) 2025-08-14T21:43:57.3283569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.3284056Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.3284539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.3284971Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3285125Z 2025-08-14T21:43:57.3285237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3285613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3285955Z return mod(**inputs) 2025-08-14T21:43:57.3286350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3286776Z outputs = self.roberta( 2025-08-14T21:43:57.3287182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3287602Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3287998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3288424Z layer_outputs = layer_module( 2025-08-14T21:43:57.3288785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3289163Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3289593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3290039Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3290494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3290906Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3291368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3291850Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3292296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.3292738Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3292920Z 2025-08-14T21:43:57.3293031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3293405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3293738Z return mod(**inputs) 2025-08-14T21:43:57.3294142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3294569Z outputs = self.roberta( 2025-08-14T21:43:57.3294993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3295414Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3295856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3296284Z layer_outputs = layer_module( 2025-08-14T21:43:57.3296645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3297029Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3297461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3297901Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3298326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3298745Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3299202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3299710Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3300175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.3300641Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.3301043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.3301397Z return self.act(input) 2025-08-14T21:43:57.3301531Z 2025-08-14T21:43:57.3301642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3302025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3302369Z return mod(**inputs) 2025-08-14T21:43:57.3302774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3303201Z outputs = self.roberta( 2025-08-14T21:43:57.3303608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3304033Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3304460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3304888Z layer_outputs = layer_module( 2025-08-14T21:43:57.3305258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3305664Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3306092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3306530Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3306954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3307368Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3307824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.3308939Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.3309439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.3309887Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3310042Z 2025-08-14T21:43:57.3310154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3310578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3310922Z return mod(**inputs) 2025-08-14T21:43:57.3311366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3311792Z outputs = self.roberta( 2025-08-14T21:43:57.3312196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3312620Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3313060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3313503Z layer_outputs = layer_module( 2025-08-14T21:43:57.3313877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3314277Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3314732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3315179Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3315590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3315678Z return func(*args, **kwargs) 2025-08-14T21:43:57.3316034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3316128Z self_outputs = self.self( 2025-08-14T21:43:57.3316399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3316481Z return func(*args, **kwargs) 2025-08-14T21:43:57.3316788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.3317016Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.3317020Z 2025-08-14T21:43:57.3317137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3317364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3317436Z return mod(**inputs) 2025-08-14T21:43:57.3317749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3317822Z outputs = self.roberta( 2025-08-14T21:43:57.3318147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3318267Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3318561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3318637Z layer_outputs = layer_module( 2025-08-14T21:43:57.3318883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3318970Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3319272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3319394Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3319647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3319732Z return func(*args, **kwargs) 2025-08-14T21:43:57.3320017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3320098Z self_outputs = self.self( 2025-08-14T21:43:57.3320372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3320447Z return func(*args, **kwargs) 2025-08-14T21:43:57.3320796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.3320870Z self.key(current_states) 2025-08-14T21:43:57.3320876Z 2025-08-14T21:43:57.3320983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3321190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3321255Z return mod(**inputs) 2025-08-14T21:43:57.3321576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3321646Z outputs = self.roberta( 2025-08-14T21:43:57.3321918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3321999Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3322271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3322343Z layer_outputs = layer_module( 2025-08-14T21:43:57.3322566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3322645Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3322920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3323000Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3323241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3323317Z return func(*args, **kwargs) 2025-08-14T21:43:57.3323586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3323662Z self_outputs = self.self( 2025-08-14T21:43:57.3323931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3323999Z return func(*args, **kwargs) 2025-08-14T21:43:57.3324275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.3324347Z self.value(current_states) 2025-08-14T21:43:57.3324350Z 2025-08-14T21:43:57.3324431Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.3324561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3324760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3324831Z return mod(**inputs) 2025-08-14T21:43:57.3325102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3325170Z outputs = self.roberta( 2025-08-14T21:43:57.3325443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3325515Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3325808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3325886Z layer_outputs = layer_module( 2025-08-14T21:43:57.3326103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3326189Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3326460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3326557Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3326807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3326893Z return func(*args, **kwargs) 2025-08-14T21:43:57.3327167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3327240Z self_outputs = self.self( 2025-08-14T21:43:57.3327479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3327552Z return func(*args, **kwargs) 2025-08-14T21:43:57.3327819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.3327951Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.3327962Z 2025-08-14T21:43:57.3328066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3328261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3328335Z return mod(**inputs) 2025-08-14T21:43:57.3328606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3328676Z outputs = self.roberta( 2025-08-14T21:43:57.3328951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3329022Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3329295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3329367Z layer_outputs = layer_module( 2025-08-14T21:43:57.3329583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3329667Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3329932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3330011Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3330256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3330328Z return func(*args, **kwargs) 2025-08-14T21:43:57.3330599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.3330752Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.3331019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.3331111Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3331115Z 2025-08-14T21:43:57.3331217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3331422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3331487Z return mod(**inputs) 2025-08-14T21:43:57.3331760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3331857Z outputs = self.roberta( 2025-08-14T21:43:57.3332124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3332195Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3332479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3332550Z layer_outputs = layer_module( 2025-08-14T21:43:57.3332791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3332870Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3333172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3333265Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3333516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3333596Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3333890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3334008Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3334278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.3334360Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3334363Z 2025-08-14T21:43:57.3334473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3334665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3334728Z return mod(**inputs) 2025-08-14T21:43:57.3334990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3335054Z outputs = self.roberta( 2025-08-14T21:43:57.3335306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3335384Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3335640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3335718Z layer_outputs = layer_module( 2025-08-14T21:43:57.3335928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3336003Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3336270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3336350Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3336598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3336680Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3336991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3337115Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3337379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.3337492Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.3337709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.3337799Z return self.act(input) 2025-08-14T21:43:57.3337803Z 2025-08-14T21:43:57.3337912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3338109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3338174Z return mod(**inputs) 2025-08-14T21:43:57.3338457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3338523Z outputs = self.roberta( 2025-08-14T21:43:57.3338811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3338893Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3339177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3339259Z layer_outputs = layer_module( 2025-08-14T21:43:57.3339485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3339561Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3339827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3339907Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3340157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3340239Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3340533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.3340667Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.3340926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.3341008Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3341011Z 2025-08-14T21:43:57.3341117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3341309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3341382Z return mod(**inputs) 2025-08-14T21:43:57.3341643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3341712Z outputs = self.roberta( 2025-08-14T21:43:57.3341986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3342061Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3342333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3342406Z layer_outputs = layer_module( 2025-08-14T21:43:57.3342635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3342725Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3343026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3343112Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3343378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3343455Z return func(*args, **kwargs) 2025-08-14T21:43:57.3343745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3343829Z self_outputs = self.self( 2025-08-14T21:43:57.3344086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3344165Z return func(*args, **kwargs) 2025-08-14T21:43:57.3344430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.3344639Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.3344649Z 2025-08-14T21:43:57.3344753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3344972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3345046Z return mod(**inputs) 2025-08-14T21:43:57.3345341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3345412Z outputs = self.roberta( 2025-08-14T21:43:57.3345701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3345772Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3346036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3346108Z layer_outputs = layer_module( 2025-08-14T21:43:57.3346316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3346401Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3346659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3346740Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3346979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3347050Z return func(*args, **kwargs) 2025-08-14T21:43:57.3347320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3347389Z self_outputs = self.self( 2025-08-14T21:43:57.3347624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3347701Z return func(*args, **kwargs) 2025-08-14T21:43:57.3347967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.3348045Z self.key(current_states) 2025-08-14T21:43:57.3348049Z 2025-08-14T21:43:57.3348152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3348345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3348416Z return mod(**inputs) 2025-08-14T21:43:57.3348684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3348752Z outputs = self.roberta( 2025-08-14T21:43:57.3349027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3349122Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3349394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3349467Z layer_outputs = layer_module( 2025-08-14T21:43:57.3349683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3349769Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3350035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3350142Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3350380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3350448Z return func(*args, **kwargs) 2025-08-14T21:43:57.3350720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3350789Z self_outputs = self.self( 2025-08-14T21:43:57.3351039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3351116Z return func(*args, **kwargs) 2025-08-14T21:43:57.3351403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.3351486Z self.value(current_states) 2025-08-14T21:43:57.3351490Z 2025-08-14T21:43:57.3351574Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.3351676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3351876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3351942Z return mod(**inputs) 2025-08-14T21:43:57.3352209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3352284Z outputs = self.roberta( 2025-08-14T21:43:57.3352553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3352635Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3352916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3352991Z layer_outputs = layer_module( 2025-08-14T21:43:57.3353229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3353309Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3353597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3353684Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3353932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3354011Z return func(*args, **kwargs) 2025-08-14T21:43:57.3354291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3354364Z self_outputs = self.self( 2025-08-14T21:43:57.3354625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3354699Z return func(*args, **kwargs) 2025-08-14T21:43:57.3354987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.3355126Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.3355130Z 2025-08-14T21:43:57.3355258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3355472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3355543Z return mod(**inputs) 2025-08-14T21:43:57.3355918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3356005Z outputs = self.roberta( 2025-08-14T21:43:57.3356295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3356380Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3356690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3356768Z layer_outputs = layer_module( 2025-08-14T21:43:57.3357004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3357087Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3357380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3357493Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3357732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3357829Z return func(*args, **kwargs) 2025-08-14T21:43:57.3358099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.3358231Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.3358504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.3358587Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3358593Z 2025-08-14T21:43:57.3358702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3358900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3358970Z return mod(**inputs) 2025-08-14T21:43:57.3359255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3359324Z outputs = self.roberta( 2025-08-14T21:43:57.3359603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3359677Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3359945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3360024Z layer_outputs = layer_module( 2025-08-14T21:43:57.3360246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3360323Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3360600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3360681Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3360948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3361025Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3361326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3361453Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3361721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.3361836Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3361839Z 2025-08-14T21:43:57.3361946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3362146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3362224Z return mod(**inputs) 2025-08-14T21:43:57.3362494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3362566Z outputs = self.roberta( 2025-08-14T21:43:57.3362862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3362935Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3363212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3363285Z layer_outputs = layer_module( 2025-08-14T21:43:57.3363504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3363605Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3363965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3364065Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3364321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3364407Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3364708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3364839Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3365125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.3365242Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.3365477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.3365548Z return self.act(input) 2025-08-14T21:43:57.3365551Z 2025-08-14T21:43:57.3365663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3365859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3365926Z return mod(**inputs) 2025-08-14T21:43:57.3366204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3366273Z outputs = self.roberta( 2025-08-14T21:43:57.3366541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3366625Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3366891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3366970Z layer_outputs = layer_module( 2025-08-14T21:43:57.3367189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3367269Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3367547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3367631Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3367898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3368008Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3368327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.3368483Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.3368757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.3368840Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3368850Z 2025-08-14T21:43:57.3368955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3369181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3369257Z return mod(**inputs) 2025-08-14T21:43:57.3369540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3369612Z outputs = self.roberta( 2025-08-14T21:43:57.3369901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3369995Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3370284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3370359Z layer_outputs = layer_module( 2025-08-14T21:43:57.3370606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3370699Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3370981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3371065Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3371324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3371399Z return func(*args, **kwargs) 2025-08-14T21:43:57.3371686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3371758Z self_outputs = self.self( 2025-08-14T21:43:57.3372013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3372094Z return func(*args, **kwargs) 2025-08-14T21:43:57.3372375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:43:57.3372599Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:43:57.3372603Z 2025-08-14T21:43:57.3372711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3372920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3372995Z return mod(**inputs) 2025-08-14T21:43:57.3373281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3373351Z outputs = self.roberta( 2025-08-14T21:43:57.3373644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3373720Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3374008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3374085Z layer_outputs = layer_module( 2025-08-14T21:43:57.3374311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3374402Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3374706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3374798Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3375051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3375124Z return func(*args, **kwargs) 2025-08-14T21:43:57.3375415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3375511Z self_outputs = self.self( 2025-08-14T21:43:57.3375761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3375843Z return func(*args, **kwargs) 2025-08-14T21:43:57.3376126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:43:57.3376210Z self.key(current_states) 2025-08-14T21:43:57.3376213Z 2025-08-14T21:43:57.3376322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3376547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3376627Z return mod(**inputs) 2025-08-14T21:43:57.3376935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3377008Z outputs = self.roberta( 2025-08-14T21:43:57.3377296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3377375Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3377667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3377740Z layer_outputs = layer_module( 2025-08-14T21:43:57.3377954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3378039Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3378310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3378397Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3378640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3378711Z return func(*args, **kwargs) 2025-08-14T21:43:57.3378988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3379059Z self_outputs = self.self( 2025-08-14T21:43:57.3379295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3379373Z return func(*args, **kwargs) 2025-08-14T21:43:57.3379644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:43:57.3379723Z self.value(current_states) 2025-08-14T21:43:57.3379727Z 2025-08-14T21:43:57.3379810Z cudagraph partition due to non gpu ops 2025-08-14T21:43:57.3379914Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3380122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3380192Z return mod(**inputs) 2025-08-14T21:43:57.3380479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3380558Z outputs = self.roberta( 2025-08-14T21:43:57.3380841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3380946Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3381232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3381307Z layer_outputs = layer_module( 2025-08-14T21:43:57.3381544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3381627Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3381926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3382039Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3382281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3382358Z return func(*args, **kwargs) 2025-08-14T21:43:57.3382630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:43:57.3382697Z self_outputs = self.self( 2025-08-14T21:43:57.3382959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3383029Z return func(*args, **kwargs) 2025-08-14T21:43:57.3383321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:43:57.3383456Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:43:57.3383461Z 2025-08-14T21:43:57.3383563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3383766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3383834Z return mod(**inputs) 2025-08-14T21:43:57.3384108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3384178Z outputs = self.roberta( 2025-08-14T21:43:57.3384445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3384524Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3384793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3384862Z layer_outputs = layer_module( 2025-08-14T21:43:57.3385095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3385177Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3385469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:43:57.3385557Z self_attention_outputs = self.attention( 2025-08-14T21:43:57.3385806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:43:57.3385887Z return func(*args, **kwargs) 2025-08-14T21:43:57.3386171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:43:57.3386308Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:43:57.3386601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:43:57.3386690Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3386693Z 2025-08-14T21:43:57.3386807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3387014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3387104Z return mod(**inputs) 2025-08-14T21:43:57.3387398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3387467Z outputs = self.roberta( 2025-08-14T21:43:57.3387758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3387834Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3388116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3388219Z layer_outputs = layer_module( 2025-08-14T21:43:57.3388445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3388526Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3388814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3388904Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3389210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3389292Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3389631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3389767Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3390056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:43:57.3390152Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3390156Z 2025-08-14T21:43:57.3390270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3390484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3390565Z return mod(**inputs) 2025-08-14T21:43:57.3390861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3390944Z outputs = self.roberta( 2025-08-14T21:43:57.3391236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3391318Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3391618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3391699Z layer_outputs = layer_module( 2025-08-14T21:43:57.3391934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3392028Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3392318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3392415Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3392694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3392779Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3393111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:43:57.3393245Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:57.3393544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:43:57.3393668Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:57.3393920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:57.3394002Z return self.act(input) 2025-08-14T21:43:57.3394006Z 2025-08-14T21:43:57.3394120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3394332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3394410Z return mod(**inputs) 2025-08-14T21:43:57.3394717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:43:57.3394800Z outputs = self.roberta( 2025-08-14T21:43:57.3395119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:43:57.3395196Z encoder_outputs = self.encoder( 2025-08-14T21:43:57.3395504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:43:57.3395584Z layer_outputs = layer_module( 2025-08-14T21:43:57.3395893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:57.3396023Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:57.3396317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:43:57.3396437Z layer_output = apply_chunking_to_forward( 2025-08-14T21:43:57.3396722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:43:57.3396815Z return forward_fn(*input_tensors) 2025-08-14T21:43:57.3397140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:43:57.3397278Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:57.3397569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:43:57.3397656Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:57.3397660Z 2025-08-14T21:43:57.3397769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3397987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3398055Z return mod(**inputs) 2025-08-14T21:43:57.3398340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1052, in forward 2025-08-14T21:43:57.3398452Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:43:57.3398736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 756, in forward 2025-08-14T21:43:57.3398818Z x = self.dense(features) 2025-08-14T21:43:57.3398823Z 2025-08-14T21:43:57.3398929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3399133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3399212Z return mod(**inputs) 2025-08-14T21:43:57.3399494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1052, in forward 2025-08-14T21:43:57.3399607Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:43:57.3399888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 761, in forward 2025-08-14T21:43:57.3399962Z x = self.decoder(x) 2025-08-14T21:43:57.3399967Z 2025-08-14T21:43:57.3400081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:57.3400283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:57.3400382Z return mod(**inputs) 2025-08-14T21:43:57.3400676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1059, in forward 2025-08-14T21:43:57.3400880Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:43:57.3400884Z 2025-08-14T21:44:06.0791665Z Compilation time (from dynamo_timed): 15.017077846 2025-08-14T21:44:06.0883670Z pass 2025-08-14T21:44:06.0884105Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:06.0884940Z TIMING: _recursive_pre_grad_passes:0.00717 _recursive_joint_graph_passes:0.36718 _recursive_post_grad_passes:0.08309 async_compile.wait:0.75221 code_gen:7.5922 inductor_compile:8.76947 backend_compile:11.91483 gc:0.00101 entire_frame_compile:15.01708 total_wall_time:15.01708 2025-08-14T21:44:06.0886266Z STATS: call_* op count: 297 | FakeTensorMode.__torch_dispatch__:12436 | FakeTensor.__torch_dispatch__:4756 | ProxyTorchDispatchMode.__torch_dispatch__:4530 2025-08-14T21:44:06.0886772Z Dynamo produced 1 graphs covering 297 ops with 0 graph breaks (0 unique) 2025-08-14T21:44:11.3161089Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:44:11.3162110Z from pkg_resources import resource_filename 2025-08-14T21:44:12.0111996Z 2025-08-14T21:44:20.9117387Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:44:20.9117716Z loading model: 0it [00:08, ?it/s] 2025-08-14T21:44:20.9146423Z cpu eval DebertaV2ForMaskedLM 2025-08-14T21:44:21.0473187Z Compilation time (from dynamo_timed): 0 2025-08-14T21:44:21.0473513Z pass_due_to_skip 2025-08-14T21:44:21.0478900Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:21.0479407Z TIMING: total_wall_time:0 2025-08-14T21:44:21.0479614Z STATS: call_* op count: 0 2025-08-14T21:44:21.0479904Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-08-14T21:44:25.6528246Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:44:25.6530865Z from pkg_resources import resource_filename 2025-08-14T21:44:26.3441423Z 2025-08-14T21:44:33.7663313Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:44:33.7665303Z loading model: 0it [00:07, ?it/s] 2025-08-14T21:44:33.7693609Z cpu eval DebertaV2ForQuestionAnswering 2025-08-14T21:44:37.0392601Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:38.6227482Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:39.8903201Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:55.7186205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7186641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7187065Z return mod(**inputs) 2025-08-14T21:44:55.7187539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7188018Z outputs = self.deberta( 2025-08-14T21:44:55.7188460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7188905Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7189761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7190222Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7190652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7191049Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7191491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7191998Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7192526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7192977Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7193422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7193979Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7194252Z 2025-08-14T21:44:55.7194426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7194848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7195230Z return mod(**inputs) 2025-08-14T21:44:55.7195882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7196347Z outputs = self.deberta( 2025-08-14T21:44:55.7196770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7197213Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7197753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7198210Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7198600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7198980Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7199416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7199867Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7200324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7200753Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7201170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7201690Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7201922Z 2025-08-14T21:44:55.7202036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7202396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7202712Z return mod(**inputs) 2025-08-14T21:44:55.7203090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7203475Z outputs = self.deberta( 2025-08-14T21:44:55.7203847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7204240Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7204625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7205049Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7205414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7205783Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7206182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7206604Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7207028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7207489Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7207910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7208428Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7209292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7209848Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7210040Z 2025-08-14T21:44:55.7210146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7210522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7210844Z return mod(**inputs) 2025-08-14T21:44:55.7211255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7211644Z outputs = self.deberta( 2025-08-14T21:44:55.7212025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7212438Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7212834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7213250Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7213627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7213992Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7214400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7214828Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7215241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7215640Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7216033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7216589Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7216863Z 2025-08-14T21:44:55.7216967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7217325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7217647Z return mod(**inputs) 2025-08-14T21:44:55.7218030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7218430Z outputs = self.deberta( 2025-08-14T21:44:55.7218811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7219208Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7219644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7220091Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7220473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7220852Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7221282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7221702Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7222135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7222541Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7222939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7223478Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7223739Z 2025-08-14T21:44:55.7223862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7224230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7224575Z return mod(**inputs) 2025-08-14T21:44:55.7224994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7225419Z outputs = self.deberta( 2025-08-14T21:44:55.7225821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7226252Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7226651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7227068Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7227432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7227793Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7228200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7228622Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7229073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7229492Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7229919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7230470Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7230738Z 2025-08-14T21:44:55.7230857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7231236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7231578Z return mod(**inputs) 2025-08-14T21:44:55.7231985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7232416Z outputs = self.deberta( 2025-08-14T21:44:55.7232823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7233265Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7233705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7234175Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7234572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7234963Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7235402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7235971Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7236455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7236928Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7237383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7237956Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7238584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7239167Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7239373Z 2025-08-14T21:44:55.7239497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7239901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7240261Z return mod(**inputs) 2025-08-14T21:44:55.7240685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7241114Z outputs = self.deberta( 2025-08-14T21:44:55.7241546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7241984Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7242421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7242878Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7243278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7243682Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7244120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7244571Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7245030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7245470Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7245903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7246340Z context_layer = torch.bmm( 2025-08-14T21:44:55.7246477Z 2025-08-14T21:44:55.7246601Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7246981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7247319Z return mod(**inputs) 2025-08-14T21:44:55.7247723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7248148Z outputs = self.deberta( 2025-08-14T21:44:55.7248550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7248969Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7249387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7249849Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7250230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7250612Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7251040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7251482Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7251939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7252370Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7252803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7253355Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7253610Z 2025-08-14T21:44:55.7253723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7254123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7254464Z return mod(**inputs) 2025-08-14T21:44:55.7254873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7255302Z outputs = self.deberta( 2025-08-14T21:44:55.7255705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7256141Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7256573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7257022Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7257410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7257805Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7258230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7258686Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7259130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7259597Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7260072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7260516Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7260661Z 2025-08-14T21:44:55.7260777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7261153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7261500Z return mod(**inputs) 2025-08-14T21:44:55.7261920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7262343Z outputs = self.deberta( 2025-08-14T21:44:55.7262754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7263181Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7263611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7264041Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7264452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7264841Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7265279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7265747Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7266223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7266686Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7266834Z 2025-08-14T21:44:55.7266959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7267338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7267687Z return mod(**inputs) 2025-08-14T21:44:55.7268093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7268507Z outputs = self.deberta( 2025-08-14T21:44:55.7268928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7269352Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7269786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7270221Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7270611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7270992Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7271420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7271890Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7272368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7272838Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7273236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7273600Z return self.act(input) 2025-08-14T21:44:55.7273725Z 2025-08-14T21:44:55.7273836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7274223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7274566Z return mod(**inputs) 2025-08-14T21:44:55.7274977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7275416Z outputs = self.deberta( 2025-08-14T21:44:55.7275909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7276354Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7276809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7277278Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7277667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7278062Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7278501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7278998Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7279513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7279969Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7280127Z 2025-08-14T21:44:55.7280248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7280642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7281015Z return mod(**inputs) 2025-08-14T21:44:55.7281439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7281894Z outputs = self.deberta( 2025-08-14T21:44:55.7282309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7282744Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7283178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7283627Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7284040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7284432Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7284890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7285342Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7285979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7286444Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7286890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7287454Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7287728Z 2025-08-14T21:44:55.7287848Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7288242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7288596Z return mod(**inputs) 2025-08-14T21:44:55.7289009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7289454Z outputs = self.deberta( 2025-08-14T21:44:55.7289874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7290316Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7290742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7291183Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7291570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7291941Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7292368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7292814Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7293271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7293707Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7294143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7294732Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7294992Z 2025-08-14T21:44:55.7295115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7295501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7295859Z return mod(**inputs) 2025-08-14T21:44:55.7296268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7296688Z outputs = self.deberta( 2025-08-14T21:44:55.7297112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7297533Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7297947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7298375Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7298758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7299151Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7299578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7300031Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7300469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7300900Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7301318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7301878Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7302467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7303003Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7303196Z 2025-08-14T21:44:55.7303307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7303686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7304026Z return mod(**inputs) 2025-08-14T21:44:55.7304432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7304861Z outputs = self.deberta( 2025-08-14T21:44:55.7305267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7305692Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7306107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7306540Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7306929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7307316Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7307734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7308183Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7308631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7309249Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7309729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7310306Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7310596Z 2025-08-14T21:44:55.7310706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7311088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7311431Z return mod(**inputs) 2025-08-14T21:44:55.7311840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7312304Z outputs = self.deberta( 2025-08-14T21:44:55.7312703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7313130Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7313565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7314014Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7314448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7314835Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7315309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7315804Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7316267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7316716Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7317161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7317731Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7318054Z 2025-08-14T21:44:55.7318169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7318555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7318903Z return mod(**inputs) 2025-08-14T21:44:55.7319311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7319754Z outputs = self.deberta( 2025-08-14T21:44:55.7320194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7320648Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7321091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7321537Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7321938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7322297Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7322706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7323134Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7323560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7323964Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7324371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7324916Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7325161Z 2025-08-14T21:44:55.7325272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7325622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7325945Z return mod(**inputs) 2025-08-14T21:44:55.7326322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7326723Z outputs = self.deberta( 2025-08-14T21:44:55.7327119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7327526Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7327923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7328336Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7328705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7329080Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7329484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7329916Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7330333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7330744Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7331133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7331648Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7332199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7332696Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7332882Z 2025-08-14T21:44:55.7332988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7333350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7333669Z return mod(**inputs) 2025-08-14T21:44:55.7334058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7334453Z outputs = self.deberta( 2025-08-14T21:44:55.7334834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7335236Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7335631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7336046Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7336431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7336811Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7337229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7337674Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7338118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7338513Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7338913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7339307Z context_layer = torch.bmm( 2025-08-14T21:44:55.7339423Z 2025-08-14T21:44:55.7339536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7339901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7340211Z return mod(**inputs) 2025-08-14T21:44:55.7340581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7340988Z outputs = self.deberta( 2025-08-14T21:44:55.7341354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7341749Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7342136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7342543Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7342911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7343261Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7343668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7344071Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7344481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7344887Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7345284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7345783Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7346026Z 2025-08-14T21:44:55.7346131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7346482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7346797Z return mod(**inputs) 2025-08-14T21:44:55.7347165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7347561Z outputs = self.deberta( 2025-08-14T21:44:55.7347936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7348325Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7348713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7349120Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7349479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7349824Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7350217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7350628Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7351047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7351490Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7351934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7352367Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7352510Z 2025-08-14T21:44:55.7352628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7353000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7353342Z return mod(**inputs) 2025-08-14T21:44:55.7353748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7354164Z outputs = self.deberta( 2025-08-14T21:44:55.7354567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7355012Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7355435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7355962Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7356366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7356761Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7357198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7357646Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7358113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7358534Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7358673Z 2025-08-14T21:44:55.7358776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7359138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7359468Z return mod(**inputs) 2025-08-14T21:44:55.7359852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7360249Z outputs = self.deberta( 2025-08-14T21:44:55.7360644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7361047Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7361439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7361854Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7362222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7362586Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7362981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7363439Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7363873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7364308Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7364668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7364995Z return self.act(input) 2025-08-14T21:44:55.7365103Z 2025-08-14T21:44:55.7365208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7365546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7365855Z return mod(**inputs) 2025-08-14T21:44:55.7366215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7366614Z outputs = self.deberta( 2025-08-14T21:44:55.7366970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7367350Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7367725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7368115Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7368458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7368817Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7369196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7369628Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7370126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7370517Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7370661Z 2025-08-14T21:44:55.7370769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7371111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7371460Z return mod(**inputs) 2025-08-14T21:44:55.7371820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7372194Z outputs = self.deberta( 2025-08-14T21:44:55.7372551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7372940Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7373327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7373735Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7374102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7374463Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7374867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7376073Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7376501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7376888Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7377283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7377783Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7378022Z 2025-08-14T21:44:55.7378125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7378481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7378801Z return mod(**inputs) 2025-08-14T21:44:55.7379182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7379591Z outputs = self.deberta( 2025-08-14T21:44:55.7379959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7380346Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7380725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7381147Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7381503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7381845Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7382241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7382652Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7383058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7383482Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7383884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7384404Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7384628Z 2025-08-14T21:44:55.7384729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7385123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7385441Z return mod(**inputs) 2025-08-14T21:44:55.7385841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7386238Z outputs = self.deberta( 2025-08-14T21:44:55.7386626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7387031Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7387419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7387837Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7388206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7388572Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7388977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7389389Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7389797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7390202Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7390600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7391121Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7391681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7392174Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7392371Z 2025-08-14T21:44:55.7392479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7392860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7393204Z return mod(**inputs) 2025-08-14T21:44:55.7393603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7394048Z outputs = self.deberta( 2025-08-14T21:44:55.7394465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7394929Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7395360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7395889Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7396298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7396689Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7397135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7397575Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7398034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7398467Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7398916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7399532Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7399823Z 2025-08-14T21:44:55.7399943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7400353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7400714Z return mod(**inputs) 2025-08-14T21:44:55.7401139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7401580Z outputs = self.deberta( 2025-08-14T21:44:55.7402010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7402459Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7402899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7403351Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7403759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7404158Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7404608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7405066Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7405531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7405977Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7406417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7407013Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7407282Z 2025-08-14T21:44:55.7407389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7407752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7408073Z return mod(**inputs) 2025-08-14T21:44:55.7408462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7409039Z outputs = self.deberta( 2025-08-14T21:44:55.7409427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7409826Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7410224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7410682Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7411039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7411399Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7411804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7412218Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7412658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7413064Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7413469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7413988Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7414232Z 2025-08-14T21:44:55.7414335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7414730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7415082Z return mod(**inputs) 2025-08-14T21:44:55.7415524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7415926Z outputs = self.deberta( 2025-08-14T21:44:55.7416310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7416709Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7417099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7417518Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7417889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7418263Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7418659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7419077Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7419495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7419905Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7420307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7420825Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7421375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7421870Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7422061Z 2025-08-14T21:44:55.7422166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7422524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7422845Z return mod(**inputs) 2025-08-14T21:44:55.7423226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7423627Z outputs = self.deberta( 2025-08-14T21:44:55.7424008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7424445Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7424823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7425233Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7425598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7425958Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7426352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7426779Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7427188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7427658Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7428053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7428447Z context_layer = torch.bmm( 2025-08-14T21:44:55.7428563Z 2025-08-14T21:44:55.7428696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7429041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7429373Z return mod(**inputs) 2025-08-14T21:44:55.7429746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7430131Z outputs = self.deberta( 2025-08-14T21:44:55.7430504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7430897Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7431286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7431687Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7432048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7432408Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7432809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7433250Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7433698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7434128Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7434545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7435096Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7435344Z 2025-08-14T21:44:55.7435450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7435867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7436197Z return mod(**inputs) 2025-08-14T21:44:55.7436606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7437032Z outputs = self.deberta( 2025-08-14T21:44:55.7437417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7437809Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7438205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7438644Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7439005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7439382Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7439778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7440189Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7440597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7441058Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7441501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7441906Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7442051Z 2025-08-14T21:44:55.7442156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7442529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7442855Z return mod(**inputs) 2025-08-14T21:44:55.7443245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7443652Z outputs = self.deberta( 2025-08-14T21:44:55.7444031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7444432Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7444820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7445232Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7445598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7445953Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7446359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7446805Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7447253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7447662Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7447810Z 2025-08-14T21:44:55.7447912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7448267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7448596Z return mod(**inputs) 2025-08-14T21:44:55.7448972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7449375Z outputs = self.deberta( 2025-08-14T21:44:55.7449759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7450152Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7450553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7450968Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7451338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7451694Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7452100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7452572Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7453022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7453455Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7453842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7454192Z return self.act(input) 2025-08-14T21:44:55.7454302Z 2025-08-14T21:44:55.7454402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7454773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7455090Z return mod(**inputs) 2025-08-14T21:44:55.7455457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7455844Z outputs = self.deberta( 2025-08-14T21:44:55.7456216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7456667Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7457047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7457465Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7457827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7458183Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7458573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7459022Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7459511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7459915Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7460053Z 2025-08-14T21:44:55.7460153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7460508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7460826Z return mod(**inputs) 2025-08-14T21:44:55.7461190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7461584Z outputs = self.deberta( 2025-08-14T21:44:55.7461955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7462347Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7462734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7463164Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7463524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7463878Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7464268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7464686Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7465121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7465509Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7465910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7466449Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7466681Z 2025-08-14T21:44:55.7466790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7467129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7467443Z return mod(**inputs) 2025-08-14T21:44:55.7467818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7468226Z outputs = self.deberta( 2025-08-14T21:44:55.7468595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7468993Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7469385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7469787Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7470164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7470516Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7470925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7471338Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7471757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7472164Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7472567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7473068Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7473310Z 2025-08-14T21:44:55.7473415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7473790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7474123Z return mod(**inputs) 2025-08-14T21:44:55.7474531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7474954Z outputs = self.deberta( 2025-08-14T21:44:55.7475357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7475859Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7476310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7476786Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7477182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7477613Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7478026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7478457Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7478877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7479295Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7479704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7480225Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7480794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7481296Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7481486Z 2025-08-14T21:44:55.7481591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7481947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7482261Z return mod(**inputs) 2025-08-14T21:44:55.7482671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7483070Z outputs = self.deberta( 2025-08-14T21:44:55.7483444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7483848Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7484243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7484675Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7485038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7485416Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7485826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7486248Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7486663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7487072Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7487475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7488025Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7488287Z 2025-08-14T21:44:55.7488391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7488749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7489076Z return mod(**inputs) 2025-08-14T21:44:55.7489452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7489854Z outputs = self.deberta( 2025-08-14T21:44:55.7490239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7490640Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7491040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7491443Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7491803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7492144Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7492547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7492976Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7493421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7493846Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7494274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7494848Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7495099Z 2025-08-14T21:44:55.7495207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7495547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7495860Z return mod(**inputs) 2025-08-14T21:44:55.7496232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7496639Z outputs = self.deberta( 2025-08-14T21:44:55.7497003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7497392Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7497777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7498178Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7498557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7498917Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7499340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7499757Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7500245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7500638Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7501027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7501523Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7501766Z 2025-08-14T21:44:55.7501867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7502213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7502519Z return mod(**inputs) 2025-08-14T21:44:55.7502892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7503290Z outputs = self.deberta( 2025-08-14T21:44:55.7503673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7504066Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7504463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7504881Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7505250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7505596Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7505991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7506405Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7506809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7507207Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7507598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7508103Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7508820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7509332Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7509529Z 2025-08-14T21:44:55.7509635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7510002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7510319Z return mod(**inputs) 2025-08-14T21:44:55.7510756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7511162Z outputs = self.deberta( 2025-08-14T21:44:55.7511539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7511947Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7512351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7512916Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7513363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7514017Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7514545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7515066Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7515578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7516169Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7516678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7517183Z context_layer = torch.bmm( 2025-08-14T21:44:55.7536878Z 2025-08-14T21:44:55.7537154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7537563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7537920Z return mod(**inputs) 2025-08-14T21:44:55.7538356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7538780Z outputs = self.deberta( 2025-08-14T21:44:55.7539176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7539577Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7539969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7540389Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7540769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7541140Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7541552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7541984Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7542412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7542818Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7543249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7543952Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7544259Z 2025-08-14T21:44:55.7544382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7544747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7545082Z return mod(**inputs) 2025-08-14T21:44:55.7545478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7545887Z outputs = self.deberta( 2025-08-14T21:44:55.7546314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7546818Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7547225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7547648Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7548028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7548446Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7548885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7549305Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7549726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7550176Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7550639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7551074Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7551234Z 2025-08-14T21:44:55.7551348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7551743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7552069Z return mod(**inputs) 2025-08-14T21:44:55.7552461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7552867Z outputs = self.deberta( 2025-08-14T21:44:55.7553249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7553665Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7554103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7554557Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7554950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7555351Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7555884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7556378Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7556864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7557308Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7557467Z 2025-08-14T21:44:55.7557587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7557951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7558271Z return mod(**inputs) 2025-08-14T21:44:55.7558681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7559087Z outputs = self.deberta( 2025-08-14T21:44:55.7559475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7559871Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7560271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7560689Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7561061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7561415Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7561807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7562245Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7562687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7563111Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7563498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7563834Z return self.act(input) 2025-08-14T21:44:55.7563945Z 2025-08-14T21:44:55.7564051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7564415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7564746Z return mod(**inputs) 2025-08-14T21:44:55.7565146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7565576Z outputs = self.deberta( 2025-08-14T21:44:55.7565978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7566401Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7566796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7567205Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7567561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7567909Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7568305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7568756Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7569205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7569601Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7569747Z 2025-08-14T21:44:55.7569850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7570201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7570525Z return mod(**inputs) 2025-08-14T21:44:55.7570892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7571293Z outputs = self.deberta( 2025-08-14T21:44:55.7571667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7572057Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7572465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7572874Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7573238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7573587Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7573986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7574414Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7574854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7575253Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7575660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7576187Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7576421Z 2025-08-14T21:44:55.7576539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7576890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7577208Z return mod(**inputs) 2025-08-14T21:44:55.7577608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7578012Z outputs = self.deberta( 2025-08-14T21:44:55.7578385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7578777Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7579164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7579560Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7579917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7580269Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7580652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7581057Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7581463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7581858Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7582245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7582751Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7582987Z 2025-08-14T21:44:55.7583090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7583449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7583762Z return mod(**inputs) 2025-08-14T21:44:55.7584141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7584545Z outputs = self.deberta( 2025-08-14T21:44:55.7584930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7585329Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7585717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7586153Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7586521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7586875Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7587281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7587705Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7588121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7588536Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7588943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7589461Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7590011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7590527Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7590723Z 2025-08-14T21:44:55.7590830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7591206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7591531Z return mod(**inputs) 2025-08-14T21:44:55.7591908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7592307Z outputs = self.deberta( 2025-08-14T21:44:55.7592711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7593129Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7593551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7593989Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7594376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7594749Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7595176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7595623Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7596172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7596602Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7597042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7597590Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7597859Z 2025-08-14T21:44:55.7597964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7598331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7598661Z return mod(**inputs) 2025-08-14T21:44:55.7599047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7599444Z outputs = self.deberta( 2025-08-14T21:44:55.7599827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7600232Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7600645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7601060Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7601431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7601795Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7602200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7602627Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7603072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7603476Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7603869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7604423Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7604710Z 2025-08-14T21:44:55.7604849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7605212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7605593Z return mod(**inputs) 2025-08-14T21:44:55.7605997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7606417Z outputs = self.deberta( 2025-08-14T21:44:55.7606790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7607194Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7607586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7608001Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7608360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7608919Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7609384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7609806Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7610215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7610618Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7611017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7611527Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7611778Z 2025-08-14T21:44:55.7611883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7612241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7612563Z return mod(**inputs) 2025-08-14T21:44:55.7612934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7613331Z outputs = self.deberta( 2025-08-14T21:44:55.7613719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7614124Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7614516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7614976Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7615342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7615717Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7616118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7616552Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7616969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7617394Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7617795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7618330Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7618878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7619385Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7619575Z 2025-08-14T21:44:55.7619676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7620055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7620377Z return mod(**inputs) 2025-08-14T21:44:55.7620749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7621143Z outputs = self.deberta( 2025-08-14T21:44:55.7621516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7621905Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7622292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7622698Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7623055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7623408Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7623806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7624221Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7624626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7625035Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7625441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7625846Z context_layer = torch.bmm( 2025-08-14T21:44:55.7625966Z 2025-08-14T21:44:55.7626072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7626293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7626361Z return mod(**inputs) 2025-08-14T21:44:55.7626633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7626701Z outputs = self.deberta( 2025-08-14T21:44:55.7626961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7627045Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7627327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7627438Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7627691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7627770Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7628041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7628135Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7628416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7628501Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7628768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7628966Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7628970Z 2025-08-14T21:44:55.7629090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7629290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7629366Z return mod(**inputs) 2025-08-14T21:44:55.7629651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7629722Z outputs = self.deberta( 2025-08-14T21:44:55.7629996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7630069Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7630344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7630431Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7630652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7630742Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7631025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7631133Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7631415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7631541Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7631833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7631922Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7631926Z 2025-08-14T21:44:55.7632043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7632254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7632324Z return mod(**inputs) 2025-08-14T21:44:55.7632621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7632695Z outputs = self.deberta( 2025-08-14T21:44:55.7632977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7633061Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7633345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7633442Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7633697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7633780Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7634062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7634197Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7634481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7634595Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7634599Z 2025-08-14T21:44:55.7634710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7634920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7634998Z return mod(**inputs) 2025-08-14T21:44:55.7635297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7635369Z outputs = self.deberta( 2025-08-14T21:44:55.7635741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7635835Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7636154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7636253Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7636490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7636584Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7636876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7637015Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7637314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7637434Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7637668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7637744Z return self.act(input) 2025-08-14T21:44:55.7637747Z 2025-08-14T21:44:55.7637860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7638081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7638150Z return mod(**inputs) 2025-08-14T21:44:55.7638447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7638523Z outputs = self.deberta( 2025-08-14T21:44:55.7638803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7638891Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7639170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7639268Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7639496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7639583Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7639872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7640012Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7640322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7640419Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7640424Z 2025-08-14T21:44:55.7640532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7640747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7640817Z return mod(**inputs) 2025-08-14T21:44:55.7641102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7641199Z outputs = self.deberta( 2025-08-14T21:44:55.7641482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7641566Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7641849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7641939Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7642195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7642279Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7642577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7642685Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7642966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7643053Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7643332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7643532Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7643536Z 2025-08-14T21:44:55.7643653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7643861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7643935Z return mod(**inputs) 2025-08-14T21:44:55.7644222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7644295Z outputs = self.deberta( 2025-08-14T21:44:55.7644580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7644656Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7644933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7645031Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7645262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7645352Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7645638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7645727Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7645992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7646068Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7646333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7646528Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7646531Z 2025-08-14T21:44:55.7646631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7646829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7646892Z return mod(**inputs) 2025-08-14T21:44:55.7647159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7647226Z outputs = self.deberta( 2025-08-14T21:44:55.7647502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7647580Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7647835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7647919Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7648136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7648224Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7648487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7648594Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7648853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7648939Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7649195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7649384Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7649683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7649811Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7649815Z 2025-08-14T21:44:55.7649924Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7650116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7650181Z return mod(**inputs) 2025-08-14T21:44:55.7650452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7650517Z outputs = self.deberta( 2025-08-14T21:44:55.7650782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7650854Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7651110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7651200Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7651411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7651495Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7651753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7651844Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7652106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7652182Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7652459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7652674Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7652678Z 2025-08-14T21:44:55.7652777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7652980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7653044Z return mod(**inputs) 2025-08-14T21:44:55.7653303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7653395Z outputs = self.deberta( 2025-08-14T21:44:55.7653653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7653731Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7653989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7654070Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7654299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7654378Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7654652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7654752Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7655008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7655090Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7655353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7655552Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7655556Z 2025-08-14T21:44:55.7655662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7655848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7655918Z return mod(**inputs) 2025-08-14T21:44:55.7656171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7656236Z outputs = self.deberta( 2025-08-14T21:44:55.7656492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7656565Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7656823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7656905Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7657112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7657193Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7657443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7657537Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7657787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7657861Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7658115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7658311Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7658315Z 2025-08-14T21:44:55.7658419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7658606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7658667Z return mod(**inputs) 2025-08-14T21:44:55.7658930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7658994Z outputs = self.deberta( 2025-08-14T21:44:55.7659260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7659337Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7659586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7659674Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7659879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7659968Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7660226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7661275Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7661550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7661635Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7661897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7662095Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7662404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7662538Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7662549Z 2025-08-14T21:44:55.7662654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7662854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7662929Z return mod(**inputs) 2025-08-14T21:44:55.7663202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7663270Z outputs = self.deberta( 2025-08-14T21:44:55.7663544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7663619Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7663891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7663977Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7664194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7664279Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7664551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7664639Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7664905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7664987Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7665260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7665329Z context_layer = torch.bmm( 2025-08-14T21:44:55.7665333Z 2025-08-14T21:44:55.7665431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7665623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7665687Z return mod(**inputs) 2025-08-14T21:44:55.7665953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7666040Z outputs = self.deberta( 2025-08-14T21:44:55.7666331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7666407Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7666661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7666743Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7666994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7667069Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7667344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7667432Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7667685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7667762Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7668016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7668204Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7668207Z 2025-08-14T21:44:55.7668307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7668495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7668564Z return mod(**inputs) 2025-08-14T21:44:55.7668822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7668888Z outputs = self.deberta( 2025-08-14T21:44:55.7669210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7669280Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7669546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7669630Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7669844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7669928Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7670189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7670287Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7670549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7670664Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7670934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7671032Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7671036Z 2025-08-14T21:44:55.7671135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7671334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7671398Z return mod(**inputs) 2025-08-14T21:44:55.7671673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7671743Z outputs = self.deberta( 2025-08-14T21:44:55.7672017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7672119Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7672409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7672503Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7672737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7672820Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7673131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7673259Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7673562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7673658Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7673661Z 2025-08-14T21:44:55.7673768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7673982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7674049Z return mod(**inputs) 2025-08-14T21:44:55.7674336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7674417Z outputs = self.deberta( 2025-08-14T21:44:55.7674700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7674785Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7675074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7675163Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7675398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7675481Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7675839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7675983Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7676281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7676407Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7676641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7676718Z return self.act(input) 2025-08-14T21:44:55.7676723Z 2025-08-14T21:44:55.7676844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7677058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7677138Z return mod(**inputs) 2025-08-14T21:44:55.7677416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7677502Z outputs = self.deberta( 2025-08-14T21:44:55.7677764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7677836Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7678094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7678184Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7678393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7678495Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7678751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7678879Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7679146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7679224Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7679244Z 2025-08-14T21:44:55.7679351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7679539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7679619Z return mod(**inputs) 2025-08-14T21:44:55.7679888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7679955Z outputs = self.deberta( 2025-08-14T21:44:55.7680209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7680285Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7680540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7680627Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7680837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7680911Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7681173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7681263Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7681526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7681599Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7681853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7682042Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7682046Z 2025-08-14T21:44:55.7682145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7682332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7682402Z return mod(**inputs) 2025-08-14T21:44:55.7682662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7682740Z outputs = self.deberta( 2025-08-14T21:44:55.7683003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7683075Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7683345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7683446Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7683672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7683751Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7684022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7684127Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7684412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7684535Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7684827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7685006Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7685010Z 2025-08-14T21:44:55.7685117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7685332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7685398Z return mod(**inputs) 2025-08-14T21:44:55.7685683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7685751Z outputs = self.deberta( 2025-08-14T21:44:55.7686019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7686092Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7686357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7686450Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7686674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7686752Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7687021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7687110Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7687372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7687446Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7687707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7687903Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7688214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7688359Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7688362Z 2025-08-14T21:44:55.7688467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7688666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7688739Z return mod(**inputs) 2025-08-14T21:44:55.7689008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7689083Z outputs = self.deberta( 2025-08-14T21:44:55.7689350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7689441Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7689723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7689805Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7690020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7690106Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7690362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7690473Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7690735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7690811Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7691087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7691315Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7691320Z 2025-08-14T21:44:55.7691436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7691659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7691730Z return mod(**inputs) 2025-08-14T21:44:55.7692035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7692105Z outputs = self.deberta( 2025-08-14T21:44:55.7692368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7692447Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7692711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7692803Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7693021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7693098Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7693375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7693469Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7693738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7693814Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7694079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7694296Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7694299Z 2025-08-14T21:44:55.7694401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7694607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7694673Z return mod(**inputs) 2025-08-14T21:44:55.7694942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7695017Z outputs = self.deberta( 2025-08-14T21:44:55.7695278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7695349Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7695620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7695724Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7695953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7696036Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7696306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7696410Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7696692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7696776Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7697037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7697228Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7697232Z 2025-08-14T21:44:55.7697355Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7697552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7697617Z return mod(**inputs) 2025-08-14T21:44:55.7697905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7697974Z outputs = self.deberta( 2025-08-14T21:44:55.7698248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7698319Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7698581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7698673Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7698889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7698976Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7699237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7699330Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7699600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7699677Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7699941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7700136Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7700443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7700580Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7700584Z 2025-08-14T21:44:55.7700687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7700880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7700952Z return mod(**inputs) 2025-08-14T21:44:55.7701222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7701297Z outputs = self.deberta( 2025-08-14T21:44:55.7701563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7701653Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7701926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7702012Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7702230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7702317Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7702584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7702700Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7702965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7703040Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7703313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7703383Z context_layer = torch.bmm( 2025-08-14T21:44:55.7703387Z 2025-08-14T21:44:55.7703514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7703711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7703792Z return mod(**inputs) 2025-08-14T21:44:55.7704070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7704141Z outputs = self.deberta( 2025-08-14T21:44:55.7704408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7704487Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7704756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7704847Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7705066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7705143Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7705420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7705509Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7705784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7705859Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7706122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7706319Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7706322Z 2025-08-14T21:44:55.7706424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7706627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7706691Z return mod(**inputs) 2025-08-14T21:44:55.7706964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7707041Z outputs = self.deberta( 2025-08-14T21:44:55.7707308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7707380Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7707666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7707766Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7707983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7708059Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7708318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7708416Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7708787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7708967Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7709232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7709315Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7709320Z 2025-08-14T21:44:55.7709427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7709644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7709718Z return mod(**inputs) 2025-08-14T21:44:55.7710041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7710115Z outputs = self.deberta( 2025-08-14T21:44:55.7710402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7710482Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7710762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7710860Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7711092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7711174Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7711462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7711589Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7711876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7711965Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7711969Z 2025-08-14T21:44:55.7712076Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7712289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7712358Z return mod(**inputs) 2025-08-14T21:44:55.7712649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7712722Z outputs = self.deberta( 2025-08-14T21:44:55.7713004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7713088Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7713368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7713466Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7713693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7713774Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7714062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7714229Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7714513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7714638Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7714860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7714941Z return self.act(input) 2025-08-14T21:44:55.7714945Z 2025-08-14T21:44:55.7715070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7715276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7715352Z return mod(**inputs) 2025-08-14T21:44:55.7715636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7715757Z outputs = self.deberta( 2025-08-14T21:44:55.7716064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7716165Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7716470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7716581Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7716820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7716915Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7717204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7717341Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7717602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7717683Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7717688Z 2025-08-14T21:44:55.7717797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7717988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7718063Z return mod(**inputs) 2025-08-14T21:44:55.7718324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7718393Z outputs = self.deberta( 2025-08-14T21:44:55.7718657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7718728Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7718990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7719080Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7719291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7719374Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7719632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7719723Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7719988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7720064Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7720321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7720532Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7720535Z 2025-08-14T21:44:55.7720636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7720833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7720902Z return mod(**inputs) 2025-08-14T21:44:55.7721164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7721255Z outputs = self.deberta( 2025-08-14T21:44:55.7721518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7721595Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7721856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7721941Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7722176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7722253Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7722531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7722622Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7722879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7722961Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7723216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7723392Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7723403Z 2025-08-14T21:44:55.7723507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7723704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7723774Z return mod(**inputs) 2025-08-14T21:44:55.7724047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7724114Z outputs = self.deberta( 2025-08-14T21:44:55.7724388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7724461Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7724732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7724826Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7725035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7725120Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7725375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7725464Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7725728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7725806Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7726067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7726247Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7726556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7726696Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7726700Z 2025-08-14T21:44:55.7726799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7726996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7727059Z return mod(**inputs) 2025-08-14T21:44:55.7727335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7727408Z outputs = self.deberta( 2025-08-14T21:44:55.7727667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7727746Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7728008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7728104Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7728318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7728409Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7728661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7728755Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7729006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7729084Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7729336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7729536Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7729540Z 2025-08-14T21:44:55.7729645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7729831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7729901Z return mod(**inputs) 2025-08-14T21:44:55.7730155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7730219Z outputs = self.deberta( 2025-08-14T21:44:55.7730475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7730544Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7730794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7730879Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7731082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7731163Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7731413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7731500Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7731757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7731828Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7732090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7732320Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7732324Z 2025-08-14T21:44:55.7732426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7732622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7732685Z return mod(**inputs) 2025-08-14T21:44:55.7732947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7733039Z outputs = self.deberta( 2025-08-14T21:44:55.7733298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7733378Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7733641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7733728Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7733980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7734057Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7734331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7734421Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7734669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7734749Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7735000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7735181Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7735193Z 2025-08-14T21:44:55.7735291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7735476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7735545Z return mod(**inputs) 2025-08-14T21:44:55.7735799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7735863Z outputs = self.deberta( 2025-08-14T21:44:55.7736124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7736192Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7736451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7736533Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7736735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7736817Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7737069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7737156Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7737411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7737485Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7737740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7737939Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7738231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7738364Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7738368Z 2025-08-14T21:44:55.7738464Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7738658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7738721Z return mod(**inputs) 2025-08-14T21:44:55.7738995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7739068Z outputs = self.deberta( 2025-08-14T21:44:55.7739320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7739397Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7739649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7739743Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7739957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7740046Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7740296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7740391Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7740641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7740721Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7740971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7741040Z context_layer = torch.bmm( 2025-08-14T21:44:55.7741045Z 2025-08-14T21:44:55.7741150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7741339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7741408Z return mod(**inputs) 2025-08-14T21:44:55.7741662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7741728Z outputs = self.deberta( 2025-08-14T21:44:55.7741985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7742053Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7742305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7742392Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7742598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7742679Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7742931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7743017Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7743278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7743350Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7743608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7743808Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7743812Z 2025-08-14T21:44:55.7743908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7744097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7744158Z return mod(**inputs) 2025-08-14T21:44:55.7744410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7744498Z outputs = self.deberta( 2025-08-14T21:44:55.7744752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7744826Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7745081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7745163Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7745393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7745468Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7745741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7745831Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7746080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7746199Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7746447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7746528Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7746538Z 2025-08-14T21:44:55.7746633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7746817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7746886Z return mod(**inputs) 2025-08-14T21:44:55.7747141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7747206Z outputs = self.deberta( 2025-08-14T21:44:55.7747461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7747528Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7747778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7747856Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7748058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7748134Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7748387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7748501Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7748759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7748839Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7748842Z 2025-08-14T21:44:55.7748944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7749135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7749222Z return mod(**inputs) 2025-08-14T21:44:55.7749498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7749566Z outputs = self.deberta( 2025-08-14T21:44:55.7749838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7749910Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7750175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7750284Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7750505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7750582Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7750859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7750979Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7751283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7751406Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7751658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7751743Z return self.act(input) 2025-08-14T21:44:55.7751747Z 2025-08-14T21:44:55.7751856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7752075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7752143Z return mod(**inputs) 2025-08-14T21:44:55.7752441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7752522Z outputs = self.deberta( 2025-08-14T21:44:55.7752812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7752890Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7753189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7753281Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7753521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7753604Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7753894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7754042Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7754332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7754428Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7754432Z 2025-08-14T21:44:55.7754538Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7754751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7754829Z return mod(**inputs) 2025-08-14T21:44:55.7755126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7755199Z outputs = self.deberta( 2025-08-14T21:44:55.7755498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7755575Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7756120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7756218Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7756465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7756561Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7756859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7756968Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7757295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7757373Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7757647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7757835Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7757839Z 2025-08-14T21:44:55.7757966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7758165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7758229Z return mod(**inputs) 2025-08-14T21:44:55.7758526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7758597Z outputs = self.deberta( 2025-08-14T21:44:55.7758861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7758937Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7759200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7759291Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7759513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7759589Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7759860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7759953Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7760216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7760301Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7760565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7760750Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7760754Z 2025-08-14T21:44:55.7760855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7761049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7761122Z return mod(**inputs) 2025-08-14T21:44:55.7761393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7761468Z outputs = self.deberta( 2025-08-14T21:44:55.7761734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7761805Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7762079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7762181Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7762396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7762485Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7762749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7762849Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7763115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7763227Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7763503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7763692Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7764008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7764161Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7764166Z 2025-08-14T21:44:55.7764268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7764486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7764552Z return mod(**inputs) 2025-08-14T21:44:55.7764830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7764899Z outputs = self.deberta( 2025-08-14T21:44:55.7765163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7765244Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7765513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7765598Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7765823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7765903Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7766188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7766278Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7766537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7766620Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7766879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7767094Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7767098Z 2025-08-14T21:44:55.7767198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7767388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7767458Z return mod(**inputs) 2025-08-14T21:44:55.7767722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7767790Z outputs = self.deberta( 2025-08-14T21:44:55.7768055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7768124Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7768412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7768497Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7768711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7768799Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7769059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7769173Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7769427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7769501Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7769761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7769962Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7769966Z 2025-08-14T21:44:55.7770086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7770276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7770355Z return mod(**inputs) 2025-08-14T21:44:55.7770632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7770699Z outputs = self.deberta( 2025-08-14T21:44:55.7770959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7771036Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7771295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7771384Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7771598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7771675Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7771944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7772031Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7772301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7772374Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7772635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7772827Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7772831Z 2025-08-14T21:44:55.7772931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7773122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7773190Z return mod(**inputs) 2025-08-14T21:44:55.7773454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7773529Z outputs = self.deberta( 2025-08-14T21:44:55.7773793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7773862Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7774130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7774228Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7774450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7774527Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7774788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7774885Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7775146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7775239Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7775505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7775688Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7776013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7776144Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7776148Z 2025-08-14T21:44:55.7776249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7776465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7776532Z return mod(**inputs) 2025-08-14T21:44:55.7776815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7776884Z outputs = self.deberta( 2025-08-14T21:44:55.7777177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7777259Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7777546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7777628Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7777871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7777952Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7778246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7778339Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7778622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7778706Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7778997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7779077Z context_layer = torch.bmm( 2025-08-14T21:44:55.7779082Z 2025-08-14T21:44:55.7779186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7779402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7779477Z return mod(**inputs) 2025-08-14T21:44:55.7779752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7779823Z outputs = self.deberta( 2025-08-14T21:44:55.7780101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7780173Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7780467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7780550Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7780766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7780851Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7781116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7781214Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7781495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7781571Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7781841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7782029Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7782033Z 2025-08-14T21:44:55.7782156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7782356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7782422Z return mod(**inputs) 2025-08-14T21:44:55.7782715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7782788Z outputs = self.deberta( 2025-08-14T21:44:55.7783057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7783137Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7783423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7783521Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7783763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7783858Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7784136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7784226Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7784503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7784619Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7784888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7784980Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7784984Z 2025-08-14T21:44:55.7785085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7785283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7785360Z return mod(**inputs) 2025-08-14T21:44:55.7785635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7785712Z outputs = self.deberta( 2025-08-14T21:44:55.7785980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7786054Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7786330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7786433Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7786654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7786730Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7786994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7787122Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7787385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7787486Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7787490Z 2025-08-14T21:44:55.7787598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7787791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7787867Z return mod(**inputs) 2025-08-14T21:44:55.7788139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7788206Z outputs = self.deberta( 2025-08-14T21:44:55.7788496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7788570Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7788858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7788950Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7789180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7789270Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7789548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7789674Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7789961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7790078Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7790308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7790383Z return self.act(input) 2025-08-14T21:44:55.7790387Z 2025-08-14T21:44:55.7790496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7790711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7790780Z return mod(**inputs) 2025-08-14T21:44:55.7791080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7791155Z outputs = self.deberta( 2025-08-14T21:44:55.7791441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7791522Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7791812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7791902Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7792139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7792224Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7792517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7792659Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7792966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7793061Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7793065Z 2025-08-14T21:44:55.7793173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7793388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7793459Z return mod(**inputs) 2025-08-14T21:44:55.7793753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7793850Z outputs = self.deberta( 2025-08-14T21:44:55.7794139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7794216Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7794514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7794603Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7794864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7794949Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7795256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7795368Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7795654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7795818Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7796115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7796320Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7796324Z 2025-08-14T21:44:55.7796444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7796660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7796732Z return mod(**inputs) 2025-08-14T21:44:55.7797038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7797113Z outputs = self.deberta( 2025-08-14T21:44:55.7797403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7797478Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7797744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7797839Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7798057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7798143Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7798408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7798501Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7798777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7798857Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7799123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7799342Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7799346Z 2025-08-14T21:44:55.7799453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7799668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7799738Z return mod(**inputs) 2025-08-14T21:44:55.7800026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7800105Z outputs = self.deberta( 2025-08-14T21:44:55.7800408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7800490Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7800770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7800858Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7801100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7801190Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7801457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7801574Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7801841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7801925Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7802194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7802390Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7802724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7802868Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7802872Z 2025-08-14T21:44:55.7802990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7803198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7803268Z return mod(**inputs) 2025-08-14T21:44:55.7803567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7803636Z outputs = self.deberta( 2025-08-14T21:44:55.7803925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7804003Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7804281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7804378Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7804612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7804695Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7804986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7805083Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7805372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7805452Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7805750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7805979Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7805983Z 2025-08-14T21:44:55.7806091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7806308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7806377Z return mod(**inputs) 2025-08-14T21:44:55.7806663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7806762Z outputs = self.deberta( 2025-08-14T21:44:55.7807048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7807123Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7807420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7807510Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7807770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7807855Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7808160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7808267Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7808548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7808634Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7809116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7809344Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7809349Z 2025-08-14T21:44:55.7809464Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7809675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7809744Z return mod(**inputs) 2025-08-14T21:44:55.7810035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7810109Z outputs = self.deberta( 2025-08-14T21:44:55.7810395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7810469Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7810752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7810851Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7811080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7811170Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7811453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7811549Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7811836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7811917Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7812198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7812451Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7812455Z 2025-08-14T21:44:55.7812566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7812782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7812850Z return mod(**inputs) 2025-08-14T21:44:55.7813136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7813245Z outputs = self.deberta( 2025-08-14T21:44:55.7813530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7813609Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7813874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7813961Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7814210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7814289Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7814579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7814678Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7814940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7815023Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7815284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7815475Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7815789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7815922Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7815925Z 2025-08-14T21:44:55.7816037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7816234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7816301Z return mod(**inputs) 2025-08-14T21:44:55.7816580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7816648Z outputs = self.deberta( 2025-08-14T21:44:55.7816922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7816997Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7817263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7817355Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7817570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7817648Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7817918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7818008Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7818278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7818354Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7818641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7818716Z context_layer = torch.bmm( 2025-08-14T21:44:55.7818721Z 2025-08-14T21:44:55.7818823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7819025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7819092Z return mod(**inputs) 2025-08-14T21:44:55.7819359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7819452Z outputs = self.deberta( 2025-08-14T21:44:55.7819724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7819795Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7820077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7820162Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7820416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7820495Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7820777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7820880Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7821142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7821223Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7821485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7821672Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7821675Z 2025-08-14T21:44:55.7821785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7821983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7822047Z return mod(**inputs) 2025-08-14T21:44:55.7822322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7822390Z outputs = self.deberta( 2025-08-14T21:44:55.7822663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7822733Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7822999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7823091Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7823310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7823393Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7823661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7823748Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7824015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7824131Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7824396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7824516Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7824519Z 2025-08-14T21:44:55.7824619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7824820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7824886Z return mod(**inputs) 2025-08-14T21:44:55.7825160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7825238Z outputs = self.deberta( 2025-08-14T21:44:55.7825504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7825600Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7825873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7825960Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7826200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7826278Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7826559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7826698Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7826971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7827057Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7827060Z 2025-08-14T21:44:55.7827157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7827348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7827421Z return mod(**inputs) 2025-08-14T21:44:55.7827684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7827756Z outputs = self.deberta( 2025-08-14T21:44:55.7828016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7828086Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7828359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7828444Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7828661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7828745Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7829012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7829137Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7829407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7829515Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7829734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7829805Z return self.act(input) 2025-08-14T21:44:55.7829808Z 2025-08-14T21:44:55.7829920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7830116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7830181Z return mod(**inputs) 2025-08-14T21:44:55.7830457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7830545Z outputs = self.deberta( 2025-08-14T21:44:55.7830817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7830897Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7831168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7831266Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7831498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7831598Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7831885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7832023Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7832309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7832412Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7832416Z 2025-08-14T21:44:55.7832525Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7832756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7832828Z return mod(**inputs) 2025-08-14T21:44:55.7833112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7833193Z outputs = self.deberta( 2025-08-14T21:44:55.7833474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7833557Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7833842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7833931Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7834169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7834251Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7834538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7834634Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7834915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7835004Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7835287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7835488Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7835499Z 2025-08-14T21:44:55.7835608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7835895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7835978Z return mod(**inputs) 2025-08-14T21:44:55.7836276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7836351Z outputs = self.deberta( 2025-08-14T21:44:55.7836655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7836734Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7837044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7837164Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7837390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7837475Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7837733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7837823Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7838103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7838179Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7838461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7838644Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7838647Z 2025-08-14T21:44:55.7838752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7838974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7839043Z return mod(**inputs) 2025-08-14T21:44:55.7839347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7839417Z outputs = self.deberta( 2025-08-14T21:44:55.7839687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7839767Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7840035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7840126Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7840351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7840427Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7840693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7840782Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7841040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7841124Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7841382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7841570Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7841872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7842001Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7842005Z 2025-08-14T21:44:55.7842111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7842305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7842375Z return mod(**inputs) 2025-08-14T21:44:55.7842640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7842708Z outputs = self.deberta( 2025-08-14T21:44:55.7842979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7843072Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7843340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7843432Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7843655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7843739Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7844017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7844123Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7844388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7844462Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7844728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7844950Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7844954Z 2025-08-14T21:44:55.7845055Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7845284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7845352Z return mod(**inputs) 2025-08-14T21:44:55.7845637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7845710Z outputs = self.deberta( 2025-08-14T21:44:55.7845972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7846052Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7846319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7846403Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7846629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7846707Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7846980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7847072Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7847338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7847419Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7847687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7847908Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7847912Z 2025-08-14T21:44:55.7848018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7848221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7848296Z return mod(**inputs) 2025-08-14T21:44:55.7848570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7848640Z outputs = self.deberta( 2025-08-14T21:44:55.7848933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7849006Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7849306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7849392Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7849613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7849699Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7849974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7850075Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7850366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7850443Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7850722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7850919Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7850924Z 2025-08-14T21:44:55.7851044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7851252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7851319Z return mod(**inputs) 2025-08-14T21:44:55.7851623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7851694Z outputs = self.deberta( 2025-08-14T21:44:55.7851968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7852051Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7852352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7852445Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7852667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7852747Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7853024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7853117Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7853388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7853476Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7853757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7853950Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7854246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7854375Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7854379Z 2025-08-14T21:44:55.7854483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7854675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7854748Z return mod(**inputs) 2025-08-14T21:44:55.7855010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7855075Z outputs = self.deberta( 2025-08-14T21:44:55.7855341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7855434Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7855698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7855779Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7855988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7856072Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7856332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7856437Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7856700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7856773Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7857039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7857109Z context_layer = torch.bmm( 2025-08-14T21:44:55.7857129Z 2025-08-14T21:44:55.7857231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7857434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7857513Z return mod(**inputs) 2025-08-14T21:44:55.7857779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7857846Z outputs = self.deberta( 2025-08-14T21:44:55.7858103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7858179Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7858437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7858517Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7858743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7858821Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7859095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7859187Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7859460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7859542Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7859798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7859989Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7859993Z 2025-08-14T21:44:55.7860093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7860281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7860350Z return mod(**inputs) 2025-08-14T21:44:55.7860613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7860680Z outputs = self.deberta( 2025-08-14T21:44:55.7860943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7861012Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7861278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7861392Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7861604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7861687Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7861943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7862041Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7862298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7862430Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7862694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7862775Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7862778Z 2025-08-14T21:44:55.7862875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7863085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7863151Z return mod(**inputs) 2025-08-14T21:44:55.7863438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7863506Z outputs = self.deberta( 2025-08-14T21:44:55.7863762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7863842Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7864104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7864193Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7864404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7864480Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7864745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7864863Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7865119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7865206Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7865210Z 2025-08-14T21:44:55.7865306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7865506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7865571Z return mod(**inputs) 2025-08-14T21:44:55.7865833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7865905Z outputs = self.deberta( 2025-08-14T21:44:55.7866166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7866242Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7866502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7866584Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7866802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7866877Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7867135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7867279Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7867536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7867649Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7867854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7867922Z return self.act(input) 2025-08-14T21:44:55.7867925Z 2025-08-14T21:44:55.7868048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7868242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7868314Z return mod(**inputs) 2025-08-14T21:44:55.7868576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7868643Z outputs = self.deberta( 2025-08-14T21:44:55.7868938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7869017Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7869299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7869411Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7869643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7869734Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7870013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7870151Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7870438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7870526Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7870529Z 2025-08-14T21:44:55.7870640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7870847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7870913Z return mod(**inputs) 2025-08-14T21:44:55.7871203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7871277Z outputs = self.deberta( 2025-08-14T21:44:55.7871555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7871637Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7871919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7872014Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7872246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7872329Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7872616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7872715Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7873000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7873080Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7873362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7873589Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7873593Z 2025-08-14T21:44:55.7873704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7873913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7873991Z return mod(**inputs) 2025-08-14T21:44:55.7874286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7874385Z outputs = self.deberta( 2025-08-14T21:44:55.7874671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7874748Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7875042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7875134Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7875393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7875480Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7875862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7875979Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7876268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7876353Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7876650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7876849Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7876853Z 2025-08-14T21:44:55.7876970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7877187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7877263Z return mod(**inputs) 2025-08-14T21:44:55.7877536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7877603Z outputs = self.deberta( 2025-08-14T21:44:55.7877869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7877939Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7878203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7878297Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7878514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7878595Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7878869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7878961Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7879235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7879315Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7879581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7879773Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7880105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7880246Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7880250Z 2025-08-14T21:44:55.7880352Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7880551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7880632Z return mod(**inputs) 2025-08-14T21:44:55.7880919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7880993Z outputs = self.deberta( 2025-08-14T21:44:55.7881252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7881322Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7881589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7881690Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7881901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7881999Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7882256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7882351Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7882616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7882691Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7882963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7883187Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7883191Z 2025-08-14T21:44:55.7883308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7883515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7883584Z return mod(**inputs) 2025-08-14T21:44:55.7883875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7883948Z outputs = self.deberta( 2025-08-14T21:44:55.7884226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7884309Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7884589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7884686Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7884917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7885009Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7885275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7885364Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7885639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7885718Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7886008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7886256Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7886261Z 2025-08-14T21:44:55.7886369Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7886584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7886655Z return mod(**inputs) 2025-08-14T21:44:55.7886941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7887042Z outputs = self.deberta( 2025-08-14T21:44:55.7887323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7887396Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7887685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7887773Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7888027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7888110Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7888409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7888515Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7888796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7888875Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7889158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7889360Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7889364Z 2025-08-14T21:44:55.7889480Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7889688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7889756Z return mod(**inputs) 2025-08-14T21:44:55.7890048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7890122Z outputs = self.deberta( 2025-08-14T21:44:55.7890412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7890489Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7890768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7890866Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7891095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7891184Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7891462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7891560Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7891844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7891924Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7892203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7892434Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7892757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7892906Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7892909Z 2025-08-14T21:44:55.7893008Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7893199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7893294Z return mod(**inputs) 2025-08-14T21:44:55.7893566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7893642Z outputs = self.deberta( 2025-08-14T21:44:55.7893919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7893993Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7894296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7894380Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7894592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7894701Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7894970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7895072Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7895338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7895427Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7895696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7895765Z context_layer = torch.bmm( 2025-08-14T21:44:55.7895770Z 2025-08-14T21:44:55.7895877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7896069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7896136Z return mod(**inputs) 2025-08-14T21:44:55.7896406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7896477Z outputs = self.deberta( 2025-08-14T21:44:55.7896736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7896816Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7897080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7897172Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7897386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7897463Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7897736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7897827Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7898096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7898171Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7898435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7898645Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7898648Z 2025-08-14T21:44:55.7898748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7898945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7899010Z return mod(**inputs) 2025-08-14T21:44:55.7899282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7899375Z outputs = self.deberta( 2025-08-14T21:44:55.7899647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7899719Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7899999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7900087Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7900578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7900657Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7900933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7901033Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7901289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7901406Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7901673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7901757Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7901761Z 2025-08-14T21:44:55.7901870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7902062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7902124Z return mod(**inputs) 2025-08-14T21:44:55.7902398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7902464Z outputs = self.deberta( 2025-08-14T21:44:55.7902732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7902806Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7903070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7903164Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7903382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7903460Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7903733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7903851Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7904120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7904203Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7904206Z 2025-08-14T21:44:55.7904305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7904518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7904603Z return mod(**inputs) 2025-08-14T21:44:55.7904871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7904939Z outputs = self.deberta( 2025-08-14T21:44:55.7905197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7905274Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7905531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7905631Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7905848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7905923Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7906185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7906300Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7906571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7906689Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7906908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7906985Z return self.act(input) 2025-08-14T21:44:55.7906989Z 2025-08-14T21:44:55.7907087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7907277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7907348Z return mod(**inputs) 2025-08-14T21:44:55.7907614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7907682Z outputs = self.deberta( 2025-08-14T21:44:55.7907950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7908018Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7908285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7908366Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7908578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7908840Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7909109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7909245Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7909504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7909588Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7909591Z 2025-08-14T21:44:55.7909698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7909893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7909960Z return mod(**inputs) 2025-08-14T21:44:55.7910237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7910307Z outputs = self.deberta( 2025-08-14T21:44:55.7910580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7910652Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7910951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7911042Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7911258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7911344Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7911611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7911728Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7911999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7912074Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7912337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7912532Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7912535Z 2025-08-14T21:44:55.7912660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7912873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7912965Z return mod(**inputs) 2025-08-14T21:44:55.7913255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7913337Z outputs = self.deberta( 2025-08-14T21:44:55.7913617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7913700Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7913983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7914073Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7914313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7914395Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7914677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7914801Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7915087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7915175Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7915458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7915656Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7915660Z 2025-08-14T21:44:55.7915834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7916056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7916132Z return mod(**inputs) 2025-08-14T21:44:55.7916432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7916506Z outputs = self.deberta( 2025-08-14T21:44:55.7916806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7916885Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7917175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7917293Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7917515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7917603Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7917869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7917961Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7918237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7918332Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7918607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7918791Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7919096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7919253Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7919257Z 2025-08-14T21:44:55.7919359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7919579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7919645Z return mod(**inputs) 2025-08-14T21:44:55.7919923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7919999Z outputs = self.deberta( 2025-08-14T21:44:55.7920266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7920338Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7920612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7920696Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7920919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7920998Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7921263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7921363Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7921632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7921714Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7921980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7922189Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7922192Z 2025-08-14T21:44:55.7922301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7922497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7922568Z return mod(**inputs) 2025-08-14T21:44:55.7922838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7922907Z outputs = self.deberta( 2025-08-14T21:44:55.7923180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7923251Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7923534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7923627Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7923847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7923934Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7924203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7924312Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7924586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7924657Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7924915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7925111Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7925137Z 2025-08-14T21:44:55.7925237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7925429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7925505Z return mod(**inputs) 2025-08-14T21:44:55.7925765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7925838Z outputs = self.deberta( 2025-08-14T21:44:55.7926092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7926167Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7926423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7926503Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7926718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7926794Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7927056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7927142Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7927396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7927474Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7927726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7927909Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7927919Z 2025-08-14T21:44:55.7928018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7928204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7928272Z return mod(**inputs) 2025-08-14T21:44:55.7928529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7928594Z outputs = self.deberta( 2025-08-14T21:44:55.7928859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7928930Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7929196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7929298Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7929508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7929592Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7929849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7929937Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7930204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7930296Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7930567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7930747Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7931052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7931186Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7931190Z 2025-08-14T21:44:55.7931301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7931494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7931558Z return mod(**inputs) 2025-08-14T21:44:55.7931819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7931893Z outputs = self.deberta( 2025-08-14T21:44:55.7932152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7932230Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7932489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7932570Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7932790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7932868Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7933131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7933230Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7933495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7933580Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7933845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.7933916Z context_layer = torch.bmm( 2025-08-14T21:44:55.7933921Z 2025-08-14T21:44:55.7934029Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7934224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7934307Z return mod(**inputs) 2025-08-14T21:44:55.7934569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7934637Z outputs = self.deberta( 2025-08-14T21:44:55.7934902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7934972Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7935244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7935334Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7935553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7935634Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7935883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7935967Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7936250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7936322Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7936585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.7936770Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.7936773Z 2025-08-14T21:44:55.7936888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7937086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7937150Z return mod(**inputs) 2025-08-14T21:44:55.7937439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7937517Z outputs = self.deberta( 2025-08-14T21:44:55.7937779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7937857Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7938127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7938209Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7938430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7938506Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7938767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7938856Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7939113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.7939237Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.7939503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.7939588Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7939598Z 2025-08-14T21:44:55.7939701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7939901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7939970Z return mod(**inputs) 2025-08-14T21:44:55.7940234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7940300Z outputs = self.deberta( 2025-08-14T21:44:55.7940565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7940637Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7940903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7941008Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7941228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7941316Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7941587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7941711Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7941990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.7942095Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7942098Z 2025-08-14T21:44:55.7942206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7942398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7942466Z return mod(**inputs) 2025-08-14T21:44:55.7942743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7942810Z outputs = self.deberta( 2025-08-14T21:44:55.7943104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7943176Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7943448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7943540Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7943749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7943825Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7944088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.7944203Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.7944469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.7944577Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.7944781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.7944855Z return self.act(input) 2025-08-14T21:44:55.7944859Z 2025-08-14T21:44:55.7944959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7945156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7945506Z return mod(**inputs) 2025-08-14T21:44:55.7945937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7946437Z outputs = self.deberta( 2025-08-14T21:44:55.7946829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7947226Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7947611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7948023Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7948392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7948750Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7949156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.7949618Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.7950122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.7950550Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.7950698Z 2025-08-14T21:44:55.7950799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7951169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7951513Z return mod(**inputs) 2025-08-14T21:44:55.7951911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7952352Z outputs = self.deberta( 2025-08-14T21:44:55.7952754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7953172Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7953592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7954027Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7954432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7954810Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7955262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7955775Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7956257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7956707Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7957161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7957706Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7957950Z 2025-08-14T21:44:55.7958057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7958429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7958803Z return mod(**inputs) 2025-08-14T21:44:55.7959230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7959673Z outputs = self.deberta( 2025-08-14T21:44:55.7960103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7960555Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7960998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7961456Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7961867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7962259Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7962705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7963177Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7963647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7964102Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7964617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.7965209Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7965443Z 2025-08-14T21:44:55.7965555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7965923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7966242Z return mod(**inputs) 2025-08-14T21:44:55.7966629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7967028Z outputs = self.deberta( 2025-08-14T21:44:55.7967425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7967828Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7968230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7968645Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7969003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7969380Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7969791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7970247Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7970660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7971065Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7971462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.7971969Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.7972527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.7973038Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.7973225Z 2025-08-14T21:44:55.7973335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7973692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7974014Z return mod(**inputs) 2025-08-14T21:44:55.7974399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7974817Z outputs = self.deberta( 2025-08-14T21:44:55.7975217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7975644Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7976045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7976455Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7976820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7977186Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7977592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7978011Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7978437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7978844Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7979271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7979810Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7980080Z 2025-08-14T21:44:55.7980181Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7980535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7980853Z return mod(**inputs) 2025-08-14T21:44:55.7981219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7981624Z outputs = self.deberta( 2025-08-14T21:44:55.7982004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7982390Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7982778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7983190Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7983583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7983935Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7984360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7984788Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7985209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7985605Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7985995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.7986518Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.7986774Z 2025-08-14T21:44:55.7986875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7987252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7987596Z return mod(**inputs) 2025-08-14T21:44:55.7988000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7988422Z outputs = self.deberta( 2025-08-14T21:44:55.7988836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7989274Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7989692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7990132Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7990518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7990900Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7991325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7991774Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7992219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.7992653Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.7993098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.7993695Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.7993960Z 2025-08-14T21:44:55.7994086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.7994476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.7994842Z return mod(**inputs) 2025-08-14T21:44:55.7995259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.7995793Z outputs = self.deberta( 2025-08-14T21:44:55.7996236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.7996691Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.7997138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.7997595Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.7998014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.7998413Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.7998874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.7999337Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.7999799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8000241Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8000688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8001244Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8001851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8002397Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8002601Z 2025-08-14T21:44:55.8002727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8003110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8003467Z return mod(**inputs) 2025-08-14T21:44:55.8003887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8004322Z outputs = self.deberta( 2025-08-14T21:44:55.8004734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8005174Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8005609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8006053Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8006450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8006847Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8007292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8007745Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8008205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8008771Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8009240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8009675Z context_layer = torch.bmm( 2025-08-14T21:44:55.8009817Z 2025-08-14T21:44:55.8009931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8010331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8010684Z return mod(**inputs) 2025-08-14T21:44:55.8011106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8011593Z outputs = self.deberta( 2025-08-14T21:44:55.8012005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8012449Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8012868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8013315Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8013726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8014110Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8014582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8015030Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8015468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8015910Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8016357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8016912Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8017167Z 2025-08-14T21:44:55.8017280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8017660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8018009Z return mod(**inputs) 2025-08-14T21:44:55.8018417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8018836Z outputs = self.deberta( 2025-08-14T21:44:55.8019243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8019670Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8020082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8020520Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8020913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8021265Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8021658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8022065Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8022469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8022904Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8023330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8023757Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8023892Z 2025-08-14T21:44:55.8024002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8024359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8024695Z return mod(**inputs) 2025-08-14T21:44:55.8025098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8025525Z outputs = self.deberta( 2025-08-14T21:44:55.8025929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8026354Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8026754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8027153Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8027510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8027887Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8028284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8028725Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8029159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8029559Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8029692Z 2025-08-14T21:44:55.8029799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8030138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8030459Z return mod(**inputs) 2025-08-14T21:44:55.8030827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8031209Z outputs = self.deberta( 2025-08-14T21:44:55.8031588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8031988Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8032383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8032793Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8033163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8033547Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8033977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8034445Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8034918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8035394Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8035876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8036272Z return self.act(input) 2025-08-14T21:44:55.8036410Z 2025-08-14T21:44:55.8036524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8036920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8037254Z return mod(**inputs) 2025-08-14T21:44:55.8037651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8038138Z outputs = self.deberta( 2025-08-14T21:44:55.8038560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8038997Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8039442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8039903Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8040291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8040722Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8041178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8041779Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8042291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8042769Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8042921Z 2025-08-14T21:44:55.8043041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8043447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8043796Z return mod(**inputs) 2025-08-14T21:44:55.8044212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8044647Z outputs = self.deberta( 2025-08-14T21:44:55.8045052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8045491Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8045931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8046384Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8046776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8047184Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8047615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8048053Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8048499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8048931Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8049358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8049902Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8050166Z 2025-08-14T21:44:55.8050282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8050669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8051011Z return mod(**inputs) 2025-08-14T21:44:55.8051404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8051830Z outputs = self.deberta( 2025-08-14T21:44:55.8052239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8052671Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8053093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8053565Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8053958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8054332Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8054770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8055220Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8055686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8056113Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8056553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8057103Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8057350Z 2025-08-14T21:44:55.8057468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8057874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8058206Z return mod(**inputs) 2025-08-14T21:44:55.8058605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8059004Z outputs = self.deberta( 2025-08-14T21:44:55.8059389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8059874Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8060276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8060685Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8061048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8061420Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8061821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8062241Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8062670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8063080Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8063476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8064001Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8064563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8065066Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8065253Z 2025-08-14T21:44:55.8065358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8065720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8066042Z return mod(**inputs) 2025-08-14T21:44:55.8066427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8066840Z outputs = self.deberta( 2025-08-14T21:44:55.8067249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8067676Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8068065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8068478Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8068842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8069200Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8069594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8070030Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8070444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8070845Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8071241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8071793Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8072062Z 2025-08-14T21:44:55.8072175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8072548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8072873Z return mod(**inputs) 2025-08-14T21:44:55.8073262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8073692Z outputs = self.deberta( 2025-08-14T21:44:55.8074091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8074522Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8074941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8075397Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8075869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8076288Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8076746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8077207Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8077677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8078113Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8078545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8079115Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8079406Z 2025-08-14T21:44:55.8079518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8079908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8080256Z return mod(**inputs) 2025-08-14T21:44:55.8080652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8081082Z outputs = self.deberta( 2025-08-14T21:44:55.8081486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8081910Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8082360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8082797Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8083185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8083556Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8083983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8084429Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8084896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8085320Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8085752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8086308Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8086569Z 2025-08-14T21:44:55.8086701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8087054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8087399Z return mod(**inputs) 2025-08-14T21:44:55.8087822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8088239Z outputs = self.deberta( 2025-08-14T21:44:55.8088642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8089063Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8089477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8089908Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8090275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8090641Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8091050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8091475Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8091891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8092299Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8092690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8093204Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8093798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8094327Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8094523Z 2025-08-14T21:44:55.8094635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8095016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8095363Z return mod(**inputs) 2025-08-14T21:44:55.8095765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8096183Z outputs = self.deberta( 2025-08-14T21:44:55.8096590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8097046Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8097464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8097905Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8098299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8098684Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8099109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8099572Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8100023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8100464Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8100891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8101326Z context_layer = torch.bmm( 2025-08-14T21:44:55.8101470Z 2025-08-14T21:44:55.8101590Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8101961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8102320Z return mod(**inputs) 2025-08-14T21:44:55.8102730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8103161Z outputs = self.deberta( 2025-08-14T21:44:55.8103558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8103985Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8104405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8104837Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8105228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8105610Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8106038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8106476Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8106920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8107349Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8107789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8108330Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8108591Z 2025-08-14T21:44:55.8108874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8109287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8109642Z return mod(**inputs) 2025-08-14T21:44:55.8110063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8110512Z outputs = self.deberta( 2025-08-14T21:44:55.8110920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8111341Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8111772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8112255Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8112641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8113021Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8113476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8113942Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8114401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8114930Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8115415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8115947Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8116117Z 2025-08-14T21:44:55.8116247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8116722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8117094Z return mod(**inputs) 2025-08-14T21:44:55.8117611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8118036Z outputs = self.deberta( 2025-08-14T21:44:55.8118441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8118866Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8119277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8119719Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8120099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8120456Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8120843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8121353Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8121773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8122167Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8122299Z 2025-08-14T21:44:55.8122397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8122740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8123055Z return mod(**inputs) 2025-08-14T21:44:55.8123419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8123814Z outputs = self.deberta( 2025-08-14T21:44:55.8124188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8124579Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8124957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8125362Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8125719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8126068Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8126454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8126917Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8127337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8127738Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8128107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8128440Z return self.act(input) 2025-08-14T21:44:55.8128568Z 2025-08-14T21:44:55.8128676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8129018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8129337Z return mod(**inputs) 2025-08-14T21:44:55.8129707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8130087Z outputs = self.deberta( 2025-08-14T21:44:55.8130480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8130858Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8131264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8131678Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8132042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8132417Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8132822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8133274Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8133732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8134156Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8134298Z 2025-08-14T21:44:55.8134402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8134755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8135075Z return mod(**inputs) 2025-08-14T21:44:55.8135444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8135831Z outputs = self.deberta( 2025-08-14T21:44:55.8136200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8136591Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8136982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8137387Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8137753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8138114Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8138507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8138931Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8139356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8139771Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8140159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8140688Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8140933Z 2025-08-14T21:44:55.8141037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8141392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8141712Z return mod(**inputs) 2025-08-14T21:44:55.8142100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8142525Z outputs = self.deberta( 2025-08-14T21:44:55.8142907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8143299Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8143684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8144084Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8144452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8144807Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8145229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8145640Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8146039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8146434Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8146823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8147308Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8147544Z 2025-08-14T21:44:55.8147645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8147997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8148309Z return mod(**inputs) 2025-08-14T21:44:55.8148672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8149062Z outputs = self.deberta( 2025-08-14T21:44:55.8149434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8149825Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8150212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8150654Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8151044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8151420Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8151846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8151953Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8152237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8152323Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8152609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8152805Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8153166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8153309Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8153313Z 2025-08-14T21:44:55.8153423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8153641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8153711Z return mod(**inputs) 2025-08-14T21:44:55.8154019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8154099Z outputs = self.deberta( 2025-08-14T21:44:55.8154419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8154505Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8154799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8154909Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8155155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8155257Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8155547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8155646Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8155998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8156102Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8156391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8156620Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8156634Z 2025-08-14T21:44:55.8156745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8156959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8157038Z return mod(**inputs) 2025-08-14T21:44:55.8157324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8157400Z outputs = self.deberta( 2025-08-14T21:44:55.8157694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8157773Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8158061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8158152Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8158383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8158476Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8158758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8158856Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8159146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8159228Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8159517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8159766Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8159771Z 2025-08-14T21:44:55.8159883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8160103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8160177Z return mod(**inputs) 2025-08-14T21:44:55.8160471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8160565Z outputs = self.deberta( 2025-08-14T21:44:55.8160848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8160931Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8161214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8161305Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8161558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8161642Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8161948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8162046Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8162329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8162416Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8162698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8162905Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8162909Z 2025-08-14T21:44:55.8163018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8163223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8163297Z return mod(**inputs) 2025-08-14T21:44:55.8163584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8163664Z outputs = self.deberta( 2025-08-14T21:44:55.8163942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8164019Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8164309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8164401Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8164624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8164709Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8164971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8165067Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8165322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8165398Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8165663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8165865Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8166169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8166300Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8166304Z 2025-08-14T21:44:55.8166405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8166604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8166685Z return mod(**inputs) 2025-08-14T21:44:55.8166951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8167025Z outputs = self.deberta( 2025-08-14T21:44:55.8167285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8167365Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8167642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8167726Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8167968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8168048Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8168314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8168406Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8168668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8168750Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8169028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8169098Z context_layer = torch.bmm( 2025-08-14T21:44:55.8169112Z 2025-08-14T21:44:55.8169212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8169403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8169475Z return mod(**inputs) 2025-08-14T21:44:55.8169746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8169814Z outputs = self.deberta( 2025-08-14T21:44:55.8170088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8170158Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8170432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8170517Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8170734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8170817Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8171083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8171174Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8171448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8171523Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8171794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8171996Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8172000Z 2025-08-14T21:44:55.8172099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8172299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8172364Z return mod(**inputs) 2025-08-14T21:44:55.8172635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8172719Z outputs = self.deberta( 2025-08-14T21:44:55.8172979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8173056Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8173318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8173400Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8173659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8173736Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8174018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8174108Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8174365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8174490Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8174750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8174841Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8174845Z 2025-08-14T21:44:55.8174941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8175136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8175207Z return mod(**inputs) 2025-08-14T21:44:55.8175470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8175538Z outputs = self.deberta( 2025-08-14T21:44:55.8175802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8175885Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8176144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8176227Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8176434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8176520Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8176770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8176898Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8177146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8177228Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8177231Z 2025-08-14T21:44:55.8177332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8177517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8177596Z return mod(**inputs) 2025-08-14T21:44:55.8177859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8177925Z outputs = self.deberta( 2025-08-14T21:44:55.8178187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8178255Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8178516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8178624Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8178834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8178916Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8179171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8179287Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8179564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8179676Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8179895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8179972Z return self.act(input) 2025-08-14T21:44:55.8179977Z 2025-08-14T21:44:55.8180074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8180279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8180342Z return mod(**inputs) 2025-08-14T21:44:55.8180604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8180678Z outputs = self.deberta( 2025-08-14T21:44:55.8180941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8181017Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8181286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8181368Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8181593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8181670Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8181934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8182071Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8182346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8182436Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8182440Z 2025-08-14T21:44:55.8182542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8182742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8182816Z return mod(**inputs) 2025-08-14T21:44:55.8183093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8183169Z outputs = self.deberta( 2025-08-14T21:44:55.8183442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8183514Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8183810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8183905Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8184114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8184198Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8184453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8184568Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8184824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8184899Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8185168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8185358Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8185362Z 2025-08-14T21:44:55.8185487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8185693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8185771Z return mod(**inputs) 2025-08-14T21:44:55.8186042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8186111Z outputs = self.deberta( 2025-08-14T21:44:55.8186369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8186450Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8186714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8186807Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8187033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8187108Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8187372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8187460Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8187724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8187800Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8188058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8188244Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8188247Z 2025-08-14T21:44:55.8188346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8188543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8188606Z return mod(**inputs) 2025-08-14T21:44:55.8188878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8188958Z outputs = self.deberta( 2025-08-14T21:44:55.8189240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8189316Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8189605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8189714Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8189950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8190035Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8190311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8190418Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8190698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8190807Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8191090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8191287Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8191620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8191776Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8191780Z 2025-08-14T21:44:55.8191898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8192143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8192215Z return mod(**inputs) 2025-08-14T21:44:55.8192524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8192596Z outputs = self.deberta( 2025-08-14T21:44:55.8192901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8192990Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8193296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8193397Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8193638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8193726Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8194032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8194133Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8194433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8194521Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8194823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8195063Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8195067Z 2025-08-14T21:44:55.8195178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8195406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8195483Z return mod(**inputs) 2025-08-14T21:44:55.8195870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8195963Z outputs = self.deberta( 2025-08-14T21:44:55.8196266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8196371Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8196671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8196765Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8197008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8197100Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8197403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8197544Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8197847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8197928Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8198240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8198460Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8198481Z 2025-08-14T21:44:55.8198598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8198830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8198916Z return mod(**inputs) 2025-08-14T21:44:55.8199214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8199287Z outputs = self.deberta( 2025-08-14T21:44:55.8199600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8199678Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8199989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8200086Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8200338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8200421Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8200735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8200834Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8201146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8201228Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8201535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8201747Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8201751Z 2025-08-14T21:44:55.8201861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8202077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8202146Z return mod(**inputs) 2025-08-14T21:44:55.8202436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8202518Z outputs = self.deberta( 2025-08-14T21:44:55.8202801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8202877Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8203196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8203307Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8203547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8203631Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8203912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8204024Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8204308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8204415Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8204692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8204894Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8205256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8205399Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8205403Z 2025-08-14T21:44:55.8205535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8205745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8205816Z return mod(**inputs) 2025-08-14T21:44:55.8206115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8206185Z outputs = self.deberta( 2025-08-14T21:44:55.8206497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8206583Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8206873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8206981Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8207202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8207280Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8207557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8207648Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8207922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8208000Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8208269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8208347Z context_layer = torch.bmm( 2025-08-14T21:44:55.8208352Z 2025-08-14T21:44:55.8208453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8208755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8208839Z return mod(**inputs) 2025-08-14T21:44:55.8209111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8209193Z outputs = self.deberta( 2025-08-14T21:44:55.8209460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8209534Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8209857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8209943Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8210170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8210251Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8210514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8210613Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8210902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8210978Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8211248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8211435Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8211439Z 2025-08-14T21:44:55.8211571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8211769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8211834Z return mod(**inputs) 2025-08-14T21:44:55.8212144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8212218Z outputs = self.deberta( 2025-08-14T21:44:55.8212494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8212567Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8212831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8212921Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8213141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8213219Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8213494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8213585Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8213860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8213975Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8214241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8214334Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8214338Z 2025-08-14T21:44:55.8214437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8214642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8214709Z return mod(**inputs) 2025-08-14T21:44:55.8214980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8215055Z outputs = self.deberta( 2025-08-14T21:44:55.8215324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8215399Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8215675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8215781Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8216005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8216085Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8216349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8216477Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8216742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8216854Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8216857Z 2025-08-14T21:44:55.8216961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8217163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8217243Z return mod(**inputs) 2025-08-14T21:44:55.8217522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8217593Z outputs = self.deberta( 2025-08-14T21:44:55.8217890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8217967Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8218260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8218351Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8218575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8218657Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8218932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8219060Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8219336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8219452Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8219675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8219748Z return self.act(input) 2025-08-14T21:44:55.8219753Z 2025-08-14T21:44:55.8219857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8220064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8220129Z return mod(**inputs) 2025-08-14T21:44:55.8220414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8220479Z outputs = self.deberta( 2025-08-14T21:44:55.8220734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8220810Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8221066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8221151Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8221356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8221433Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8221691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8221816Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8222099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8222185Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8222189Z 2025-08-14T21:44:55.8222284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8222480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8222541Z return mod(**inputs) 2025-08-14T21:44:55.8222805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8222898Z outputs = self.deberta( 2025-08-14T21:44:55.8223147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8223224Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8223477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8223556Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8223782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8223859Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8224125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8224224Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8224481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8224561Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8224819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8225003Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8225006Z 2025-08-14T21:44:55.8225114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8225316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8225384Z return mod(**inputs) 2025-08-14T21:44:55.8225646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8225711Z outputs = self.deberta( 2025-08-14T21:44:55.8225976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8226047Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8226308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8226398Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8226621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8226700Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8226956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8227042Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8227307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8227381Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8227640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8227837Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8227841Z 2025-08-14T21:44:55.8227939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8228131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8228194Z return mod(**inputs) 2025-08-14T21:44:55.8228453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8228523Z outputs = self.deberta( 2025-08-14T21:44:55.8228790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8228864Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8229112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8229194Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8229405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8229494Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8229754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8229856Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8230111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8230192Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8230453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8230635Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8230941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8231072Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8231075Z 2025-08-14T21:44:55.8231184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8231376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8231443Z return mod(**inputs) 2025-08-14T21:44:55.8231721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8231788Z outputs = self.deberta( 2025-08-14T21:44:55.8232059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8232133Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8232394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8232489Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8232704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8232790Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8233052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8233144Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8233413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8233488Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8233766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8233978Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8233981Z 2025-08-14T21:44:55.8234082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8234281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8234347Z return mod(**inputs) 2025-08-14T21:44:55.8234618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8234712Z outputs = self.deberta( 2025-08-14T21:44:55.8234975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8235055Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8235317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8235415Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8235639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8235774Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8236069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8236173Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8236455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8236544Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8236827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8237041Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8237046Z 2025-08-14T21:44:55.8237156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8237351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8237428Z return mod(**inputs) 2025-08-14T21:44:55.8237698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8237767Z outputs = self.deberta( 2025-08-14T21:44:55.8238038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8238110Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8238377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8238469Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8238686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8238771Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8239035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8239128Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8239399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8239474Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8239743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8239955Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8239958Z 2025-08-14T21:44:55.8240063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8240266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8240330Z return mod(**inputs) 2025-08-14T21:44:55.8240608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8240693Z outputs = self.deberta( 2025-08-14T21:44:55.8240960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8241040Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8241307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8241393Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8241632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8241713Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8242007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8242100Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8242367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8242455Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8242721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8242922Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8243232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8243366Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8243370Z 2025-08-14T21:44:55.8243484Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8243681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8243751Z return mod(**inputs) 2025-08-14T21:44:55.8244032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8244104Z outputs = self.deberta( 2025-08-14T21:44:55.8244379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8244457Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8244726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8244822Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8245042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8245131Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8245398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8245495Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8245770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8245868Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8246134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8246214Z context_layer = torch.bmm( 2025-08-14T21:44:55.8246218Z 2025-08-14T21:44:55.8246319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8246523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8246587Z return mod(**inputs) 2025-08-14T21:44:55.8246858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8246950Z outputs = self.deberta( 2025-08-14T21:44:55.8247222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8247302Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8247571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8247655Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8247899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8247978Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8248261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8248361Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8248625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8248708Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8248970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8249158Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8249161Z 2025-08-14T21:44:55.8249272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8249467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8249538Z return mod(**inputs) 2025-08-14T21:44:55.8249806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8249875Z outputs = self.deberta( 2025-08-14T21:44:55.8250143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8250213Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8250480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8250568Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8250773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8250855Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8251108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8251193Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8251454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8251565Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8251823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8251923Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8251930Z 2025-08-14T21:44:55.8252025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8252218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8252282Z return mod(**inputs) 2025-08-14T21:44:55.8252548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8252612Z outputs = self.deberta( 2025-08-14T21:44:55.8252864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8252961Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8253210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8253292Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8253504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8253593Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8253857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8253987Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8254244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8254334Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8254337Z 2025-08-14T21:44:55.8254434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8254631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8254706Z return mod(**inputs) 2025-08-14T21:44:55.8254957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8255029Z outputs = self.deberta( 2025-08-14T21:44:55.8255278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8255347Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8255606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8255690Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8255903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8255975Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8256228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8256347Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8256599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8256712Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8256909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8256976Z return self.act(input) 2025-08-14T21:44:55.8256981Z 2025-08-14T21:44:55.8257081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8257265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8257328Z return mod(**inputs) 2025-08-14T21:44:55.8257587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8257668Z outputs = self.deberta( 2025-08-14T21:44:55.8257928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8257996Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8258245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8258332Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8258534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8259533Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8259792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8259921Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8260177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8260273Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8260277Z 2025-08-14T21:44:55.8260376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8260588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8260654Z return mod(**inputs) 2025-08-14T21:44:55.8260917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8260981Z outputs = self.deberta( 2025-08-14T21:44:55.8261233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8261309Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8261561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8261642Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8261852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8261926Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8262184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8262273Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8262521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8262604Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8262852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8263035Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8263038Z 2025-08-14T21:44:55.8263137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8263321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8263391Z return mod(**inputs) 2025-08-14T21:44:55.8263649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8263724Z outputs = self.deberta( 2025-08-14T21:44:55.8263980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8264051Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8264314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8264416Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8264634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8264720Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8264975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8265073Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8265345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8265420Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8265682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8265856Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8265860Z 2025-08-14T21:44:55.8265988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8266185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8266250Z return mod(**inputs) 2025-08-14T21:44:55.8266546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8266616Z outputs = self.deberta( 2025-08-14T21:44:55.8266885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8266963Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8267228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8267329Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8267548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8267628Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8267910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8268002Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8268276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8268355Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8268623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8268816Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8269125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8269258Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8269268Z 2025-08-14T21:44:55.8269372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8269569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8269642Z return mod(**inputs) 2025-08-14T21:44:55.8269916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8269985Z outputs = self.deberta( 2025-08-14T21:44:55.8270260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8270350Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8270622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8270710Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8270938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8271031Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8271311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8271443Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8271729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8271808Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8272104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8272343Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8272348Z 2025-08-14T21:44:55.8272459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8272697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8272763Z return mod(**inputs) 2025-08-14T21:44:55.8273040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8273110Z outputs = self.deberta( 2025-08-14T21:44:55.8273375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8273456Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8273721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8273817Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8274048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8274148Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8274438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8274533Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8274811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8274896Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8275176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8275405Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8275411Z 2025-08-14T21:44:55.8275519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8275792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8275882Z return mod(**inputs) 2025-08-14T21:44:55.8276170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8276251Z outputs = self.deberta( 2025-08-14T21:44:55.8276532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8276607Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8276922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8277011Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8277245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8277336Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8277621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8277725Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8278031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8278121Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8278399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8278593Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8278597Z 2025-08-14T21:44:55.8278724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8278920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8278987Z return mod(**inputs) 2025-08-14T21:44:55.8279286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8279362Z outputs = self.deberta( 2025-08-14T21:44:55.8279645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8279729Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8280011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8280110Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8280345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8280428Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8280721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8280817Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8281105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8281188Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8281466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8281663Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8281965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8282104Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8282108Z 2025-08-14T21:44:55.8282213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8282412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8282489Z return mod(**inputs) 2025-08-14T21:44:55.8282776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8282848Z outputs = self.deberta( 2025-08-14T21:44:55.8283138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8283237Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8283533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8283622Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8283862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8283952Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8284244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8284366Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8284657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8284738Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8285032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8285126Z context_layer = torch.bmm( 2025-08-14T21:44:55.8285131Z 2025-08-14T21:44:55.8285241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8285475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8285545Z return mod(**inputs) 2025-08-14T21:44:55.8285837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8285911Z outputs = self.deberta( 2025-08-14T21:44:55.8286191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8286276Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8286561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8286656Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8286887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8286969Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8287256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8287351Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8287631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8287719Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8288000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8288205Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8288209Z 2025-08-14T21:44:55.8288318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8288523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8288600Z return mod(**inputs) 2025-08-14T21:44:55.8288889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8288966Z outputs = self.deberta( 2025-08-14T21:44:55.8289247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8289324Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8289615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8289723Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8289953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8290045Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8290328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8290429Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8290734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8290856Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8291151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8291240Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8291244Z 2025-08-14T21:44:55.8291359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8291584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8291655Z return mod(**inputs) 2025-08-14T21:44:55.8291960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8292033Z outputs = self.deberta( 2025-08-14T21:44:55.8292319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8292402Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8292684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8292779Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8293008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8293091Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8293377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8293503Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8293787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8293874Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8293878Z 2025-08-14T21:44:55.8293982Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8294194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8294266Z return mod(**inputs) 2025-08-14T21:44:55.8294551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8294634Z outputs = self.deberta( 2025-08-14T21:44:55.8294915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8294993Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8295258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8295344Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8295567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8295644Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8295913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8296050Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8296319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8296437Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8296650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8296721Z return self.act(input) 2025-08-14T21:44:55.8296747Z 2025-08-14T21:44:55.8296850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8297044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8297117Z return mod(**inputs) 2025-08-14T21:44:55.8297387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8297457Z outputs = self.deberta( 2025-08-14T21:44:55.8297747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8297824Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8298112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8298199Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8298458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8298546Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8298817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8298950Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8299227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8299312Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8299315Z 2025-08-14T21:44:55.8299424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8299621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8299686Z return mod(**inputs) 2025-08-14T21:44:55.8299964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8300033Z outputs = self.deberta( 2025-08-14T21:44:55.8300310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8300383Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8300650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8300741Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8301013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8301093Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8301368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8301464Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8301744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8301821Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8302091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8302306Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8302310Z 2025-08-14T21:44:55.8302414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8302614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8302682Z return mod(**inputs) 2025-08-14T21:44:55.8302953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8303044Z outputs = self.deberta( 2025-08-14T21:44:55.8303311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8303384Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8303674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8303764Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8304060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8304139Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8304421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8304523Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8304789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8304873Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8305139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8305319Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8305322Z 2025-08-14T21:44:55.8305433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8305625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8305689Z return mod(**inputs) 2025-08-14T21:44:55.8305968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8306035Z outputs = self.deberta( 2025-08-14T21:44:55.8306310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8306380Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8306647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8306740Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8306959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8307041Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8307309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8307398Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8307670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8307748Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8308009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8308221Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8308528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8308789Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8308795Z 2025-08-14T21:44:55.8308905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8309111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8309188Z return mod(**inputs) 2025-08-14T21:44:55.8309518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8309597Z outputs = self.deberta( 2025-08-14T21:44:55.8309890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8309968Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8310254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8310369Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8310604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8310709Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8310990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8311095Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8311371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8311453Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8311741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8311964Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8311968Z 2025-08-14T21:44:55.8312083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8312288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8312356Z return mod(**inputs) 2025-08-14T21:44:55.8312646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8312719Z outputs = self.deberta( 2025-08-14T21:44:55.8313007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8313084Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8313363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8313461Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8313690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8313772Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8314064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8314161Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8314447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8314528Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8314842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8315068Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8315073Z 2025-08-14T21:44:55.8315181Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8315393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8315464Z return mod(**inputs) 2025-08-14T21:44:55.8315813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8315923Z outputs = self.deberta( 2025-08-14T21:44:55.8316212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8316297Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8316589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8316691Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8316946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8317032Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8317332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8317441Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8317723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8317814Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8318093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8318293Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8318297Z 2025-08-14T21:44:55.8318417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8318625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8318705Z return mod(**inputs) 2025-08-14T21:44:55.8318992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8319066Z outputs = self.deberta( 2025-08-14T21:44:55.8319356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8319435Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8319714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8319814Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8320044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8320132Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8320413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8320508Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8320794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8320875Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8321168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8321377Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8321684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8321823Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8321827Z 2025-08-14T21:44:55.8321929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8322133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8322216Z return mod(**inputs) 2025-08-14T21:44:55.8322489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8322562Z outputs = self.deberta( 2025-08-14T21:44:55.8322828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8322901Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8323192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8323278Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8323521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8323601Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8323864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8323963Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8324226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8324305Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8324573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8324645Z context_layer = torch.bmm( 2025-08-14T21:44:55.8324648Z 2025-08-14T21:44:55.8324757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8324953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8325021Z return mod(**inputs) 2025-08-14T21:44:55.8325312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8325385Z outputs = self.deberta( 2025-08-14T21:44:55.8325677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8325748Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8326012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8326104Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8326324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8326402Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8326673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8326763Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8327033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8327108Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8327371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8327583Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8327587Z 2025-08-14T21:44:55.8327690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8327891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8327957Z return mod(**inputs) 2025-08-14T21:44:55.8328228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8328321Z outputs = self.deberta( 2025-08-14T21:44:55.8328588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8328662Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8328935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8329022Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8329262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8329342Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8329623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8329725Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8329993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8330117Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8330383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8330469Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8330472Z 2025-08-14T21:44:55.8330585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8330783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8330855Z return mod(**inputs) 2025-08-14T21:44:55.8331130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8331200Z outputs = self.deberta( 2025-08-14T21:44:55.8331468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8331540Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8331803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8331896Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8332111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8332195Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8332459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8332578Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8332849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8332934Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8332937Z 2025-08-14T21:44:55.8333052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8333249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8333329Z return mod(**inputs) 2025-08-14T21:44:55.8333604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8333675Z outputs = self.deberta( 2025-08-14T21:44:55.8333939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8334017Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8334281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8334393Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8334610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8334687Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8334959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8335078Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8335364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8335476Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8335710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8335788Z return self.act(input) 2025-08-14T21:44:55.8335793Z 2025-08-14T21:44:55.8335895Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8336090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8336164Z return mod(**inputs) 2025-08-14T21:44:55.8336434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8336508Z outputs = self.deberta( 2025-08-14T21:44:55.8336774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8336846Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8337118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8337200Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8337416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8337504Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8337766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8337906Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8338171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8338254Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8338257Z 2025-08-14T21:44:55.8338365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8338560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8338634Z return mod(**inputs) 2025-08-14T21:44:55.8338904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8338972Z outputs = self.deberta( 2025-08-14T21:44:55.8339242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8339332Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8339600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8339692Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8339908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8339994Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8340258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8340371Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8340645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8340722Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8340994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8341181Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8341184Z 2025-08-14T21:44:55.8341304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8341506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8341584Z return mod(**inputs) 2025-08-14T21:44:55.8341867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8341936Z outputs = self.deberta( 2025-08-14T21:44:55.8342204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8342282Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8342551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8342633Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8342858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8342936Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8343213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8343304Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8343572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8343655Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8343922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8344111Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8344114Z 2025-08-14T21:44:55.8344218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8344427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8344502Z return mod(**inputs) 2025-08-14T21:44:55.8344793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8344877Z outputs = self.deberta( 2025-08-14T21:44:55.8345150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8345222Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8345496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8345599Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8345818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8345906Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8346182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8346279Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8346536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8346630Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8346894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8347076Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8347388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8347526Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8347529Z 2025-08-14T21:44:55.8347629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8347840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8347905Z return mod(**inputs) 2025-08-14T21:44:55.8348170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8348243Z outputs = self.deberta( 2025-08-14T21:44:55.8348500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8348577Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8348836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8348919Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8349140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8349217Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8349474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8349572Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8349830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8349912Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8350170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8350373Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8350384Z 2025-08-14T21:44:55.8350488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8350683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8350756Z return mod(**inputs) 2025-08-14T21:44:55.8351030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8351101Z outputs = self.deberta( 2025-08-14T21:44:55.8351388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8351483Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8351778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8351868Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8352104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8352198Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8352484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8352604Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8352890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8352969Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8353257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8353475Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8353496Z 2025-08-14T21:44:55.8353606Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8353838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8353909Z return mod(**inputs) 2025-08-14T21:44:55.8354204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8354278Z outputs = self.deberta( 2025-08-14T21:44:55.8354556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8354639Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8354922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8355011Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8355247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8355328Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8355617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8355911Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8356205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8356294Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8356577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8356787Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8356792Z 2025-08-14T21:44:55.8356900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8357103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8357182Z return mod(**inputs) 2025-08-14T21:44:55.8357468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8357541Z outputs = self.deberta( 2025-08-14T21:44:55.8357831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8357908Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8358198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8358313Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8358542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8358634Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8358916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8359021Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8359315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8359396Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8359686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8359885Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8360235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8360376Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8360380Z 2025-08-14T21:44:55.8360506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8360723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8360793Z return mod(**inputs) 2025-08-14T21:44:55.8361081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8361161Z outputs = self.deberta( 2025-08-14T21:44:55.8361443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8361528Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8361813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8361904Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8362143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8362226Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8362517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8362618Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8362897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8362989Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8363271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8363350Z context_layer = torch.bmm( 2025-08-14T21:44:55.8363362Z 2025-08-14T21:44:55.8363472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8363681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8363756Z return mod(**inputs) 2025-08-14T21:44:55.8364041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8364114Z outputs = self.deberta( 2025-08-14T21:44:55.8364401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8364476Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8364811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8364902Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8365155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8365246Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8365536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8365647Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8365921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8365998Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8366269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8366457Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8366461Z 2025-08-14T21:44:55.8366580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8366784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8366849Z return mod(**inputs) 2025-08-14T21:44:55.8367142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8367212Z outputs = self.deberta( 2025-08-14T21:44:55.8367483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8367563Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8367836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8367922Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8368155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8368233Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8368509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8368600Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8368876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8368997Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8369260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8369350Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8369353Z 2025-08-14T21:44:55.8369453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8369650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8369721Z return mod(**inputs) 2025-08-14T21:44:55.8369990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8370057Z outputs = self.deberta( 2025-08-14T21:44:55.8370327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8370396Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8370665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8370784Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8370993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8371078Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8371333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8371456Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8371711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8371807Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8371811Z 2025-08-14T21:44:55.8371916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8372108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8372173Z return mod(**inputs) 2025-08-14T21:44:55.8372446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8372526Z outputs = self.deberta( 2025-08-14T21:44:55.8372794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8372881Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8373150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8373244Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8373464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8373547Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8373813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8373930Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8374200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8374310Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8374522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8374598Z return self.act(input) 2025-08-14T21:44:55.8374603Z 2025-08-14T21:44:55.8374702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8374903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8374968Z return mod(**inputs) 2025-08-14T21:44:55.8375238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8375313Z outputs = self.deberta( 2025-08-14T21:44:55.8375578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8375656Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8375922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8376006Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8376230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8376309Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8376574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8376734Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8376999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8377090Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8377093Z 2025-08-14T21:44:55.8377195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8377390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8377465Z return mod(**inputs) 2025-08-14T21:44:55.8377736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8377828Z outputs = self.deberta( 2025-08-14T21:44:55.8378093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8378166Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8378441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8378542Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8378760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8378847Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8379127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8379232Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8379499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8379579Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8379854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8380044Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8380049Z 2025-08-14T21:44:55.8380170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8380364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8380433Z return mod(**inputs) 2025-08-14T21:44:55.8380710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8380782Z outputs = self.deberta( 2025-08-14T21:44:55.8381051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8381136Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8381406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8381512Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8381727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8381805Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8382074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8382167Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8382436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8382515Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8382776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8382978Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8382982Z 2025-08-14T21:44:55.8383082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8383281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8383344Z return mod(**inputs) 2025-08-14T21:44:55.8383606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8383704Z outputs = self.deberta( 2025-08-14T21:44:55.8383963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8384033Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8384299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8384383Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8384619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8384698Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8384972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8385073Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8385330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8385406Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8385669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8385848Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8386149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8386276Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8386279Z 2025-08-14T21:44:55.8386379Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8386575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8386639Z return mod(**inputs) 2025-08-14T21:44:55.8386910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8386976Z outputs = self.deberta( 2025-08-14T21:44:55.8387232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8387310Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8387567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8387655Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8387864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8387941Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8388207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8388299Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8388560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8388642Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8388926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8389146Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8389150Z 2025-08-14T21:44:55.8389251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8389447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8389520Z return mod(**inputs) 2025-08-14T21:44:55.8389788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8389878Z outputs = self.deberta( 2025-08-14T21:44:55.8390146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8390219Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8390499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8390605Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8390835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8390941Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8391225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8391329Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8391611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8391693Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8391988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8392208Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8392212Z 2025-08-14T21:44:55.8392327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8392538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8392607Z return mod(**inputs) 2025-08-14T21:44:55.8392902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8392975Z outputs = self.deberta( 2025-08-14T21:44:55.8393265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8393340Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8393624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8393719Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8393952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8394035Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8394326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8394423Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8394714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8394795Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8395079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8395303Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8395307Z 2025-08-14T21:44:55.8395417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8395639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8395783Z return mod(**inputs) 2025-08-14T21:44:55.8396093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8396200Z outputs = self.deberta( 2025-08-14T21:44:55.8396492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8396571Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8396883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8396980Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8397243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8397331Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8397627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8397741Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8398009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8398093Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8398359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8398552Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8398886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8399027Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8399031Z 2025-08-14T21:44:55.8399148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8399357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8399427Z return mod(**inputs) 2025-08-14T21:44:55.8399724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8399795Z outputs = self.deberta( 2025-08-14T21:44:55.8400079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8400165Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8400461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8400553Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8400775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8400854Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8401128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8401222Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8401489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8401592Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8401862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8401940Z context_layer = torch.bmm( 2025-08-14T21:44:55.8401944Z 2025-08-14T21:44:55.8402047Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8402247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8402322Z return mod(**inputs) 2025-08-14T21:44:55.8402592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8402686Z outputs = self.deberta( 2025-08-14T21:44:55.8402951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8403022Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8403295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8403380Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8403625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8403717Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8404015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8404120Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8404398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8404478Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8404770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8404969Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8404973Z 2025-08-14T21:44:55.8405089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8405293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8405368Z return mod(**inputs) 2025-08-14T21:44:55.8405646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8405713Z outputs = self.deberta( 2025-08-14T21:44:55.8405987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8406058Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8406322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8406415Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8406632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8406712Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8406984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8407074Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8407346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8407460Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8407726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8407841Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8407844Z 2025-08-14T21:44:55.8407945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8408160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8408229Z return mod(**inputs) 2025-08-14T21:44:55.8408513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8408599Z outputs = self.deberta( 2025-08-14T21:44:55.8409091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8409170Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8409447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8409535Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8409763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8409882Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8410150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8410311Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8410580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8410671Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8410675Z 2025-08-14T21:44:55.8410774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8410972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8411048Z return mod(**inputs) 2025-08-14T21:44:55.8411317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8411389Z outputs = self.deberta( 2025-08-14T21:44:55.8411662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8411737Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8412012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8412098Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8412315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8412403Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8412667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8412799Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8413048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8413155Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8413360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8413427Z return self.act(input) 2025-08-14T21:44:55.8413432Z 2025-08-14T21:44:55.8413532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8413731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8413796Z return mod(**inputs) 2025-08-14T21:44:55.8414077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8414167Z outputs = self.deberta( 2025-08-14T21:44:55.8414427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8414502Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8414758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8414839Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8415054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8415158Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8415415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8415542Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8415797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8415900Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8415904Z 2025-08-14T21:44:55.8416002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8416211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8416275Z return mod(**inputs) 2025-08-14T21:44:55.8416531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8416603Z outputs = self.deberta( 2025-08-14T21:44:55.8416854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8416924Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8417181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8417262Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8417472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8417545Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8417870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8418024Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8418312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8418447Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8418714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8419126Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8419134Z 2025-08-14T21:44:55.8419255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8419465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8419585Z return mod(**inputs) 2025-08-14T21:44:55.8419871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8419950Z outputs = self.deberta( 2025-08-14T21:44:55.8420317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8420410Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8420756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8420862Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8421091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8421223Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8421507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8421650Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8421940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8422036Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8422328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8422544Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8422548Z 2025-08-14T21:44:55.8422727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8422944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8423041Z return mod(**inputs) 2025-08-14T21:44:55.8423371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8423448Z outputs = self.deberta( 2025-08-14T21:44:55.8423787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8423880Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8424154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8424293Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8424544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8424631Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8424978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8425089Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8425402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8425502Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8425787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8426033Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8426368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8426537Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8426541Z 2025-08-14T21:44:55.8426659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8426872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8426981Z return mod(**inputs) 2025-08-14T21:44:55.8427276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8427397Z outputs = self.deberta( 2025-08-14T21:44:55.8427672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8427786Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8428101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8428195Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8428484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8428584Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8428863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8428992Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8445191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8445386Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8445742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8446112Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8446119Z 2025-08-14T21:44:55.8446239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8446497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8446572Z return mod(**inputs) 2025-08-14T21:44:55.8446858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8446950Z outputs = self.deberta( 2025-08-14T21:44:55.8447220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8447309Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8447577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8447672Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8447903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8447988Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8448252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8448359Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8448629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8448713Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8448973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8449178Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8449183Z 2025-08-14T21:44:55.8449302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8449500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8449576Z return mod(**inputs) 2025-08-14T21:44:55.8449837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8449910Z outputs = self.deberta( 2025-08-14T21:44:55.8450173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8450246Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8450531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8450627Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8450845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8450932Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8451195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8451286Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8451585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8451663Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8451930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8452136Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8452140Z 2025-08-14T21:44:55.8452259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8452462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8452527Z return mod(**inputs) 2025-08-14T21:44:55.8452816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8452894Z outputs = self.deberta( 2025-08-14T21:44:55.8453151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8453230Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8453491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8453579Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8453804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8453885Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8454157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8454251Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8454517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8454602Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8454866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8455063Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8455386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8455516Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8455520Z 2025-08-14T21:44:55.8455626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8455819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8455887Z return mod(**inputs) 2025-08-14T21:44:55.8456159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8456228Z outputs = self.deberta( 2025-08-14T21:44:55.8456492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8456583Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8456843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8456934Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8457148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8457234Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8457494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8457602Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8457867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8457943Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8458200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8458311Z context_layer = torch.bmm( 2025-08-14T21:44:55.8458316Z 2025-08-14T21:44:55.8458419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8458636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8458704Z return mod(**inputs) 2025-08-14T21:44:55.8458977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8459052Z outputs = self.deberta( 2025-08-14T21:44:55.8459304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8459375Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8459634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8459717Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8459933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8460008Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8460261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8460358Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8460609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8460689Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8460940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8461120Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8461124Z 2025-08-14T21:44:55.8461233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8461425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8461496Z return mod(**inputs) 2025-08-14T21:44:55.8461760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8461828Z outputs = self.deberta( 2025-08-14T21:44:55.8462092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8462163Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8462421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8462535Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8462748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8462834Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8463099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8463190Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8463478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8463595Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8463871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8463958Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8463962Z 2025-08-14T21:44:55.8464064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8464284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8464353Z return mod(**inputs) 2025-08-14T21:44:55.8464638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8464716Z outputs = self.deberta( 2025-08-14T21:44:55.8464990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8465071Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8465343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8465431Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8465662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8465743Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8466023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8466151Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8466427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8466523Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8466526Z 2025-08-14T21:44:55.8466635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8466836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8466914Z return mod(**inputs) 2025-08-14T21:44:55.8467195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8467273Z outputs = self.deberta( 2025-08-14T21:44:55.8467546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8467623Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8467905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8467993Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8468226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8468308Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8468583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8468731Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8469002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8469118Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8469346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8469418Z return self.act(input) 2025-08-14T21:44:55.8469446Z 2025-08-14T21:44:55.8469561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8469761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8469827Z return mod(**inputs) 2025-08-14T21:44:55.8470109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8470180Z outputs = self.deberta( 2025-08-14T21:44:55.8470473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8470549Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8470841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8470937Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8471168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8471253Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8471546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8471687Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8471976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8472064Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8472068Z 2025-08-14T21:44:55.8472174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8472390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8472461Z return mod(**inputs) 2025-08-14T21:44:55.8472755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8472828Z outputs = self.deberta( 2025-08-14T21:44:55.8473111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8473195Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8473474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8473567Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8473805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8473888Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8474182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8474285Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8474579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8474672Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8474974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8475200Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8475205Z 2025-08-14T21:44:55.8475314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8475522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8475602Z return mod(**inputs) 2025-08-14T21:44:55.8475985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8476086Z outputs = self.deberta( 2025-08-14T21:44:55.8476387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8476467Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8476770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8476863Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8477118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8477216Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8477526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8477647Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8477925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8478005Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8478286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:44:55.8478474Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8478478Z 2025-08-14T21:44:55.8478596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8478797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8478873Z return mod(**inputs) 2025-08-14T21:44:55.8479152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8479224Z outputs = self.deberta( 2025-08-14T21:44:55.8479489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8479570Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8479833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8479927Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8480144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8480222Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8480496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8480589Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8480854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8480938Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8481200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:44:55.8481416Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:44:55.8481724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8481858Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8481862Z 2025-08-14T21:44:55.8481974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8482172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8482263Z return mod(**inputs) 2025-08-14T21:44:55.8482538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8482616Z outputs = self.deberta( 2025-08-14T21:44:55.8482886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8482959Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8483239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8483323Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8483534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8483633Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8483894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8483987Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8484262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8484338Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8484616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8484826Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8484829Z 2025-08-14T21:44:55.8484930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8485132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8485195Z return mod(**inputs) 2025-08-14T21:44:55.8485465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8485532Z outputs = self.deberta( 2025-08-14T21:44:55.8485795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8485877Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8486142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8486227Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8486445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8486537Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8486802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8486894Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8487167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8487244Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8487535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:44:55.8487745Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:44:55.8487749Z 2025-08-14T21:44:55.8487854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8488062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8488128Z return mod(**inputs) 2025-08-14T21:44:55.8488407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8488492Z outputs = self.deberta( 2025-08-14T21:44:55.8488767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8488849Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8489123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8489209Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8489456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8489536Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8489826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8489917Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8490186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8490269Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8490535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8490738Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8490742Z 2025-08-14T21:44:55.8490843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8491032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8491102Z return mod(**inputs) 2025-08-14T21:44:55.8491364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8491431Z outputs = self.deberta( 2025-08-14T21:44:55.8491696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8491765Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8492025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8492108Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8492321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8492406Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8492671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8492768Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8493039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8493113Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8493388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:44:55.8493605Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:44:55.8493914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:44:55.8494054Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:44:55.8494058Z 2025-08-14T21:44:55.8494161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8494370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8494449Z return mod(**inputs) 2025-08-14T21:44:55.8494710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8494786Z outputs = self.deberta( 2025-08-14T21:44:55.8495061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8495141Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8495421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8495519Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8495755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8495832Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8496089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8496186Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8496447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8496529Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8496795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:44:55.8496869Z context_layer = torch.bmm( 2025-08-14T21:44:55.8496873Z 2025-08-14T21:44:55.8496984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8497185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8497257Z return mod(**inputs) 2025-08-14T21:44:55.8497526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8497595Z outputs = self.deberta( 2025-08-14T21:44:55.8497869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8497941Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8498210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8498302Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8498521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8498607Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8498874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8498966Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8499238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:44:55.8499312Z self_output, att_matrix = self.self( 2025-08-14T21:44:55.8499580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:44:55.8499785Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:44:55.8499788Z 2025-08-14T21:44:55.8499891Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8500097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8500163Z return mod(**inputs) 2025-08-14T21:44:55.8500446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8500531Z outputs = self.deberta( 2025-08-14T21:44:55.8500798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8500878Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8501148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8501233Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8501478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8501558Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8501846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:44:55.8501938Z attention_output, att_matrix = self.attention( 2025-08-14T21:44:55.8502203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:44:55.8502325Z attention_output = self.output(self_output, query_states) 2025-08-14T21:44:55.8502591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:44:55.8502682Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8502685Z 2025-08-14T21:44:55.8502787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8502985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8503057Z return mod(**inputs) 2025-08-14T21:44:55.8503328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8503395Z outputs = self.deberta( 2025-08-14T21:44:55.8503671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8503743Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8504020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8504110Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8504337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8504429Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8504709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8504834Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8505120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:44:55.8505210Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8505213Z 2025-08-14T21:44:55.8505326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8505532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8505622Z return mod(**inputs) 2025-08-14T21:44:55.8505926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8505995Z outputs = self.deberta( 2025-08-14T21:44:55.8506265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8506338Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8506602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8506711Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8506931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8507008Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8507282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:44:55.8507401Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:44:55.8507690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:44:55.8507803Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:44:55.8508027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:55.8508108Z return self.act(input) 2025-08-14T21:44:55.8508113Z 2025-08-14T21:44:55.8508214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8508415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8508480Z return mod(**inputs) 2025-08-14T21:44:55.8508903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:44:55.8508990Z outputs = self.deberta( 2025-08-14T21:44:55.8509260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:44:55.8509332Z encoder_outputs = self.encoder( 2025-08-14T21:44:55.8509611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:44:55.8509697Z output_states, attn_weights = layer_module( 2025-08-14T21:44:55.8509925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:55.8510007Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:55.8510283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:44:55.8510434Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:44:55.8510712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:44:55.8510809Z hidden_states = self.dense(hidden_states) 2025-08-14T21:44:55.8510813Z 2025-08-14T21:44:55.8510920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8511130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8511209Z return mod(**inputs) 2025-08-14T21:44:55.8511495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1244, in forward 2025-08-14T21:44:55.8511587Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:44:55.8511599Z 2025-08-14T21:44:55.8511707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8511912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8512047Z return mod(**inputs) 2025-08-14T21:44:55.8512333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1262, in forward 2025-08-14T21:44:55.8512444Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:44:55.8512448Z 2025-08-14T21:44:55.8512560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:55.8512764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:55.8512839Z return mod(**inputs) 2025-08-14T21:44:55.8513155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1263, in forward 2025-08-14T21:44:55.8513251Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:44:55.8513255Z 2025-08-14T21:45:07.8739163Z Compilation time (from dynamo_timed): 25.644883218 2025-08-14T21:45:07.8743803Z pass 2025-08-14T21:45:07.8744177Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:07.8745333Z TIMING: _recursive_pre_grad_passes:0.01402 _recursive_joint_graph_passes:1.18607 _recursive_post_grad_passes:0.3154 async_compile.wait:0.5112 code_gen:10.79393 inductor_compile:13.764 backend_compile:20.41872 gc:0.00059 entire_frame_compile:25.64488 total_wall_time:25.64488 2025-08-14T21:45:07.8746374Z STATS: call_* op count: 1087 | FakeTensorMode.__torch_dispatch__:30540 | FakeTensor.__torch_dispatch__:11359 | ProxyTorchDispatchMode.__torch_dispatch__:11524 2025-08-14T21:45:07.8746937Z Dynamo produced 1 graphs covering 1087 ops with 0 graph breaks (0 unique) 2025-08-14T21:45:13.6813374Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:45:13.6814378Z from pkg_resources import resource_filename 2025-08-14T21:45:14.3523443Z 2025-08-14T21:45:15.0996585Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:45:15.0996925Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:45:15.1001706Z cpu eval DistilBertForMaskedLM 2025-08-14T21:45:15.4475506Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:15.5043721Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:15.5624860Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:20.5962139Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.5966744Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.5966995Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.5967759Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.5968119Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.5968359Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.5968625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.5969046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.5969429Z return mod(**inputs) 2025-08-14T21:45:20.5969907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.5970378Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.5970835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.5971293Z return self.transformer( 2025-08-14T21:45:20.5971743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.5972504Z layer_outputs = layer_module( 2025-08-14T21:45:20.5972888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.5973290Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.5973752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.5974321Z sa_output = self.attention( 2025-08-14T21:45:20.5974830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:20.5975396Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:20.5975596Z 2025-08-14T21:45:20.5975719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.5976103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.5976457Z return mod(**inputs) 2025-08-14T21:45:20.5976883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.5977400Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.5977854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.5978352Z return self.transformer( 2025-08-14T21:45:20.5978793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.5979246Z layer_outputs = layer_module( 2025-08-14T21:45:20.5979641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.5980052Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.5980511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.5980963Z sa_output = self.attention( 2025-08-14T21:45:20.5981438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:20.5981959Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.5982157Z 2025-08-14T21:45:20.5982287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.5982677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.5983037Z return mod(**inputs) 2025-08-14T21:45:20.5983462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.5983916Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.5984372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.5984834Z return self.transformer( 2025-08-14T21:45:20.5985274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.5985724Z layer_outputs = layer_module( 2025-08-14T21:45:20.5986117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.5986524Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.5986980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.5987444Z sa_output = self.attention( 2025-08-14T21:45:20.5987886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:20.5988696Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.5988909Z 2025-08-14T21:45:20.5988997Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.5989256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.5989652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.5990005Z return mod(**inputs) 2025-08-14T21:45:20.5990423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.5990874Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.5991358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.5991804Z return self.transformer( 2025-08-14T21:45:20.5992253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.5992722Z layer_outputs = layer_module( 2025-08-14T21:45:20.5993118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.5993545Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.5994012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.5994534Z sa_output = self.attention( 2025-08-14T21:45:20.5994973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:20.5995518Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:20.5995986Z 2025-08-14T21:45:20.5996128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.5996538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.5996895Z return mod(**inputs) 2025-08-14T21:45:20.5997340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.5997792Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.5998221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.5998655Z return self.transformer( 2025-08-14T21:45:20.5999080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.5999510Z layer_outputs = layer_module( 2025-08-14T21:45:20.5999875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6000266Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6000723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6001156Z sa_output = self.attention( 2025-08-14T21:45:20.6001579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:20.6002028Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:20.6002181Z 2025-08-14T21:45:20.6002300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6002679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6003017Z return mod(**inputs) 2025-08-14T21:45:20.6003435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6003881Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6004315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6004777Z return self.transformer( 2025-08-14T21:45:20.6005192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6005632Z layer_outputs = layer_module( 2025-08-14T21:45:20.6005989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6006369Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6006807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6007299Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6007765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6008308Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6009062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6009549Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6010001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:20.6010483Z x = self.lin1(input) 2025-08-14T21:45:20.6010612Z 2025-08-14T21:45:20.6010754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6011130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6011482Z return mod(**inputs) 2025-08-14T21:45:20.6011871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6012291Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6012696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6013110Z return self.transformer( 2025-08-14T21:45:20.6013509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6013945Z layer_outputs = layer_module( 2025-08-14T21:45:20.6014317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6014702Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6015139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6015602Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6016065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6016599Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6017134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6017556Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6017993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:20.6018427Z x = self.activation(x) 2025-08-14T21:45:20.6018777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:20.6019127Z return self.act(input) 2025-08-14T21:45:20.6019242Z 2025-08-14T21:45:20.6019351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6019749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6020066Z return mod(**inputs) 2025-08-14T21:45:20.6020452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6020860Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6021269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6021671Z return self.transformer( 2025-08-14T21:45:20.6022067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6022497Z layer_outputs = layer_module( 2025-08-14T21:45:20.6022837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6023204Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6023644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6024135Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6024599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6025164Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6025707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6026128Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6026556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:20.6026982Z x = self.lin2(x) 2025-08-14T21:45:20.6027090Z 2025-08-14T21:45:20.6027206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6027580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6027927Z return mod(**inputs) 2025-08-14T21:45:20.6028330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6028765Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6029188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6029624Z return self.transformer( 2025-08-14T21:45:20.6030040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6030472Z layer_outputs = layer_module( 2025-08-14T21:45:20.6030834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6031222Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6031660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6032082Z sa_output = self.attention( 2025-08-14T21:45:20.6032503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:20.6032991Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:20.6033181Z 2025-08-14T21:45:20.6033300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6033669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6034012Z return mod(**inputs) 2025-08-14T21:45:20.6034419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6034872Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6035301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6035803Z return self.transformer( 2025-08-14T21:45:20.6036247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6036716Z layer_outputs = layer_module( 2025-08-14T21:45:20.6037093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6037515Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6037965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6038396Z sa_output = self.attention( 2025-08-14T21:45:20.6038851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:20.6039365Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6039550Z 2025-08-14T21:45:20.6039667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6040059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6040386Z return mod(**inputs) 2025-08-14T21:45:20.6040774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6041180Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6041590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6042002Z return self.transformer( 2025-08-14T21:45:20.6042402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6042806Z layer_outputs = layer_module( 2025-08-14T21:45:20.6043153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6043515Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6043949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6044375Z sa_output = self.attention( 2025-08-14T21:45:20.6044809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:20.6045307Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6045494Z 2025-08-14T21:45:20.6045583Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.6045839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6046197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6046520Z return mod(**inputs) 2025-08-14T21:45:20.6046910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6047329Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6047740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6048147Z return self.transformer( 2025-08-14T21:45:20.6048547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6048956Z layer_outputs = layer_module( 2025-08-14T21:45:20.6049303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6049675Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6050086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6050492Z sa_output = self.attention( 2025-08-14T21:45:20.6050882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:20.6051350Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:20.6051564Z 2025-08-14T21:45:20.6051671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6052030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6052347Z return mod(**inputs) 2025-08-14T21:45:20.6052735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6053149Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6053590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6054016Z return self.transformer( 2025-08-14T21:45:20.6054448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6054878Z layer_outputs = layer_module( 2025-08-14T21:45:20.6055234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6055616Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6056028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6056434Z sa_output = self.attention( 2025-08-14T21:45:20.6056823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:20.6057239Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:20.6057377Z 2025-08-14T21:45:20.6057489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6057837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6058162Z return mod(**inputs) 2025-08-14T21:45:20.6058542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6058951Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6059367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6059800Z return self.transformer( 2025-08-14T21:45:20.6060217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6060645Z layer_outputs = layer_module( 2025-08-14T21:45:20.6061008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6061368Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6061779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6062216Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6062657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6063189Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6063728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6064132Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6064569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:20.6065002Z x = self.lin1(input) 2025-08-14T21:45:20.6065116Z 2025-08-14T21:45:20.6065234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6065610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6066716Z return mod(**inputs) 2025-08-14T21:45:20.6067135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6067575Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6068003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6068414Z return self.transformer( 2025-08-14T21:45:20.6068831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6069265Z layer_outputs = layer_module( 2025-08-14T21:45:20.6069639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6070053Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6070508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6070987Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6071469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6072048Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6072593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6073025Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6073487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:20.6073937Z x = self.activation(x) 2025-08-14T21:45:20.6074298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:20.6074679Z return self.act(input) 2025-08-14T21:45:20.6074799Z 2025-08-14T21:45:20.6074920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6075321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6075771Z return mod(**inputs) 2025-08-14T21:45:20.6076217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6076680Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6077116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6077564Z return self.transformer( 2025-08-14T21:45:20.6078005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6078450Z layer_outputs = layer_module( 2025-08-14T21:45:20.6078820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6079216Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6079668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6080172Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6080662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6081241Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6081802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6082229Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6082696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:20.6083131Z x = self.lin2(x) 2025-08-14T21:45:20.6083242Z 2025-08-14T21:45:20.6083363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6083746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6084097Z return mod(**inputs) 2025-08-14T21:45:20.6084529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6084975Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6085445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6085889Z return self.transformer( 2025-08-14T21:45:20.6086326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6086773Z layer_outputs = layer_module( 2025-08-14T21:45:20.6087161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6087563Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6088031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6088464Z sa_output = self.attention( 2025-08-14T21:45:20.6088894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:20.6089389Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:20.6089580Z 2025-08-14T21:45:20.6089701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6090080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6090427Z return mod(**inputs) 2025-08-14T21:45:20.6090839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6091273Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6091712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6092153Z return self.transformer( 2025-08-14T21:45:20.6092576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6093006Z layer_outputs = layer_module( 2025-08-14T21:45:20.6093376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6093764Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6094200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6094638Z sa_output = self.attention( 2025-08-14T21:45:20.6095066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:20.6095567Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6095748Z 2025-08-14T21:45:20.6095859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6096238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6096582Z return mod(**inputs) 2025-08-14T21:45:20.6096988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6097412Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6097863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6098297Z return self.transformer( 2025-08-14T21:45:20.6098724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6099161Z layer_outputs = layer_module( 2025-08-14T21:45:20.6099529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6099942Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6100370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6100821Z sa_output = self.attention( 2025-08-14T21:45:20.6101245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:20.6101730Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6101922Z 2025-08-14T21:45:20.6102009Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.6102262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6102643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6102978Z return mod(**inputs) 2025-08-14T21:45:20.6103385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6103829Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6104282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6104710Z return self.transformer( 2025-08-14T21:45:20.6105128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6105556Z layer_outputs = layer_module( 2025-08-14T21:45:20.6105914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6106302Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6106745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6107187Z sa_output = self.attention( 2025-08-14T21:45:20.6107614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:20.6108115Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:20.6108310Z 2025-08-14T21:45:20.6108426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6108963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6109309Z return mod(**inputs) 2025-08-14T21:45:20.6109721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6110179Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6110669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6111112Z return self.transformer( 2025-08-14T21:45:20.6111547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6111982Z layer_outputs = layer_module( 2025-08-14T21:45:20.6112343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6112737Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6113248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6113688Z sa_output = self.attention( 2025-08-14T21:45:20.6114124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:20.6114570Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:20.6114716Z 2025-08-14T21:45:20.6114835Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6115232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6115583Z return mod(**inputs) 2025-08-14T21:45:20.6116116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6116570Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6117014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6117446Z return self.transformer( 2025-08-14T21:45:20.6117852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6118267Z layer_outputs = layer_module( 2025-08-14T21:45:20.6118619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6118991Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6119411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6119865Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6120316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6120862Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6121384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6121792Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6122234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:20.6122674Z x = self.lin1(input) 2025-08-14T21:45:20.6122788Z 2025-08-14T21:45:20.6122902Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6123287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6123634Z return mod(**inputs) 2025-08-14T21:45:20.6124059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6124519Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6124959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6125407Z return self.transformer( 2025-08-14T21:45:20.6125836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6126264Z layer_outputs = layer_module( 2025-08-14T21:45:20.6126635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6127030Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6127485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6127971Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6128484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6128995Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6129483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6129935Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6130393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:20.6130827Z x = self.activation(x) 2025-08-14T21:45:20.6131188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:20.6131558Z return self.act(input) 2025-08-14T21:45:20.6131676Z 2025-08-14T21:45:20.6131797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6132182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6132537Z return mod(**inputs) 2025-08-14T21:45:20.6132910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6133306Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6133693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6134089Z return self.transformer( 2025-08-14T21:45:20.6134495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6134922Z layer_outputs = layer_module( 2025-08-14T21:45:20.6135288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6135669Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6136105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6136571Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6137038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6137599Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6138140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6138553Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6138985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:20.6139408Z x = self.lin2(x) 2025-08-14T21:45:20.6139511Z 2025-08-14T21:45:20.6139621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6140005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6140365Z return mod(**inputs) 2025-08-14T21:45:20.6140777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6141213Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6141640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6142064Z return self.transformer( 2025-08-14T21:45:20.6142486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6142923Z layer_outputs = layer_module( 2025-08-14T21:45:20.6143291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6143683Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6144109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6144544Z sa_output = self.attention( 2025-08-14T21:45:20.6145034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:20.6145529Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:20.6145725Z 2025-08-14T21:45:20.6145862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6146244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6146591Z return mod(**inputs) 2025-08-14T21:45:20.6147002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6147454Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6147894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6148330Z return self.transformer( 2025-08-14T21:45:20.6148752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6149186Z layer_outputs = layer_module( 2025-08-14T21:45:20.6149561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6149929Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6150336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6150745Z sa_output = self.attention( 2025-08-14T21:45:20.6151143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:20.6151592Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6151784Z 2025-08-14T21:45:20.6151888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6152248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6152574Z return mod(**inputs) 2025-08-14T21:45:20.6152976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6153412Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6153841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6154275Z return self.transformer( 2025-08-14T21:45:20.6154693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6155141Z layer_outputs = layer_module( 2025-08-14T21:45:20.6155534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6156006Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6156464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6156910Z sa_output = self.attention( 2025-08-14T21:45:20.6157339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:20.6157793Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6158007Z 2025-08-14T21:45:20.6158090Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.6158334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6158697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6159014Z return mod(**inputs) 2025-08-14T21:45:20.6159395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6159802Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6160215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6160625Z return self.transformer( 2025-08-14T21:45:20.6161035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6161442Z layer_outputs = layer_module( 2025-08-14T21:45:20.6161777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6162136Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6162544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6162940Z sa_output = self.attention( 2025-08-14T21:45:20.6163338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:20.6163803Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:20.6163984Z 2025-08-14T21:45:20.6164096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6164442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6164767Z return mod(**inputs) 2025-08-14T21:45:20.6165155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6165572Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6165972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6166402Z return self.transformer( 2025-08-14T21:45:20.6166824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6167222Z layer_outputs = layer_module( 2025-08-14T21:45:20.6167564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6167928Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6168364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6168783Z sa_output = self.attention( 2025-08-14T21:45:20.6169180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:20.6169633Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:20.6169815Z 2025-08-14T21:45:20.6169941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6170290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6170612Z return mod(**inputs) 2025-08-14T21:45:20.6170996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6171401Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6171805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6172228Z return self.transformer( 2025-08-14T21:45:20.6172623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6173033Z layer_outputs = layer_module( 2025-08-14T21:45:20.6173379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6173745Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6174168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6174622Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6175083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6175619Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6176133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6176530Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6176962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:20.6177394Z x = self.lin1(input) 2025-08-14T21:45:20.6177506Z 2025-08-14T21:45:20.6177617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6178005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6178329Z return mod(**inputs) 2025-08-14T21:45:20.6178717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6179148Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6179579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6180007Z return self.transformer( 2025-08-14T21:45:20.6180413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6180849Z layer_outputs = layer_module( 2025-08-14T21:45:20.6181219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6181609Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6182040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6182520Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6183004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6183560Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6184107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6184540Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6184969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:20.6185399Z x = self.activation(x) 2025-08-14T21:45:20.6185747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:20.6186117Z return self.act(input) 2025-08-14T21:45:20.6186237Z 2025-08-14T21:45:20.6186357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6186732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6187089Z return mod(**inputs) 2025-08-14T21:45:20.6187503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6187938Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6188373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6188811Z return self.transformer( 2025-08-14T21:45:20.6189249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6189674Z layer_outputs = layer_module( 2025-08-14T21:45:20.6190057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6190446Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6190877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6191357Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6191833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6192407Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6192958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6193386Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6193837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:20.6194282Z x = self.lin2(x) 2025-08-14T21:45:20.6194392Z 2025-08-14T21:45:20.6194504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6194894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6195246Z return mod(**inputs) 2025-08-14T21:45:20.6195740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6196188Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6196629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6197084Z return self.transformer( 2025-08-14T21:45:20.6197505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6197946Z layer_outputs = layer_module( 2025-08-14T21:45:20.6198324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6198720Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6199159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6199600Z sa_output = self.attention( 2025-08-14T21:45:20.6200063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:20.6200554Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:20.6200754Z 2025-08-14T21:45:20.6200867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6201254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6201695Z return mod(**inputs) 2025-08-14T21:45:20.6202104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6202644Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6203086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6203529Z return self.transformer( 2025-08-14T21:45:20.6203954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6204400Z layer_outputs = layer_module( 2025-08-14T21:45:20.6204797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6205186Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6205655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6206098Z sa_output = self.attention( 2025-08-14T21:45:20.6206532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:20.6207023Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6207221Z 2025-08-14T21:45:20.6207335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6207735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6208079Z return mod(**inputs) 2025-08-14T21:45:20.6208481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6209072Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6209515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6209940Z return self.transformer( 2025-08-14T21:45:20.6210364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6210804Z layer_outputs = layer_module( 2025-08-14T21:45:20.6211166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6211548Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6211987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6212421Z sa_output = self.attention( 2025-08-14T21:45:20.6212836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:20.6213312Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6213509Z 2025-08-14T21:45:20.6213596Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.6213852Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6214224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6214563Z return mod(**inputs) 2025-08-14T21:45:20.6214970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6215459Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6215882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6216312Z return self.transformer( 2025-08-14T21:45:20.6216728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6217147Z layer_outputs = layer_module( 2025-08-14T21:45:20.6217513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6217926Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6218364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6218784Z sa_output = self.attention( 2025-08-14T21:45:20.6219205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:20.6219699Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:20.6219920Z 2025-08-14T21:45:20.6220038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6220410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6220789Z return mod(**inputs) 2025-08-14T21:45:20.6221203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6221609Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6222015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6222421Z return self.transformer( 2025-08-14T21:45:20.6222822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6223235Z layer_outputs = layer_module( 2025-08-14T21:45:20.6223602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6223986Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6224396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6224807Z sa_output = self.attention( 2025-08-14T21:45:20.6225230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:20.6225701Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:20.6225848Z 2025-08-14T21:45:20.6225958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6226341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6226687Z return mod(**inputs) 2025-08-14T21:45:20.6227114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6227530Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6227953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6228373Z return self.transformer( 2025-08-14T21:45:20.6228789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6229221Z layer_outputs = layer_module( 2025-08-14T21:45:20.6229587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6229993Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6230432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6230918Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6231401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6231983Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6232521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6232956Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6233388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:20.6233814Z x = self.lin1(input) 2025-08-14T21:45:20.6233925Z 2025-08-14T21:45:20.6234034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6234409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6234766Z return mod(**inputs) 2025-08-14T21:45:20.6235166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6235617Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6236118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6236558Z return self.transformer( 2025-08-14T21:45:20.6236978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6237424Z layer_outputs = layer_module( 2025-08-14T21:45:20.6237790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6238164Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6238602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6239075Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6239544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6240101Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6240649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6241065Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6241500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:20.6241926Z x = self.activation(x) 2025-08-14T21:45:20.6242274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:20.6242637Z return self.act(input) 2025-08-14T21:45:20.6242753Z 2025-08-14T21:45:20.6242865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6243250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6243595Z return mod(**inputs) 2025-08-14T21:45:20.6244007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6244432Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6244859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6245320Z return self.transformer( 2025-08-14T21:45:20.6245746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6246184Z layer_outputs = layer_module( 2025-08-14T21:45:20.6246548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6246944Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6247372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6247871Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6248347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6248920Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6249459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6249908Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6250352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:20.6250777Z x = self.lin2(x) 2025-08-14T21:45:20.6250897Z 2025-08-14T21:45:20.6251008Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6251385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6251728Z return mod(**inputs) 2025-08-14T21:45:20.6252136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6252579Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6253005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6253444Z return self.transformer( 2025-08-14T21:45:20.6253866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6254294Z layer_outputs = layer_module( 2025-08-14T21:45:20.6254658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6255032Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6255461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6255866Z sa_output = self.attention( 2025-08-14T21:45:20.6256262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:20.6256715Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:20.6256902Z 2025-08-14T21:45:20.6257005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6257358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6257681Z return mod(**inputs) 2025-08-14T21:45:20.6258080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6258513Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6258940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6259358Z return self.transformer( 2025-08-14T21:45:20.6259777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6260202Z layer_outputs = layer_module( 2025-08-14T21:45:20.6260546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6260904Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6261311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6261719Z sa_output = self.attention( 2025-08-14T21:45:20.6262111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:20.6262581Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6262761Z 2025-08-14T21:45:20.6262866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6263227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6263562Z return mod(**inputs) 2025-08-14T21:45:20.6263966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6264415Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6264844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6265289Z return self.transformer( 2025-08-14T21:45:20.6265711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6266146Z layer_outputs = layer_module( 2025-08-14T21:45:20.6266506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6266898Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6267330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6267739Z sa_output = self.attention( 2025-08-14T21:45:20.6268131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:20.6268618Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:20.6268816Z 2025-08-14T21:45:20.6268905Z cudagraph partition due to non gpu ops 2025-08-14T21:45:20.6269162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6269535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6269881Z return mod(**inputs) 2025-08-14T21:45:20.6270298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6270721Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6271158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6271598Z return self.transformer( 2025-08-14T21:45:20.6272025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6272451Z layer_outputs = layer_module( 2025-08-14T21:45:20.6272819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6273212Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6273663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6274124Z sa_output = self.attention( 2025-08-14T21:45:20.6274568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:20.6275102Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:20.6275304Z 2025-08-14T21:45:20.6275415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6275899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6276256Z return mod(**inputs) 2025-08-14T21:45:20.6276678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6277121Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6277570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6278000Z return self.transformer( 2025-08-14T21:45:20.6278408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6278837Z layer_outputs = layer_module( 2025-08-14T21:45:20.6279203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6279618Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6280052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:20.6280496Z sa_output = self.attention( 2025-08-14T21:45:20.6280915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:20.6281359Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:20.6281506Z 2025-08-14T21:45:20.6281614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6281996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6282347Z return mod(**inputs) 2025-08-14T21:45:20.6282746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6283179Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6283609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6284039Z return self.transformer( 2025-08-14T21:45:20.6284448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6284883Z layer_outputs = layer_module( 2025-08-14T21:45:20.6285245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6285622Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6286056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6286526Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6286995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6287551Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6288095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6288513Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6288947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:20.6289366Z x = self.lin1(input) 2025-08-14T21:45:20.6289485Z 2025-08-14T21:45:20.6289595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6289993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6290334Z return mod(**inputs) 2025-08-14T21:45:20.6290729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6291161Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6291587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6292009Z return self.transformer( 2025-08-14T21:45:20.6292441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6292871Z layer_outputs = layer_module( 2025-08-14T21:45:20.6293236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6293613Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6294052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6294545Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6295006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6295585Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6296128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6296542Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6296967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:20.6297398Z x = self.activation(x) 2025-08-14T21:45:20.6297741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:20.6298097Z return self.act(input) 2025-08-14T21:45:20.6298213Z 2025-08-14T21:45:20.6298325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6298705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6299047Z return mod(**inputs) 2025-08-14T21:45:20.6299446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:45:20.6299880Z dlbrt_output = self.distilbert( 2025-08-14T21:45:20.6300307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:20.6300739Z return self.transformer( 2025-08-14T21:45:20.6301147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:20.6301581Z layer_outputs = layer_module( 2025-08-14T21:45:20.6301949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:20.6302331Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:20.6302760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:20.6303228Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:20.6303697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:20.6304257Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:20.6304802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:20.6305238Z return forward_fn(*input_tensors) 2025-08-14T21:45:20.6305685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:20.6306106Z x = self.lin2(x) 2025-08-14T21:45:20.6306220Z 2025-08-14T21:45:20.6306330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6306715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6307059Z return mod(**inputs) 2025-08-14T21:45:20.6307475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 836, in forward 2025-08-14T21:45:20.6307993Z prediction_logits = self.vocab_transform(hidden_states) # (bs, seq_length, dim) 2025-08-14T21:45:20.6308221Z 2025-08-14T21:45:20.6308338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6308841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6309189Z return mod(**inputs) 2025-08-14T21:45:20.6309647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 839, in forward 2025-08-14T21:45:20.6310221Z prediction_logits = self.vocab_projector(prediction_logits) # (bs, seq_length, vocab_size) 2025-08-14T21:45:20.6310523Z 2025-08-14T21:45:20.6310634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:20.6311010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:20.6311351Z return mod(**inputs) 2025-08-14T21:45:20.6311761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 843, in forward 2025-08-14T21:45:20.6312326Z mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1)) 2025-08-14T21:45:20.6312599Z 2025-08-14T21:45:27.8612751Z Compilation time (from dynamo_timed): 11.159708823 2025-08-14T21:45:27.8616873Z pass 2025-08-14T21:45:27.8623585Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:27.8624415Z TIMING: _recursive_pre_grad_passes:0.00508 _recursive_joint_graph_passes:0.25885 _recursive_post_grad_passes:0.05785 async_compile.wait:0.78086 code_gen:6.9238 inductor_compile:7.90671 backend_compile:9.71923 gc:0.00204 entire_frame_compile:11.15971 total_wall_time:11.15971 2025-08-14T21:45:27.8625447Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:6660 | FakeTensor.__torch_dispatch__:2532 | ProxyTorchDispatchMode.__torch_dispatch__:2359 2025-08-14T21:45:27.8625951Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique) 2025-08-14T21:45:33.0375011Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:45:33.0376016Z from pkg_resources import resource_filename 2025-08-14T21:45:33.6384624Z 2025-08-14T21:45:34.2055682Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:45:34.2055983Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:45:34.2064595Z cpu eval DistilBertForQuestionAnswering 2025-08-14T21:45:34.5438803Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:34.5936025Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:34.6709788Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:39.7134394Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7137171Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7137566Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7137810Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7138042Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7138303Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7138566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7139004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7139387Z return mod(**inputs) 2025-08-14T21:45:39.7139870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7140799Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7141274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7141754Z return self.transformer( 2025-08-14T21:45:39.7142223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7142667Z layer_outputs = layer_module( 2025-08-14T21:45:39.7143115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7143516Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7144028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7144476Z sa_output = self.attention( 2025-08-14T21:45:39.7144904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:39.7145399Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:39.7145606Z 2025-08-14T21:45:39.7145723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7146112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7146473Z return mod(**inputs) 2025-08-14T21:45:39.7146896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7147354Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7147799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7148238Z return self.transformer( 2025-08-14T21:45:39.7148663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7149103Z layer_outputs = layer_module( 2025-08-14T21:45:39.7149474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7149863Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7150311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7150752Z sa_output = self.attention( 2025-08-14T21:45:39.7151182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:39.7154590Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7154793Z 2025-08-14T21:45:39.7154915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7155313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7155953Z return mod(**inputs) 2025-08-14T21:45:39.7156432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7156941Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7157385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7157821Z return self.transformer( 2025-08-14T21:45:39.7158252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7158707Z layer_outputs = layer_module( 2025-08-14T21:45:39.7159082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7159554Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7160002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7160439Z sa_output = self.attention( 2025-08-14T21:45:39.7160864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:39.7161329Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7161537Z 2025-08-14T21:45:39.7161630Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7161870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7162266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7162596Z return mod(**inputs) 2025-08-14T21:45:39.7162991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7163408Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7163827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7164240Z return self.transformer( 2025-08-14T21:45:39.7164634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7165047Z layer_outputs = layer_module( 2025-08-14T21:45:39.7165396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7165763Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7166170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7166586Z sa_output = self.attention( 2025-08-14T21:45:39.7166978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:39.7167440Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:39.7167631Z 2025-08-14T21:45:39.7167736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7168092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7168412Z return mod(**inputs) 2025-08-14T21:45:39.7168822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7169269Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7169713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7170206Z return self.transformer( 2025-08-14T21:45:39.7170596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7171009Z layer_outputs = layer_module( 2025-08-14T21:45:39.7171358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7171719Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7172139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7172552Z sa_output = self.attention( 2025-08-14T21:45:39.7172951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:39.7173383Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:39.7173545Z 2025-08-14T21:45:39.7173683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7174060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7174400Z return mod(**inputs) 2025-08-14T21:45:39.7174782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7175198Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7175629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7176037Z return self.transformer( 2025-08-14T21:45:39.7176421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7176858Z layer_outputs = layer_module( 2025-08-14T21:45:39.7177216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7177568Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7177971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7178425Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7178878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7179430Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7179960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7180366Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7180780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:39.7181198Z x = self.lin1(input) 2025-08-14T21:45:39.7181311Z 2025-08-14T21:45:39.7181415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7181778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7182111Z return mod(**inputs) 2025-08-14T21:45:39.7182533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7183012Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7183465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7183909Z return self.transformer( 2025-08-14T21:45:39.7184333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7184819Z layer_outputs = layer_module( 2025-08-14T21:45:39.7185164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7185529Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7185960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7186448Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7186896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7187573Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7188093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7188494Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7189724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:39.7190142Z x = self.activation(x) 2025-08-14T21:45:39.7190478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:39.7190848Z return self.act(input) 2025-08-14T21:45:39.7190975Z 2025-08-14T21:45:39.7191086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7191505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7191856Z return mod(**inputs) 2025-08-14T21:45:39.7192316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7192779Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7193232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7193681Z return self.transformer( 2025-08-14T21:45:39.7194116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7194562Z layer_outputs = layer_module( 2025-08-14T21:45:39.7194962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7195348Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7195909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7196428Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7196934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7197493Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7198045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7198462Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7198896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:39.7199321Z x = self.lin2(x) 2025-08-14T21:45:39.7199435Z 2025-08-14T21:45:39.7199547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7199930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7200267Z return mod(**inputs) 2025-08-14T21:45:39.7200726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7201169Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7201606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7202033Z return self.transformer( 2025-08-14T21:45:39.7202450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7202882Z layer_outputs = layer_module( 2025-08-14T21:45:39.7203252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7203639Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7204078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7204523Z sa_output = self.attention( 2025-08-14T21:45:39.7204915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:39.7205414Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:39.7205598Z 2025-08-14T21:45:39.7205701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7206055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7206362Z return mod(**inputs) 2025-08-14T21:45:39.7206762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7207169Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7207578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7207973Z return self.transformer( 2025-08-14T21:45:39.7208356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7209067Z layer_outputs = layer_module( 2025-08-14T21:45:39.7209403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7209763Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7210169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7210571Z sa_output = self.attention( 2025-08-14T21:45:39.7210957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:39.7211409Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7211587Z 2025-08-14T21:45:39.7211697Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7212043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7212363Z return mod(**inputs) 2025-08-14T21:45:39.7212740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7213169Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7213578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7213991Z return self.transformer( 2025-08-14T21:45:39.7214396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7214812Z layer_outputs = layer_module( 2025-08-14T21:45:39.7215145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7215591Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7215995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7216388Z sa_output = self.attention( 2025-08-14T21:45:39.7216774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:39.7217244Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7217419Z 2025-08-14T21:45:39.7217508Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7217738Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7218086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7218407Z return mod(**inputs) 2025-08-14T21:45:39.7218781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7219224Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7219630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7220036Z return self.transformer( 2025-08-14T21:45:39.7220427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7220838Z layer_outputs = layer_module( 2025-08-14T21:45:39.7221214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7221584Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7222014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7222411Z sa_output = self.attention( 2025-08-14T21:45:39.7222799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:39.7223325Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:39.7223520Z 2025-08-14T21:45:39.7223627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7223991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7224309Z return mod(**inputs) 2025-08-14T21:45:39.7224686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7225098Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7225510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7225907Z return self.transformer( 2025-08-14T21:45:39.7226299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7226708Z layer_outputs = layer_module( 2025-08-14T21:45:39.7227053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7227404Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7227816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7228228Z sa_output = self.attention( 2025-08-14T21:45:39.7228656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:39.7229080Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:39.7229287Z 2025-08-14T21:45:39.7229389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7229749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7230079Z return mod(**inputs) 2025-08-14T21:45:39.7230498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7230948Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7231382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7231814Z return self.transformer( 2025-08-14T21:45:39.7232241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7232729Z layer_outputs = layer_module( 2025-08-14T21:45:39.7233095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7233475Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7233934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7234425Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7234902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7235517Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7236246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7236697Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7237143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:39.7237585Z x = self.lin1(input) 2025-08-14T21:45:39.7237700Z 2025-08-14T21:45:39.7237820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7238207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7238546Z return mod(**inputs) 2025-08-14T21:45:39.7238969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7239421Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7239862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7240297Z return self.transformer( 2025-08-14T21:45:39.7240723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7241159Z layer_outputs = layer_module( 2025-08-14T21:45:39.7241519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7241903Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7242340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7242815Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7243281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7243848Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7244391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7244822Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7245252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:39.7245691Z x = self.activation(x) 2025-08-14T21:45:39.7246037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:39.7246387Z return self.act(input) 2025-08-14T21:45:39.7246511Z 2025-08-14T21:45:39.7246620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7247000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7247341Z return mod(**inputs) 2025-08-14T21:45:39.7247722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7248138Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7248555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7248977Z return self.transformer( 2025-08-14T21:45:39.7249373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7249781Z layer_outputs = layer_module( 2025-08-14T21:45:39.7250130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7250487Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7250941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7251390Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7251847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7252384Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7252898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7253291Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7253694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:39.7254096Z x = self.lin2(x) 2025-08-14T21:45:39.7254203Z 2025-08-14T21:45:39.7254307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7254667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7254985Z return mod(**inputs) 2025-08-14T21:45:39.7255374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7255791Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7256199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7256605Z return self.transformer( 2025-08-14T21:45:39.7257005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7257417Z layer_outputs = layer_module( 2025-08-14T21:45:39.7257756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7258121Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7258538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7258944Z sa_output = self.attention( 2025-08-14T21:45:39.7259360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:39.7259817Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:39.7259995Z 2025-08-14T21:45:39.7260105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7260453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7260773Z return mod(**inputs) 2025-08-14T21:45:39.7261160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7261579Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7261990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7262398Z return self.transformer( 2025-08-14T21:45:39.7262798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7263236Z layer_outputs = layer_module( 2025-08-14T21:45:39.7263577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7263946Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7264348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7264739Z sa_output = self.attention( 2025-08-14T21:45:39.7265149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:39.7265593Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7265768Z 2025-08-14T21:45:39.7265896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7266252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7266582Z return mod(**inputs) 2025-08-14T21:45:39.7266972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7267395Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7267812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7268215Z return self.transformer( 2025-08-14T21:45:39.7268616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7269018Z layer_outputs = layer_module( 2025-08-14T21:45:39.7269365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7269733Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7270148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7270575Z sa_output = self.attention( 2025-08-14T21:45:39.7270971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:39.7271427Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7271604Z 2025-08-14T21:45:39.7271686Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7271928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7272291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7272615Z return mod(**inputs) 2025-08-14T21:45:39.7273002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7273449Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7273862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7274266Z return self.transformer( 2025-08-14T21:45:39.7274674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7275227Z layer_outputs = layer_module( 2025-08-14T21:45:39.7275603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7276100Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7276568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7277057Z sa_output = self.attention( 2025-08-14T21:45:39.7277446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:39.7277924Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:39.7278111Z 2025-08-14T21:45:39.7278211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7278569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7278887Z return mod(**inputs) 2025-08-14T21:45:39.7279277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7279715Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7280135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7280562Z return self.transformer( 2025-08-14T21:45:39.7280969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7281374Z layer_outputs = layer_module( 2025-08-14T21:45:39.7281722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7282074Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7282482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7282886Z sa_output = self.attention( 2025-08-14T21:45:39.7283279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:39.7283715Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:39.7283869Z 2025-08-14T21:45:39.7283979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7284357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7284697Z return mod(**inputs) 2025-08-14T21:45:39.7285115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7285578Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7286013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7286434Z return self.transformer( 2025-08-14T21:45:39.7286849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7287273Z layer_outputs = layer_module( 2025-08-14T21:45:39.7287629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7288043Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7288489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7288975Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7289450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7290033Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7290581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7291001Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7291436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:39.7291882Z x = self.lin1(input) 2025-08-14T21:45:39.7291996Z 2025-08-14T21:45:39.7292119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7292527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7292905Z return mod(**inputs) 2025-08-14T21:45:39.7293327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7293780Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7294221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7294695Z return self.transformer( 2025-08-14T21:45:39.7295113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7295565Z layer_outputs = layer_module( 2025-08-14T21:45:39.7295944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7296337Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7296793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7297280Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7297772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7298363Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7298918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7299348Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7299799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:39.7300248Z x = self.activation(x) 2025-08-14T21:45:39.7300607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:39.7300981Z return self.act(input) 2025-08-14T21:45:39.7301111Z 2025-08-14T21:45:39.7301224Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7301617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7301978Z return mod(**inputs) 2025-08-14T21:45:39.7302402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7302861Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7303315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7303779Z return self.transformer( 2025-08-14T21:45:39.7304212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7304661Z layer_outputs = layer_module( 2025-08-14T21:45:39.7305031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7305421Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7305874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7306366Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7306846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7307422Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7308019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7308445Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7309065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:39.7309518Z x = self.lin2(x) 2025-08-14T21:45:39.7309628Z 2025-08-14T21:45:39.7309752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7310214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7310571Z return mod(**inputs) 2025-08-14T21:45:39.7311017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7311481Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7311923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7312371Z return self.transformer( 2025-08-14T21:45:39.7312804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7313250Z layer_outputs = layer_module( 2025-08-14T21:45:39.7313619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7314011Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7314460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7314895Z sa_output = self.attention( 2025-08-14T21:45:39.7315329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:39.7315971Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:39.7316176Z 2025-08-14T21:45:39.7316297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7316755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7317116Z return mod(**inputs) 2025-08-14T21:45:39.7317542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7317998Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7318423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7318826Z return self.transformer( 2025-08-14T21:45:39.7319224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7319671Z layer_outputs = layer_module( 2025-08-14T21:45:39.7320018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7320376Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7320784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7321183Z sa_output = self.attention( 2025-08-14T21:45:39.7321579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:39.7322025Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7322194Z 2025-08-14T21:45:39.7322300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7322649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7322976Z return mod(**inputs) 2025-08-14T21:45:39.7323390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7323799Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7324213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7324622Z return self.transformer( 2025-08-14T21:45:39.7325011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7325429Z layer_outputs = layer_module( 2025-08-14T21:45:39.7325779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7326150Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7326551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7326951Z sa_output = self.attention( 2025-08-14T21:45:39.7327339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:39.7327790Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7327967Z 2025-08-14T21:45:39.7328049Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7328288Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7328647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7328970Z return mod(**inputs) 2025-08-14T21:45:39.7329356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7329776Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7330195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7330601Z return self.transformer( 2025-08-14T21:45:39.7330990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7331388Z layer_outputs = layer_module( 2025-08-14T21:45:39.7331728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7332074Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7332479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7332886Z sa_output = self.attention( 2025-08-14T21:45:39.7333279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:39.7333778Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:39.7333970Z 2025-08-14T21:45:39.7334071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7334431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7334750Z return mod(**inputs) 2025-08-14T21:45:39.7335142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7335564Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7335967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7336357Z return self.transformer( 2025-08-14T21:45:39.7336753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7337176Z layer_outputs = layer_module( 2025-08-14T21:45:39.7337517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7337879Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7338294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7338702Z sa_output = self.attention( 2025-08-14T21:45:39.7339141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:39.7339580Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:39.7339726Z 2025-08-14T21:45:39.7339843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7340239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7340560Z return mod(**inputs) 2025-08-14T21:45:39.7340951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7341369Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7341777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7342185Z return self.transformer( 2025-08-14T21:45:39.7342581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7342989Z layer_outputs = layer_module( 2025-08-14T21:45:39.7343348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7343733Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7344174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7344643Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7345119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7345690Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7346202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7346589Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7346999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:39.7347419Z x = self.lin1(input) 2025-08-14T21:45:39.7347528Z 2025-08-14T21:45:39.7347663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7348013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7348341Z return mod(**inputs) 2025-08-14T21:45:39.7348723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7349134Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7349550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7349965Z return self.transformer( 2025-08-14T21:45:39.7350383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7350799Z layer_outputs = layer_module( 2025-08-14T21:45:39.7351158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7351543Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7352002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7352465Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7352934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7353494Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7354056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7354467Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7354917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:39.7355358Z x = self.activation(x) 2025-08-14T21:45:39.7355773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:39.7356177Z return self.act(input) 2025-08-14T21:45:39.7356299Z 2025-08-14T21:45:39.7356422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7356818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7357162Z return mod(**inputs) 2025-08-14T21:45:39.7357591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7358035Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7358475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7358911Z return self.transformer( 2025-08-14T21:45:39.7359328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7359761Z layer_outputs = layer_module( 2025-08-14T21:45:39.7360121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7360500Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7360936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7361403Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7361865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7362429Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7363013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7363420Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7363855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:39.7364266Z x = self.lin2(x) 2025-08-14T21:45:39.7364365Z 2025-08-14T21:45:39.7364475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7364839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7365178Z return mod(**inputs) 2025-08-14T21:45:39.7365585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7366011Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7366419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7366848Z return self.transformer( 2025-08-14T21:45:39.7367237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7367633Z layer_outputs = layer_module( 2025-08-14T21:45:39.7367978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7368339Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7368768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7369170Z sa_output = self.attention( 2025-08-14T21:45:39.7369594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:39.7370062Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:39.7370241Z 2025-08-14T21:45:39.7370353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7370709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7371035Z return mod(**inputs) 2025-08-14T21:45:39.7371427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7371838Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7372250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7372655Z return self.transformer( 2025-08-14T21:45:39.7373049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7373449Z layer_outputs = layer_module( 2025-08-14T21:45:39.7373792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7374157Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7374563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7374971Z sa_output = self.attention( 2025-08-14T21:45:39.7375363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:39.7375819Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7375991Z 2025-08-14T21:45:39.7376094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7376454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7376802Z return mod(**inputs) 2025-08-14T21:45:39.7377194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7377612Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7378033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7378449Z return self.transformer( 2025-08-14T21:45:39.7378843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7379258Z layer_outputs = layer_module( 2025-08-14T21:45:39.7379618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7379988Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7380408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7380831Z sa_output = self.attention( 2025-08-14T21:45:39.7381254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:39.7381715Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7381891Z 2025-08-14T21:45:39.7381972Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7382211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7382571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7382903Z return mod(**inputs) 2025-08-14T21:45:39.7383298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7383745Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7384193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7384593Z return self.transformer( 2025-08-14T21:45:39.7384990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7385400Z layer_outputs = layer_module( 2025-08-14T21:45:39.7385743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7386109Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7386526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7386940Z sa_output = self.attention( 2025-08-14T21:45:39.7387334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:39.7387811Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:39.7388001Z 2025-08-14T21:45:39.7388114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7388475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7388796Z return mod(**inputs) 2025-08-14T21:45:39.7389188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7389612Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7390020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7390428Z return self.transformer( 2025-08-14T21:45:39.7390842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7391292Z layer_outputs = layer_module( 2025-08-14T21:45:39.7391652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7392036Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7392468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7392888Z sa_output = self.attention( 2025-08-14T21:45:39.7393315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:39.7393772Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:39.7393923Z 2025-08-14T21:45:39.7394043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7394424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7394776Z return mod(**inputs) 2025-08-14T21:45:39.7395202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7395755Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7396212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7396654Z return self.transformer( 2025-08-14T21:45:39.7397085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7397509Z layer_outputs = layer_module( 2025-08-14T21:45:39.7397903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7398293Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7398761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7399232Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7399707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7400279Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7400827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7401241Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7401686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:39.7402126Z x = self.lin1(input) 2025-08-14T21:45:39.7402239Z 2025-08-14T21:45:39.7402351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7402738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7403088Z return mod(**inputs) 2025-08-14T21:45:39.7403497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7403937Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7404385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7404820Z return self.transformer( 2025-08-14T21:45:39.7405240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7405666Z layer_outputs = layer_module( 2025-08-14T21:45:39.7406033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7406462Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7406889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7407365Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7407833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7408394Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7409081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7409502Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7409941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:39.7410378Z x = self.activation(x) 2025-08-14T21:45:39.7410718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:39.7411123Z return self.act(input) 2025-08-14T21:45:39.7411240Z 2025-08-14T21:45:39.7411358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7411736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7412078Z return mod(**inputs) 2025-08-14T21:45:39.7412483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7412955Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7413384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7413808Z return self.transformer( 2025-08-14T21:45:39.7414223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7414635Z layer_outputs = layer_module( 2025-08-14T21:45:39.7414978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7415336Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7415746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7416183Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7416632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7417165Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7417676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7418068Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7418477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:39.7418881Z x = self.lin2(x) 2025-08-14T21:45:39.7418981Z 2025-08-14T21:45:39.7419094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7419462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7419778Z return mod(**inputs) 2025-08-14T21:45:39.7420159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7420558Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7420968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7421396Z return self.transformer( 2025-08-14T21:45:39.7421781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7422170Z layer_outputs = layer_module( 2025-08-14T21:45:39.7422505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7422858Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7423264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7423660Z sa_output = self.attention( 2025-08-14T21:45:39.7424053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:45:39.7424515Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:45:39.7424695Z 2025-08-14T21:45:39.7424798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7425177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7425501Z return mod(**inputs) 2025-08-14T21:45:39.7425892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7426308Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7426708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7427129Z return self.transformer( 2025-08-14T21:45:39.7427521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7427947Z layer_outputs = layer_module( 2025-08-14T21:45:39.7428301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7428664Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7429072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7429481Z sa_output = self.attention( 2025-08-14T21:45:39.7429876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:45:39.7430332Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7430504Z 2025-08-14T21:45:39.7430605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7430957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7431275Z return mod(**inputs) 2025-08-14T21:45:39.7431657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7432100Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7432535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7432961Z return self.transformer( 2025-08-14T21:45:39.7433370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7433799Z layer_outputs = layer_module( 2025-08-14T21:45:39.7434165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7434536Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7434975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7435422Z sa_output = self.attention( 2025-08-14T21:45:39.7435911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:45:39.7436407Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:45:39.7436604Z 2025-08-14T21:45:39.7436691Z cudagraph partition due to non gpu ops 2025-08-14T21:45:39.7436947Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7437342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7437665Z return mod(**inputs) 2025-08-14T21:45:39.7438064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7438523Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7438962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7439377Z return self.transformer( 2025-08-14T21:45:39.7439794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7440200Z layer_outputs = layer_module( 2025-08-14T21:45:39.7440542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7440905Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7441320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7441744Z sa_output = self.attention( 2025-08-14T21:45:39.7442133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:45:39.7442619Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:39.7442805Z 2025-08-14T21:45:39.7442914Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7443267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7443591Z return mod(**inputs) 2025-08-14T21:45:39.7444003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7444450Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7444880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7445314Z return self.transformer( 2025-08-14T21:45:39.7445732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7446134Z layer_outputs = layer_module( 2025-08-14T21:45:39.7446482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7446892Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7447353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:45:39.7447897Z sa_output = self.attention( 2025-08-14T21:45:39.7448317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:45:39.7448881Z attn_output = self.out_lin(attn_output) 2025-08-14T21:45:39.7449040Z 2025-08-14T21:45:39.7449153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7449538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7449874Z return mod(**inputs) 2025-08-14T21:45:39.7450287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7450776Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7451202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7451628Z return self.transformer( 2025-08-14T21:45:39.7452039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7452465Z layer_outputs = layer_module( 2025-08-14T21:45:39.7452821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7453203Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7453640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7454117Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7454580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7455176Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7455753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7456187Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7456641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:45:39.7457070Z x = self.lin1(input) 2025-08-14T21:45:39.7457175Z 2025-08-14T21:45:39.7457286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7457696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7458023Z return mod(**inputs) 2025-08-14T21:45:39.7458412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7458827Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7459235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7459647Z return self.transformer( 2025-08-14T21:45:39.7460040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7460447Z layer_outputs = layer_module( 2025-08-14T21:45:39.7460785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7461153Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7461572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7462011Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7462481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7463014Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7463526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7463910Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7464324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:45:39.7464737Z x = self.activation(x) 2025-08-14T21:45:39.7465068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:39.7465432Z return self.act(input) 2025-08-14T21:45:39.7465551Z 2025-08-14T21:45:39.7465657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7466036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7466371Z return mod(**inputs) 2025-08-14T21:45:39.7466757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:45:39.7467174Z distilbert_output = self.distilbert( 2025-08-14T21:45:39.7467589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:45:39.7467989Z return self.transformer( 2025-08-14T21:45:39.7468398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:45:39.7468830Z layer_outputs = layer_module( 2025-08-14T21:45:39.7469210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:39.7469590Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:39.7470003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:45:39.7470443Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:45:39.7470900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:45:39.7471491Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:45:39.7472050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:39.7472477Z return forward_fn(*input_tensors) 2025-08-14T21:45:39.7472920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:45:39.7473346Z x = self.lin2(x) 2025-08-14T21:45:39.7473451Z 2025-08-14T21:45:39.7473567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7473940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7474289Z return mod(**inputs) 2025-08-14T21:45:39.7474711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1043, in forward 2025-08-14T21:45:39.7475214Z logits = self.qa_outputs(hidden_states) # (bs, max_query_len, 2) 2025-08-14T21:45:39.7475403Z 2025-08-14T21:45:39.7475515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7475987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7476361Z return mod(**inputs) 2025-08-14T21:45:39.7476785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1061, in forward 2025-08-14T21:45:39.7477260Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:45:39.7477441Z 2025-08-14T21:45:39.7477552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:39.7477941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:39.7478284Z return mod(**inputs) 2025-08-14T21:45:39.7478695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1062, in forward 2025-08-14T21:45:39.7479156Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:45:39.7479314Z 2025-08-14T21:45:46.7955589Z Compilation time (from dynamo_timed): 11.007819935 2025-08-14T21:45:46.7956487Z pass 2025-08-14T21:45:46.7956810Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:46.7957702Z TIMING: _recursive_pre_grad_passes:0.0054 _recursive_joint_graph_passes:0.25719 _recursive_post_grad_passes:0.06328 async_compile.wait:0.66464 code_gen:6.75289 inductor_compile:7.7336 backend_compile:9.56566 gc:0.00193 entire_frame_compile:11.00782 total_wall_time:11.00782 2025-08-14T21:45:46.7958662Z STATS: call_* op count: 161 | FakeTensorMode.__torch_dispatch__:6705 | FakeTensor.__torch_dispatch__:2556 | ProxyTorchDispatchMode.__torch_dispatch__:2400 2025-08-14T21:45:46.7959199Z Dynamo produced 1 graphs covering 161 ops with 0 graph breaks (0 unique) 2025-08-14T21:45:52.0878800Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:45:52.0884985Z from pkg_resources import resource_filename 2025-08-14T21:45:52.6830865Z 2025-08-14T21:45:54.8049131Z loading model: 0it [00:00, ?it/s]`loss_type=None` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`. 2025-08-14T21:45:54.8049836Z WARNING:transformers.modeling_utils:`loss_type=None` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`. 2025-08-14T21:45:54.8350149Z 2025-08-14T21:45:54.8351083Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:45:54.8357123Z cpu eval DistillGPT2 2025-08-14T21:45:55.2514733Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:55.4357155Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:55.6162094Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:01.8699251Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8699601Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8699825Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8700056Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8700284Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8700507Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8700788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8701300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8701780Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8702187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8702605Z outputs = block( 2025-08-14T21:46:01.8702991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8703404Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8703822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8704247Z return func(*args, **kwargs) 2025-08-14T21:46:01.8704650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8705109Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8705545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8705942Z return func(*args, **kwargs) 2025-08-14T21:46:01.8706337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:46:01.8706876Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:46:01.8707700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8708143Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8708334Z 2025-08-14T21:46:01.8708429Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8708887Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8709124Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8709343Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8709589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8710049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8710488Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8710916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8711311Z outputs = block( 2025-08-14T21:46:01.8711753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8712174Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8712576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8712977Z return func(*args, **kwargs) 2025-08-14T21:46:01.8713382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8713863Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8714281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8714680Z return func(*args, **kwargs) 2025-08-14T21:46:01.8715131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8715577Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8716334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:46:01.8716977Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:46:01.8717181Z 2025-08-14T21:46:01.8717305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8717778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8718222Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8718646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8719053Z outputs = block( 2025-08-14T21:46:01.8719404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8719792Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8720200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8720601Z return func(*args, **kwargs) 2025-08-14T21:46:01.8721002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8721507Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8721897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8722259Z return func(*args, **kwargs) 2025-08-14T21:46:01.8722628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8723093Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8723572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:46:01.8724059Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:46:01.8724241Z 2025-08-14T21:46:01.8724352Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8724799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8725244Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8725630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8725999Z outputs = block( 2025-08-14T21:46:01.8726370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8726761Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8727171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8727604Z return func(*args, **kwargs) 2025-08-14T21:46:01.8728001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8728415Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8728806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8729181Z return func(*args, **kwargs) 2025-08-14T21:46:01.8729560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:46:01.8729999Z attn_output = self.c_proj(attn_output) 2025-08-14T21:46:01.8730413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8730851Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8731041Z 2025-08-14T21:46:01.8731157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8731607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8732051Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8732458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8732821Z outputs = block( 2025-08-14T21:46:01.8733144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8733506Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8733898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8734289Z return func(*args, **kwargs) 2025-08-14T21:46:01.8734676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8735122Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8735546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:46:01.8735951Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:46:01.8736329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8736756Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8736945Z 2025-08-14T21:46:01.8737055Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8737494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8737940Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8738345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8738750Z outputs = block( 2025-08-14T21:46:01.8739091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8739482Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8739876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8740274Z return func(*args, **kwargs) 2025-08-14T21:46:01.8740661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8741089Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8741520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:46:01.8741944Z hidden_states = self.act(hidden_states) 2025-08-14T21:46:01.8742313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:01.8742789Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:01.8743044Z 2025-08-14T21:46:01.8743155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8743596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8744040Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8744449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8744890Z outputs = block( 2025-08-14T21:46:01.8745233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8745606Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8746008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8746400Z return func(*args, **kwargs) 2025-08-14T21:46:01.8746788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8747210Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8747641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:46:01.8748055Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:46:01.8748443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8748858Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8749055Z 2025-08-14T21:46:01.8749168Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8749608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8750018Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8750427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8750818Z outputs = block( 2025-08-14T21:46:01.8751159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8751534Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8751946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8752374Z return func(*args, **kwargs) 2025-08-14T21:46:01.8752765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8753193Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8753614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8754022Z return func(*args, **kwargs) 2025-08-14T21:46:01.8754410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:46:01.8754962Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:46:01.8755466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8756092Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8756287Z 2025-08-14T21:46:01.8756382Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8756656Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8756891Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8757110Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8757372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8757852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8758294Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8758761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8759164Z outputs = block( 2025-08-14T21:46:01.8759515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8759928Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8760339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8760743Z return func(*args, **kwargs) 2025-08-14T21:46:01.8761140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8761572Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8761992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8762393Z return func(*args, **kwargs) 2025-08-14T21:46:01.8762785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8763225Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8763712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:46:01.8764236Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:46:01.8764438Z 2025-08-14T21:46:01.8764553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8765002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8765403Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8765792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8766157Z outputs = block( 2025-08-14T21:46:01.8766485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8766847Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8767221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8767609Z return func(*args, **kwargs) 2025-08-14T21:46:01.8767980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8768373Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8768749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8769115Z return func(*args, **kwargs) 2025-08-14T21:46:01.8769477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8769873Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8770332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:46:01.8770791Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:46:01.8770956Z 2025-08-14T21:46:01.8771065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8771495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8771893Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8772275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8772643Z outputs = block( 2025-08-14T21:46:01.8772960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8773337Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8773751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8774139Z return func(*args, **kwargs) 2025-08-14T21:46:01.8774553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8774972Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8775370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8775729Z return func(*args, **kwargs) 2025-08-14T21:46:01.8776093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:46:01.8776479Z attn_output = self.c_proj(attn_output) 2025-08-14T21:46:01.8776836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8777225Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8777402Z 2025-08-14T21:46:01.8777506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8777916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8778303Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8778685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8779049Z outputs = block( 2025-08-14T21:46:01.8779367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8779714Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8780091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8780459Z return func(*args, **kwargs) 2025-08-14T21:46:01.8780811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8781221Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8781646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:46:01.8782033Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:46:01.8782379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8782770Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8782947Z 2025-08-14T21:46:01.8783056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8783491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8783911Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8784326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8784695Z outputs = block( 2025-08-14T21:46:01.8785034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8785484Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8785892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8786315Z return func(*args, **kwargs) 2025-08-14T21:46:01.8786708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8787153Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8787614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:46:01.8788032Z hidden_states = self.act(hidden_states) 2025-08-14T21:46:01.8788413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:01.8788903Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:01.8789147Z 2025-08-14T21:46:01.8789265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8789714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8790131Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8790549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8790950Z outputs = block( 2025-08-14T21:46:01.8791294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8791685Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8792096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8792493Z return func(*args, **kwargs) 2025-08-14T21:46:01.8792878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8793309Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8793736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:46:01.8794152Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:46:01.8794533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8794958Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8795144Z 2025-08-14T21:46:01.8795263Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8795829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8796297Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8796721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8797132Z outputs = block( 2025-08-14T21:46:01.8797476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8797884Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8798282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8798679Z return func(*args, **kwargs) 2025-08-14T21:46:01.8799067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8799446Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8799821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8800196Z return func(*args, **kwargs) 2025-08-14T21:46:01.8800561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:46:01.8801053Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:46:01.8801511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8801895Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8802068Z 2025-08-14T21:46:01.8802165Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8802385Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8802584Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8802814Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8803047Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8803458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8803879Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8804285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8804676Z outputs = block( 2025-08-14T21:46:01.8805006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8805387Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8805788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8806181Z return func(*args, **kwargs) 2025-08-14T21:46:01.8806559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8806973Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8807370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8807721Z return func(*args, **kwargs) 2025-08-14T21:46:01.8808075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8808462Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8809050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:46:01.8809568Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:46:01.8809773Z 2025-08-14T21:46:01.8809885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8810327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8810817Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8811223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8811595Z outputs = block( 2025-08-14T21:46:01.8811919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8812288Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8812699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8813097Z return func(*args, **kwargs) 2025-08-14T21:46:01.8813533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8813995Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8814390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8814833Z return func(*args, **kwargs) 2025-08-14T21:46:01.8815186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8815579Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8816014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:46:01.8816470Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:46:01.8816634Z 2025-08-14T21:46:01.8816777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8817193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8817616Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8818006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8818377Z outputs = block( 2025-08-14T21:46:01.8818711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8819089Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8819476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8819870Z return func(*args, **kwargs) 2025-08-14T21:46:01.8820260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8820677Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8821073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8821441Z return func(*args, **kwargs) 2025-08-14T21:46:01.8821809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:46:01.8822191Z attn_output = self.c_proj(attn_output) 2025-08-14T21:46:01.8822548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8822949Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8823122Z 2025-08-14T21:46:01.8823234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8823665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8824085Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8824496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8824914Z outputs = block( 2025-08-14T21:46:01.8825244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8825616Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8825993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8826356Z return func(*args, **kwargs) 2025-08-14T21:46:01.8826722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8827129Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8827540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:46:01.8827933Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:46:01.8828310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8828727Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8828945Z 2025-08-14T21:46:01.8829057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8829465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8829877Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8830281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8830663Z outputs = block( 2025-08-14T21:46:01.8831021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8831404Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8831826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8832215Z return func(*args, **kwargs) 2025-08-14T21:46:01.8832602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8833035Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8833456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:46:01.8833860Z hidden_states = self.act(hidden_states) 2025-08-14T21:46:01.8834230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:01.8834708Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:01.8834951Z 2025-08-14T21:46:01.8835063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8835504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8836078Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8836500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8836928Z outputs = block( 2025-08-14T21:46:01.8837274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8837655Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8838051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8838475Z return func(*args, **kwargs) 2025-08-14T21:46:01.8838874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8839340Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8839798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:46:01.8840213Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:46:01.8840597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8841048Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8841230Z 2025-08-14T21:46:01.8841343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8841787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8842234Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8842634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8843057Z outputs = block( 2025-08-14T21:46:01.8843400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8843821Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8844212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8844603Z return func(*args, **kwargs) 2025-08-14T21:46:01.8844991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:46:01.8845424Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:46:01.8845595Z 2025-08-14T21:46:01.8845729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8846179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8846627Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8847046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8847451Z outputs = block( 2025-08-14T21:46:01.8847787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8848161Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8848553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8848949Z return func(*args, **kwargs) 2025-08-14T21:46:01.8849334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8849739Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8850151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8850536Z return func(*args, **kwargs) 2025-08-14T21:46:01.8850925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:46:01.8851435Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:46:01.8851892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8852308Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8852488Z 2025-08-14T21:46:01.8852581Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8852805Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8853032Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8853261Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8853488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8853904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8854360Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8854744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8855107Z outputs = block( 2025-08-14T21:46:01.8855447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8855825Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8856220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8856621Z return func(*args, **kwargs) 2025-08-14T21:46:01.8857008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8857435Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8857840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8858254Z return func(*args, **kwargs) 2025-08-14T21:46:01.8858635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8859026Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8859468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:46:01.8859974Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:46:01.8860175Z 2025-08-14T21:46:01.8860312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8860718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8861139Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8861527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8861899Z outputs = block( 2025-08-14T21:46:01.8862212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8862573Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8862947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8863317Z return func(*args, **kwargs) 2025-08-14T21:46:01.8863704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8864116Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8864520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8864918Z return func(*args, **kwargs) 2025-08-14T21:46:01.8865340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8865774Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8866246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:46:01.8866735Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:46:01.8866914Z 2025-08-14T21:46:01.8867025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8867467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8867895Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8868315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8868759Z outputs = block( 2025-08-14T21:46:01.8869134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8869525Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8869944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8870355Z return func(*args, **kwargs) 2025-08-14T21:46:01.8870751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8871188Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8871664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8872056Z return func(*args, **kwargs) 2025-08-14T21:46:01.8872447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:46:01.8872869Z attn_output = self.c_proj(attn_output) 2025-08-14T21:46:01.8873284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8873715Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8873900Z 2025-08-14T21:46:01.8874009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8874454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8874885Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8875307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8875821Z outputs = block( 2025-08-14T21:46:01.8876203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8876600Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8877010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8877414Z return func(*args, **kwargs) 2025-08-14T21:46:01.8877812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8878328Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8879048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:46:01.8892336Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:46:01.8892827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8893280Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8893490Z 2025-08-14T21:46:01.8893613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8894090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8894531Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8894953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8895358Z outputs = block( 2025-08-14T21:46:01.8895712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8896116Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8896532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8896940Z return func(*args, **kwargs) 2025-08-14T21:46:01.8897349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8897883Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8898339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:46:01.8898755Z hidden_states = self.act(hidden_states) 2025-08-14T21:46:01.8899120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:01.8899607Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:01.8899858Z 2025-08-14T21:46:01.8900048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8900491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8900924Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8901339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8901791Z outputs = block( 2025-08-14T21:46:01.8902145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8902534Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8902955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8903362Z return func(*args, **kwargs) 2025-08-14T21:46:01.8903822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8904268Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8904738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:46:01.8905170Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:46:01.8905559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8905994Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8906194Z 2025-08-14T21:46:01.8906311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8906761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8907246Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8907669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8908074Z outputs = block( 2025-08-14T21:46:01.8908428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8909065Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8909487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8909905Z return func(*args, **kwargs) 2025-08-14T21:46:01.8910306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8910747Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8911177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8911589Z return func(*args, **kwargs) 2025-08-14T21:46:01.8911989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:46:01.8912537Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:46:01.8913137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8913570Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8913769Z 2025-08-14T21:46:01.8913862Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8914105Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8914336Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8914559Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8914819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8915278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8915829Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8916288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8916701Z outputs = block( 2025-08-14T21:46:01.8917069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8917520Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8917935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8918352Z return func(*args, **kwargs) 2025-08-14T21:46:01.8918743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8919187Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8919658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8920074Z return func(*args, **kwargs) 2025-08-14T21:46:01.8920469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8920874Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8921325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:46:01.8921811Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:46:01.8921997Z 2025-08-14T21:46:01.8922102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8922523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8922922Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8923324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8923710Z outputs = block( 2025-08-14T21:46:01.8924053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8924445Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8924840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8925243Z return func(*args, **kwargs) 2025-08-14T21:46:01.8925630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8926055Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8926452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8926849Z return func(*args, **kwargs) 2025-08-14T21:46:01.8927228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8927627Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8928072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:46:01.8928556Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:46:01.8928720Z 2025-08-14T21:46:01.8928833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8929238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8929639Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8930031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8930405Z outputs = block( 2025-08-14T21:46:01.8930722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8931088Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8931467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8931859Z return func(*args, **kwargs) 2025-08-14T21:46:01.8932256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8932668Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8933054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8933417Z return func(*args, **kwargs) 2025-08-14T21:46:01.8933826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:46:01.8934250Z attn_output = self.c_proj(attn_output) 2025-08-14T21:46:01.8934633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8935086Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8935281Z 2025-08-14T21:46:01.8935393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8935847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8936279Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8936709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8937121Z outputs = block( 2025-08-14T21:46:01.8937475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8937867Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8938281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8938692Z return func(*args, **kwargs) 2025-08-14T21:46:01.8939081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8939522Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8939954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:46:01.8940378Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:46:01.8940767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8941196Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8941389Z 2025-08-14T21:46:01.8941503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8941945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8942379Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8942803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8943202Z outputs = block( 2025-08-14T21:46:01.8943549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8943942Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8944349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8944746Z return func(*args, **kwargs) 2025-08-14T21:46:01.8945131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8945554Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8945979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:46:01.8946381Z hidden_states = self.act(hidden_states) 2025-08-14T21:46:01.8946783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:01.8947231Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:01.8947468Z 2025-08-14T21:46:01.8947576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8947995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8948393Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8948810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8949200Z outputs = block( 2025-08-14T21:46:01.8949559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8949937Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8950341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8950739Z return func(*args, **kwargs) 2025-08-14T21:46:01.8951126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8951553Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8951983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:46:01.8952399Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:46:01.8952777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8953200Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8953389Z 2025-08-14T21:46:01.8953499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8953940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8954350Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8954761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8955153Z outputs = block( 2025-08-14T21:46:01.8955492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8955968Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8956376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8956777Z return func(*args, **kwargs) 2025-08-14T21:46:01.8957160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:46:01.8957632Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:46:01.8957803Z 2025-08-14T21:46:01.8957908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8958322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8958710Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8959099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8959472Z outputs = block( 2025-08-14T21:46:01.8959811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8960198Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8960596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8961031Z return func(*args, **kwargs) 2025-08-14T21:46:01.8961409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8961835Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8962242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8962640Z return func(*args, **kwargs) 2025-08-14T21:46:01.8963020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:46:01.8963559Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:46:01.8964069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8964489Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8964672Z 2025-08-14T21:46:01.8964760Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8964993Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8965217Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8965429Z cudagraph partition due to non gpu ops 2025-08-14T21:46:01.8965679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8966117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8966538Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8966950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8967343Z outputs = block( 2025-08-14T21:46:01.8967687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8968062Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8968464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8968857Z return func(*args, **kwargs) 2025-08-14T21:46:01.8969243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8969654Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8970060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8970449Z return func(*args, **kwargs) 2025-08-14T21:46:01.8970826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8971252Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8971723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:46:01.8972253Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:46:01.8972443Z 2025-08-14T21:46:01.8972553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8972988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8973405Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8973814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8974197Z outputs = block( 2025-08-14T21:46:01.8974534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8974915Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8975305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8975716Z return func(*args, **kwargs) 2025-08-14T21:46:01.8976100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8976517Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8976916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8977310Z return func(*args, **kwargs) 2025-08-14T21:46:01.8977716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:46:01.8978135Z attn_output, attn_weights = attention_interface( 2025-08-14T21:46:01.8978629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:46:01.8979117Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:46:01.8979291Z 2025-08-14T21:46:01.8979409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8979837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8980253Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8980658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8981046Z outputs = block( 2025-08-14T21:46:01.8981381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8981760Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8982159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8982544Z return func(*args, **kwargs) 2025-08-14T21:46:01.8982933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:46:01.8983354Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:46:01.8983761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8984156Z return func(*args, **kwargs) 2025-08-14T21:46:01.8984541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:46:01.8984954Z attn_output = self.c_proj(attn_output) 2025-08-14T21:46:01.8985325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8985724Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8985905Z 2025-08-14T21:46:01.8986013Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8986452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8986842Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8987231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8987600Z outputs = block( 2025-08-14T21:46:01.8987920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8988271Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8988648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8989018Z return func(*args, **kwargs) 2025-08-14T21:46:01.8989377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8989789Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8990234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:46:01.8990654Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:46:01.8991002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.8991413Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.8991594Z 2025-08-14T21:46:01.8991712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8992182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8992605Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8993032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8993432Z outputs = block( 2025-08-14T21:46:01.8993780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.8994176Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.8994591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.8994997Z return func(*args, **kwargs) 2025-08-14T21:46:01.8995383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.8995917Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.8996371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:46:01.8996812Z hidden_states = self.act(hidden_states) 2025-08-14T21:46:01.8997199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:01.8997695Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:01.8997950Z 2025-08-14T21:46:01.8998073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.8998536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:46:01.8998988Z transformer_outputs = self.transformer( 2025-08-14T21:46:01.8999417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:46:01.8999838Z outputs = block( 2025-08-14T21:46:01.9000187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:01.9000586Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:01.9001030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:01.9001436Z return func(*args, **kwargs) 2025-08-14T21:46:01.9001833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:46:01.9002279Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:46:01.9002717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:46:01.9003143Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:46:01.9003538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:46:01.9003970Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:46:01.9004155Z 2025-08-14T21:46:01.9004276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:01.9004729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1207, in forward 2025-08-14T21:46:01.9005217Z logits = self.lm_head(hidden_states[:, slice_indices, :]) 2025-08-14T21:46:01.9005408Z 2025-08-14T21:46:10.1796021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:10.1796606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:46:10.1797156Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:46:10.1798079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:46:10.1798681Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:46:10.1798977Z 2025-08-14T21:46:11.3704676Z Compilation time (from dynamo_timed): 14.291164061 2025-08-14T21:46:11.3862169Z pass 2025-08-14T21:46:11.3862608Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:11.3863547Z TIMING: gc:0.00517 entire_frame_compile:14.29116 _recursive_pre_grad_passes:0.00786 _recursive_joint_graph_passes:0.23067 _recursive_post_grad_passes:0.06007 async_compile.wait:1.44233 code_gen:8.80338 inductor_compile:9.50704 backend_compile:11.22931 total_wall_time:14.29116 2025-08-14T21:46:11.3864515Z STATS: call_* op count: 299 | FakeTensorMode.__torch_dispatch__:7245 | FakeTensor.__torch_dispatch__:2465 | ProxyTorchDispatchMode.__torch_dispatch__:2190 2025-08-14T21:46:11.3865052Z Dynamo produced 2 graphs covering 299 ops with 2 graph breaks (1 unique) 2025-08-14T21:46:16.7205649Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:46:16.7206697Z from pkg_resources import resource_filename 2025-08-14T21:46:17.3129765Z 2025-08-14T21:46:17.3139341Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-14T21:46:17.3139998Z WARNING:transformers.models.electra.modeling_electra:If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-14T21:46:17.7278451Z 2025-08-14T21:46:17.7279122Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:46:17.7298440Z cpu eval ElectraForCausalLM 2025-08-14T21:46:17.8907849Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:17.9791579Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:18.0641505Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:26.3364401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3365258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3365843Z return mod(**inputs) 2025-08-14T21:46:26.3366580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3367281Z outputs = self.electra( 2025-08-14T21:46:26.3367921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 797, in forward 2025-08-14T21:46:26.3368647Z hidden_states = self.embeddings_project(hidden_states) 2025-08-14T21:46:26.3368931Z 2025-08-14T21:46:26.3369121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3369709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3370272Z return mod(**inputs) 2025-08-14T21:46:26.3370971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3372063Z outputs = self.electra( 2025-08-14T21:46:26.3372759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3373502Z hidden_states = self.encoder( 2025-08-14T21:46:26.3374216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3374925Z layer_outputs = layer_module( 2025-08-14T21:46:26.3375635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3376287Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3376998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3377776Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3378514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3379221Z return func(*args, **kwargs) 2025-08-14T21:46:26.3379979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3380668Z self_outputs = self.self( 2025-08-14T21:46:26.3381315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3381980Z return func(*args, **kwargs) 2025-08-14T21:46:26.3382654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.3383354Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.3383597Z 2025-08-14T21:46:26.3383768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3384386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3384964Z return mod(**inputs) 2025-08-14T21:46:26.3385626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3386348Z outputs = self.electra( 2025-08-14T21:46:26.3386998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3387694Z hidden_states = self.encoder( 2025-08-14T21:46:26.3388395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3389102Z layer_outputs = layer_module( 2025-08-14T21:46:26.3389733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3390491Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3391234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3391967Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3392675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3393330Z return func(*args, **kwargs) 2025-08-14T21:46:26.3394028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3394746Z self_outputs = self.self( 2025-08-14T21:46:26.3395405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3396179Z return func(*args, **kwargs) 2025-08-14T21:46:26.3396873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.3397651Z key_layer = self.key(current_states) 2025-08-14T21:46:26.3397887Z 2025-08-14T21:46:26.3398078Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3398725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3399290Z return mod(**inputs) 2025-08-14T21:46:26.3399958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3400674Z outputs = self.electra( 2025-08-14T21:46:26.3401377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3402120Z hidden_states = self.encoder( 2025-08-14T21:46:26.3402874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3403598Z layer_outputs = layer_module( 2025-08-14T21:46:26.3404218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3404859Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3405610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3406354Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3406976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3407585Z return func(*args, **kwargs) 2025-08-14T21:46:26.3408211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3409210Z self_outputs = self.self( 2025-08-14T21:46:26.3409896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3410609Z return func(*args, **kwargs) 2025-08-14T21:46:26.3411280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.3411987Z value_layer = self.value(current_states) 2025-08-14T21:46:26.3412224Z 2025-08-14T21:46:26.3412352Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3412719Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3413129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3413779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3414351Z return mod(**inputs) 2025-08-14T21:46:26.3415026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3415722Z outputs = self.electra( 2025-08-14T21:46:26.3416531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3417254Z hidden_states = self.encoder( 2025-08-14T21:46:26.3417945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3418633Z layer_outputs = layer_module( 2025-08-14T21:46:26.3419261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3419905Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3420621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3421338Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3422047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3422738Z return func(*args, **kwargs) 2025-08-14T21:46:26.3423439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.3424176Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.3425000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.3425752Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3426011Z 2025-08-14T21:46:26.3426192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3426892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3427465Z return mod(**inputs) 2025-08-14T21:46:26.3428191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3428914Z outputs = self.electra( 2025-08-14T21:46:26.3429629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3430366Z hidden_states = self.encoder( 2025-08-14T21:46:26.3431071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3431801Z layer_outputs = layer_module( 2025-08-14T21:46:26.3432452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3433138Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3433906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3434697Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3435473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3436358Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3437205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3438114Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3438978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.3439744Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3440012Z 2025-08-14T21:46:26.3440204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3440884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3441482Z return mod(**inputs) 2025-08-14T21:46:26.3442191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3443077Z outputs = self.electra( 2025-08-14T21:46:26.3443792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3444528Z hidden_states = self.encoder( 2025-08-14T21:46:26.3445273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3446003Z layer_outputs = layer_module( 2025-08-14T21:46:26.3446621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3447266Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3447983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3448770Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3449552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3450375Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3451163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3452022Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3452813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.3453636Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.3454332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.3454953Z return self.act(input) 2025-08-14T21:46:26.3455179Z 2025-08-14T21:46:26.3455359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3456024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3456614Z return mod(**inputs) 2025-08-14T21:46:26.3457290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3458007Z outputs = self.electra( 2025-08-14T21:46:26.3458693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3459418Z hidden_states = self.encoder( 2025-08-14T21:46:26.3460119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3460821Z layer_outputs = layer_module( 2025-08-14T21:46:26.3461438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3462079Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3462791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3463538Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3464271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3464979Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3465723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.3466584Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.3467367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.3468062Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3468360Z 2025-08-14T21:46:26.3468533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3469185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3469757Z return mod(**inputs) 2025-08-14T21:46:26.3470433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3471158Z outputs = self.electra( 2025-08-14T21:46:26.3471844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3472572Z hidden_states = self.encoder( 2025-08-14T21:46:26.3473320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3474081Z layer_outputs = layer_module( 2025-08-14T21:46:26.3474754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3475477Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3476395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3477195Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3477931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3478654Z return func(*args, **kwargs) 2025-08-14T21:46:26.3479444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3480235Z self_outputs = self.self( 2025-08-14T21:46:26.3480936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3481699Z return func(*args, **kwargs) 2025-08-14T21:46:26.3482459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.3483248Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.3483506Z 2025-08-14T21:46:26.3483707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3484370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3485263Z return mod(**inputs) 2025-08-14T21:46:26.3485924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3486619Z outputs = self.electra( 2025-08-14T21:46:26.3487280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3487980Z hidden_states = self.encoder( 2025-08-14T21:46:26.3488663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3489379Z layer_outputs = layer_module( 2025-08-14T21:46:26.3490005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3490640Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3491358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3492073Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3492773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3493447Z return func(*args, **kwargs) 2025-08-14T21:46:26.3494156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3494910Z self_outputs = self.self( 2025-08-14T21:46:26.3495595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3496271Z return func(*args, **kwargs) 2025-08-14T21:46:26.3496963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.3497668Z key_layer = self.key(current_states) 2025-08-14T21:46:26.3497894Z 2025-08-14T21:46:26.3498094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3498747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3499321Z return mod(**inputs) 2025-08-14T21:46:26.3499999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3500747Z outputs = self.electra( 2025-08-14T21:46:26.3501476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3502268Z hidden_states = self.encoder( 2025-08-14T21:46:26.3502995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3503762Z layer_outputs = layer_module( 2025-08-14T21:46:26.3504435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3505138Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3505894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3506655Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3507379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3508067Z return func(*args, **kwargs) 2025-08-14T21:46:26.3509179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3509916Z self_outputs = self.self( 2025-08-14T21:46:26.3510573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3511240Z return func(*args, **kwargs) 2025-08-14T21:46:26.3511938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.3512704Z value_layer = self.value(current_states) 2025-08-14T21:46:26.3512960Z 2025-08-14T21:46:26.3513114Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3513499Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3513937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3514634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3515239Z return mod(**inputs) 2025-08-14T21:46:26.3516060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3516843Z outputs = self.electra( 2025-08-14T21:46:26.3517553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3518304Z hidden_states = self.encoder( 2025-08-14T21:46:26.3519057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3519820Z layer_outputs = layer_module( 2025-08-14T21:46:26.3520482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3521180Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3522122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3522908Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3523635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3524357Z return func(*args, **kwargs) 2025-08-14T21:46:26.3525103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.3525928Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.3526800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.3527542Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3527792Z 2025-08-14T21:46:26.3527999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3528629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3529278Z return mod(**inputs) 2025-08-14T21:46:26.3529953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3530650Z outputs = self.electra( 2025-08-14T21:46:26.3531308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3531995Z hidden_states = self.encoder( 2025-08-14T21:46:26.3532737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3533428Z layer_outputs = layer_module( 2025-08-14T21:46:26.3534092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3534867Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3535613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3536355Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3537093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3537818Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3538599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3539427Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3540215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.3540931Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3541177Z 2025-08-14T21:46:26.3541356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3542001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3542583Z return mod(**inputs) 2025-08-14T21:46:26.3543267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3543977Z outputs = self.electra( 2025-08-14T21:46:26.3544677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3545411Z hidden_states = self.encoder( 2025-08-14T21:46:26.3546112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3546822Z layer_outputs = layer_module( 2025-08-14T21:46:26.3547451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3548162Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3548881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3549618Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3550348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3551056Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3551855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3552770Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3553607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.3554441Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.3555181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.3555996Z return self.act(input) 2025-08-14T21:46:26.3556211Z 2025-08-14T21:46:26.3556412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3557106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3557727Z return mod(**inputs) 2025-08-14T21:46:26.3558416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3559204Z outputs = self.electra( 2025-08-14T21:46:26.3559895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3560878Z hidden_states = self.encoder( 2025-08-14T21:46:26.3562662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3563423Z layer_outputs = layer_module( 2025-08-14T21:46:26.3564037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3564681Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3565393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3566132Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3566886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3567629Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3568395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.3569273Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.3570054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.3570769Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3571004Z 2025-08-14T21:46:26.3571177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3571804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3572368Z return mod(**inputs) 2025-08-14T21:46:26.3573048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3573752Z outputs = self.electra( 2025-08-14T21:46:26.3574418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3575168Z hidden_states = self.encoder( 2025-08-14T21:46:26.3575861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3576587Z layer_outputs = layer_module( 2025-08-14T21:46:26.3577219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3577891Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3578616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3579364Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3580075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3580768Z return func(*args, **kwargs) 2025-08-14T21:46:26.3581462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3582177Z self_outputs = self.self( 2025-08-14T21:46:26.3582891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3583570Z return func(*args, **kwargs) 2025-08-14T21:46:26.3584262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.3585003Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.3585250Z 2025-08-14T21:46:26.3585441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3586122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3586701Z return mod(**inputs) 2025-08-14T21:46:26.3587410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3588141Z outputs = self.electra( 2025-08-14T21:46:26.3588836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3589566Z hidden_states = self.encoder( 2025-08-14T21:46:26.3590278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3590980Z layer_outputs = layer_module( 2025-08-14T21:46:26.3591591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3592237Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3592987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3593724Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3594452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3595173Z return func(*args, **kwargs) 2025-08-14T21:46:26.3596018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3596814Z self_outputs = self.self( 2025-08-14T21:46:26.3597516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3598218Z return func(*args, **kwargs) 2025-08-14T21:46:26.3598896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.3599624Z key_layer = self.key(current_states) 2025-08-14T21:46:26.3599854Z 2025-08-14T21:46:26.3600044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3600688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3601325Z return mod(**inputs) 2025-08-14T21:46:26.3602015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3602734Z outputs = self.electra( 2025-08-14T21:46:26.3603409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3604126Z hidden_states = self.encoder( 2025-08-14T21:46:26.3604831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3605559Z layer_outputs = layer_module( 2025-08-14T21:46:26.3606212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3606909Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3607683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3608479Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3609431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3610164Z return func(*args, **kwargs) 2025-08-14T21:46:26.3610899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3611641Z self_outputs = self.self( 2025-08-14T21:46:26.3612340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3613144Z return func(*args, **kwargs) 2025-08-14T21:46:26.3613874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.3614691Z value_layer = self.value(current_states) 2025-08-14T21:46:26.3614961Z 2025-08-14T21:46:26.3615105Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3615503Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3615893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3616533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3617104Z return mod(**inputs) 2025-08-14T21:46:26.3617768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3618470Z outputs = self.electra( 2025-08-14T21:46:26.3619147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3619844Z hidden_states = self.encoder( 2025-08-14T21:46:26.3620517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3621220Z layer_outputs = layer_module( 2025-08-14T21:46:26.3621830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3622466Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3623184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3623907Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3624583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3625244Z return func(*args, **kwargs) 2025-08-14T21:46:26.3625928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.3626717Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.3627513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.3628321Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3628563Z 2025-08-14T21:46:26.3628738Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3629366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3629922Z return mod(**inputs) 2025-08-14T21:46:26.3630603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3631306Z outputs = self.electra( 2025-08-14T21:46:26.3631960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3632660Z hidden_states = self.encoder( 2025-08-14T21:46:26.3633371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3634085Z layer_outputs = layer_module( 2025-08-14T21:46:26.3634765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3635443Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3636384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3637179Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3637874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3638635Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3639398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3640269Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3641063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.3641776Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3642006Z 2025-08-14T21:46:26.3642187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3642802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3643356Z return mod(**inputs) 2025-08-14T21:46:26.3644003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3644714Z outputs = self.electra( 2025-08-14T21:46:26.3645370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3646067Z hidden_states = self.encoder( 2025-08-14T21:46:26.3646766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3647479Z layer_outputs = layer_module( 2025-08-14T21:46:26.3648087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3648728Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3649446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3650158Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3650879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3651600Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3652359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3653228Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3654010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.3654786Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.3655475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.3656082Z return self.act(input) 2025-08-14T21:46:26.3656279Z 2025-08-14T21:46:26.3656462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3657116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3657693Z return mod(**inputs) 2025-08-14T21:46:26.3658357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3659070Z outputs = self.electra( 2025-08-14T21:46:26.3659725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3660467Z hidden_states = self.encoder( 2025-08-14T21:46:26.3661184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3661874Z layer_outputs = layer_module( 2025-08-14T21:46:26.3662482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3663125Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3663876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3664617Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3665363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3666094Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3666874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.3667772Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.3668607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.3669371Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3669625Z 2025-08-14T21:46:26.3669827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3670525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3671155Z return mod(**inputs) 2025-08-14T21:46:26.3671832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3672585Z outputs = self.electra( 2025-08-14T21:46:26.3673308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3674068Z hidden_states = self.encoder( 2025-08-14T21:46:26.3674810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3675586Z layer_outputs = layer_module( 2025-08-14T21:46:26.3676363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3677088Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3677827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3678567Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3679328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3680013Z return func(*args, **kwargs) 2025-08-14T21:46:26.3680715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3681426Z self_outputs = self.self( 2025-08-14T21:46:26.3682079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3682752Z return func(*args, **kwargs) 2025-08-14T21:46:26.3683480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.3684253Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.3684525Z 2025-08-14T21:46:26.3684717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3685380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3686027Z return mod(**inputs) 2025-08-14T21:46:26.3686700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3687419Z outputs = self.electra( 2025-08-14T21:46:26.3688113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3688829Z hidden_states = self.encoder( 2025-08-14T21:46:26.3689560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3690306Z layer_outputs = layer_module( 2025-08-14T21:46:26.3690919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3691602Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3692340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3693092Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3693785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3694472Z return func(*args, **kwargs) 2025-08-14T21:46:26.3695203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3695930Z self_outputs = self.self( 2025-08-14T21:46:26.3696615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3697338Z return func(*args, **kwargs) 2025-08-14T21:46:26.3698089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.3698859Z key_layer = self.key(current_states) 2025-08-14T21:46:26.3699119Z 2025-08-14T21:46:26.3699309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3700003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3700618Z return mod(**inputs) 2025-08-14T21:46:26.3701324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3702084Z outputs = self.electra( 2025-08-14T21:46:26.3702810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3703552Z hidden_states = self.encoder( 2025-08-14T21:46:26.3704277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3705003Z layer_outputs = layer_module( 2025-08-14T21:46:26.3705665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3706327Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3707052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3707793Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3708515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3709402Z return func(*args, **kwargs) 2025-08-14T21:46:26.3710152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3710912Z self_outputs = self.self( 2025-08-14T21:46:26.3711558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3712205Z return func(*args, **kwargs) 2025-08-14T21:46:26.3712924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.3713785Z value_layer = self.value(current_states) 2025-08-14T21:46:26.3714056Z 2025-08-14T21:46:26.3714193Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3714587Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3715041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3715806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3716451Z return mod(**inputs) 2025-08-14T21:46:26.3717254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3717970Z outputs = self.electra( 2025-08-14T21:46:26.3718702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3719375Z hidden_states = self.encoder( 2025-08-14T21:46:26.3720062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3720753Z layer_outputs = layer_module( 2025-08-14T21:46:26.3721361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3721995Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3722708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3723418Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3724110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3724765Z return func(*args, **kwargs) 2025-08-14T21:46:26.3725430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.3726242Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.3727041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.3727747Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3727985Z 2025-08-14T21:46:26.3728158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3728806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3729375Z return mod(**inputs) 2025-08-14T21:46:26.3730041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3730745Z outputs = self.electra( 2025-08-14T21:46:26.3731467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3732168Z hidden_states = self.encoder( 2025-08-14T21:46:26.3732845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3733548Z layer_outputs = layer_module( 2025-08-14T21:46:26.3734156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3734778Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3735459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3736164Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3736863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3737536Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3738287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3739199Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3739996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.3740709Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3740973Z 2025-08-14T21:46:26.3741150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3741821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3742390Z return mod(**inputs) 2025-08-14T21:46:26.3743066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3743774Z outputs = self.electra( 2025-08-14T21:46:26.3744450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3745113Z hidden_states = self.encoder( 2025-08-14T21:46:26.3745735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3746384Z layer_outputs = layer_module( 2025-08-14T21:46:26.3746952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3747542Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3748166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3748792Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3749445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3750150Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3750902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3751698Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3752488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.3753261Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.3753943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.3754569Z return self.act(input) 2025-08-14T21:46:26.3754772Z 2025-08-14T21:46:26.3754952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3755590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3756341Z return mod(**inputs) 2025-08-14T21:46:26.3756982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3757595Z outputs = self.electra( 2025-08-14T21:46:26.3758232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3758917Z hidden_states = self.encoder( 2025-08-14T21:46:26.3759558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3760193Z layer_outputs = layer_module( 2025-08-14T21:46:26.3760752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3761332Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3761977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3762540Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3762987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3763419Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3763881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.3764416Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.3765046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.3765741Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3765972Z 2025-08-14T21:46:26.3766164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3766755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3767277Z return mod(**inputs) 2025-08-14T21:46:26.3767856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3768476Z outputs = self.electra( 2025-08-14T21:46:26.3769081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3769735Z hidden_states = self.encoder( 2025-08-14T21:46:26.3770394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3771009Z layer_outputs = layer_module( 2025-08-14T21:46:26.3771499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3772027Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3772617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3773215Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3773720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3774212Z return func(*args, **kwargs) 2025-08-14T21:46:26.3774598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3775020Z self_outputs = self.self( 2025-08-14T21:46:26.3775411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3775787Z return func(*args, **kwargs) 2025-08-14T21:46:26.3776175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.3776619Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.3776771Z 2025-08-14T21:46:26.3776885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3777264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3777608Z return mod(**inputs) 2025-08-14T21:46:26.3777996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3778434Z outputs = self.electra( 2025-08-14T21:46:26.3778866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3779292Z hidden_states = self.encoder( 2025-08-14T21:46:26.3779707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3780122Z layer_outputs = layer_module( 2025-08-14T21:46:26.3780469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3780852Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3781256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3781695Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3782103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3782497Z return func(*args, **kwargs) 2025-08-14T21:46:26.3782943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3783358Z self_outputs = self.self( 2025-08-14T21:46:26.3783865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3784426Z return func(*args, **kwargs) 2025-08-14T21:46:26.3784985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.3785399Z key_layer = self.key(current_states) 2025-08-14T21:46:26.3785537Z 2025-08-14T21:46:26.3785646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3786014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3786389Z return mod(**inputs) 2025-08-14T21:46:26.3786976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3787639Z outputs = self.electra( 2025-08-14T21:46:26.3788298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3788983Z hidden_states = self.encoder( 2025-08-14T21:46:26.3789596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3790302Z layer_outputs = layer_module( 2025-08-14T21:46:26.3790778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3791245Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3791792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3792343Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3792877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3793383Z return func(*args, **kwargs) 2025-08-14T21:46:26.3793909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3794502Z self_outputs = self.self( 2025-08-14T21:46:26.3795001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3795506Z return func(*args, **kwargs) 2025-08-14T21:46:26.3796142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.3796705Z value_layer = self.value(current_states) 2025-08-14T21:46:26.3796888Z 2025-08-14T21:46:26.3797001Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3797280Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3797594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3798103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3798563Z return mod(**inputs) 2025-08-14T21:46:26.3799107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3799693Z outputs = self.electra( 2025-08-14T21:46:26.3800178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3800680Z hidden_states = self.encoder( 2025-08-14T21:46:26.3801215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3801760Z layer_outputs = layer_module( 2025-08-14T21:46:26.3802258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3802749Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3803311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3803906Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3804427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3804932Z return func(*args, **kwargs) 2025-08-14T21:46:26.3805446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.3806047Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.3806657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.3807207Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3807391Z 2025-08-14T21:46:26.3807533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3808011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3808455Z return mod(**inputs) 2025-08-14T21:46:26.3809190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3809763Z outputs = self.electra( 2025-08-14T21:46:26.3810291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3810836Z hidden_states = self.encoder( 2025-08-14T21:46:26.3811371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3811909Z layer_outputs = layer_module( 2025-08-14T21:46:26.3812389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3812891Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3813429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3814100Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3814612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3815121Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3815656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3816266Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3816828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.3817341Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3817511Z 2025-08-14T21:46:26.3817640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3818096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3818563Z return mod(**inputs) 2025-08-14T21:46:26.3819042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3819544Z outputs = self.electra( 2025-08-14T21:46:26.3820028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3820538Z hidden_states = self.encoder( 2025-08-14T21:46:26.3821021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3821567Z layer_outputs = layer_module( 2025-08-14T21:46:26.3822006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3822498Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3823073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3823609Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3824142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3824672Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3825229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3825867Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3826464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.3827033Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.3827538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.3827994Z return self.act(input) 2025-08-14T21:46:26.3828133Z 2025-08-14T21:46:26.3828266Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3828734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3829156Z return mod(**inputs) 2025-08-14T21:46:26.3829644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3830178Z outputs = self.electra( 2025-08-14T21:46:26.3830683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3831211Z hidden_states = self.encoder( 2025-08-14T21:46:26.3831746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3832332Z layer_outputs = layer_module( 2025-08-14T21:46:26.3832810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3833320Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3833873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3834440Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3835002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3835558Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3836257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.3836939Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.3837547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.3838122Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3838302Z 2025-08-14T21:46:26.3838431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3838907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3839330Z return mod(**inputs) 2025-08-14T21:46:26.3839822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3840342Z outputs = self.electra( 2025-08-14T21:46:26.3840864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3841395Z hidden_states = self.encoder( 2025-08-14T21:46:26.3841924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3842451Z layer_outputs = layer_module( 2025-08-14T21:46:26.3842910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3843383Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3843897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3844430Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3844933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3845414Z return func(*args, **kwargs) 2025-08-14T21:46:26.3845924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3846450Z self_outputs = self.self( 2025-08-14T21:46:26.3846927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3847419Z return func(*args, **kwargs) 2025-08-14T21:46:26.3847906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.3848428Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.3848597Z 2025-08-14T21:46:26.3848729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3849182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3849597Z return mod(**inputs) 2025-08-14T21:46:26.3850076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3850577Z outputs = self.electra( 2025-08-14T21:46:26.3851060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3851612Z hidden_states = self.encoder( 2025-08-14T21:46:26.3852103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3852603Z layer_outputs = layer_module( 2025-08-14T21:46:26.3853058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3853535Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3854078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3854603Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3855096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3855574Z return func(*args, **kwargs) 2025-08-14T21:46:26.3856057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3856596Z self_outputs = self.self( 2025-08-14T21:46:26.3857061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3857526Z return func(*args, **kwargs) 2025-08-14T21:46:26.3858003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.3858521Z key_layer = self.key(current_states) 2025-08-14T21:46:26.3858687Z 2025-08-14T21:46:26.3858844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3859297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3859701Z return mod(**inputs) 2025-08-14T21:46:26.3860209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3860743Z outputs = self.electra( 2025-08-14T21:46:26.3861230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3861733Z hidden_states = self.encoder( 2025-08-14T21:46:26.3862221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3862749Z layer_outputs = layer_module( 2025-08-14T21:46:26.3863192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3863649Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3864160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3864675Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3865163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3865642Z return func(*args, **kwargs) 2025-08-14T21:46:26.3866123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3866604Z self_outputs = self.self( 2025-08-14T21:46:26.3867051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3867518Z return func(*args, **kwargs) 2025-08-14T21:46:26.3867980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.3868478Z value_layer = self.value(current_states) 2025-08-14T21:46:26.3868648Z 2025-08-14T21:46:26.3868747Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3869046Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3869321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3869762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3870161Z return mod(**inputs) 2025-08-14T21:46:26.3870616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3871104Z outputs = self.electra( 2025-08-14T21:46:26.3871566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3872080Z hidden_states = self.encoder( 2025-08-14T21:46:26.3872577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3873113Z layer_outputs = layer_module( 2025-08-14T21:46:26.3873569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3874073Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3874607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3875154Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3875762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3876281Z return func(*args, **kwargs) 2025-08-14T21:46:26.3876857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.3877507Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.3878169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.3878691Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3878875Z 2025-08-14T21:46:26.3879003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3879456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3879867Z return mod(**inputs) 2025-08-14T21:46:26.3880390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3880951Z outputs = self.electra( 2025-08-14T21:46:26.3881475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3882025Z hidden_states = self.encoder( 2025-08-14T21:46:26.3882535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3883040Z layer_outputs = layer_module( 2025-08-14T21:46:26.3883517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3884027Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3884586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3885167Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3885730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3886283Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3886893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3887554Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3888177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.3888856Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3889047Z 2025-08-14T21:46:26.3889194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3889693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3890150Z return mod(**inputs) 2025-08-14T21:46:26.3890707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3891274Z outputs = self.electra( 2025-08-14T21:46:26.3891827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3892400Z hidden_states = self.encoder( 2025-08-14T21:46:26.3892900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3893411Z layer_outputs = layer_module( 2025-08-14T21:46:26.3893874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3894347Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3894861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3895381Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3895994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3896650Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3897324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3898062Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3898713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.3899365Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.3899900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.3900342Z return self.act(input) 2025-08-14T21:46:26.3900485Z 2025-08-14T21:46:26.3900616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3901076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3901474Z return mod(**inputs) 2025-08-14T21:46:26.3901955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3902464Z outputs = self.electra( 2025-08-14T21:46:26.3902944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3903450Z hidden_states = self.encoder( 2025-08-14T21:46:26.3903961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3904504Z layer_outputs = layer_module( 2025-08-14T21:46:26.3904962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3905440Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3905983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3906558Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3907121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3907679Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3908347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.3909236Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.3909882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.3910456Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3910649Z 2025-08-14T21:46:26.3910792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3911334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3911791Z return mod(**inputs) 2025-08-14T21:46:26.3912318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3912877Z outputs = self.electra( 2025-08-14T21:46:26.3913399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3914040Z hidden_states = self.encoder( 2025-08-14T21:46:26.3914594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3915138Z layer_outputs = layer_module( 2025-08-14T21:46:26.3915624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3916231Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3916868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3917451Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3918021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3918552Z return func(*args, **kwargs) 2025-08-14T21:46:26.3919102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3919647Z self_outputs = self.self( 2025-08-14T21:46:26.3920160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3920687Z return func(*args, **kwargs) 2025-08-14T21:46:26.3921216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.3921800Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.3921978Z 2025-08-14T21:46:26.3922105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3922549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3922950Z return mod(**inputs) 2025-08-14T21:46:26.3923445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3923957Z outputs = self.electra( 2025-08-14T21:46:26.3924419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3924924Z hidden_states = self.encoder( 2025-08-14T21:46:26.3925423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3925954Z layer_outputs = layer_module( 2025-08-14T21:46:26.3926404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3926944Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3927515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3928116Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3928588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3929048Z return func(*args, **kwargs) 2025-08-14T21:46:26.3929516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3929997Z self_outputs = self.self( 2025-08-14T21:46:26.3930457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3930940Z return func(*args, **kwargs) 2025-08-14T21:46:26.3931490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.3932074Z key_layer = self.key(current_states) 2025-08-14T21:46:26.3932265Z 2025-08-14T21:46:26.3932410Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3932917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3933393Z return mod(**inputs) 2025-08-14T21:46:26.3933937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3934476Z outputs = self.electra( 2025-08-14T21:46:26.3934959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3935519Z hidden_states = self.encoder( 2025-08-14T21:46:26.3936072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3936564Z layer_outputs = layer_module( 2025-08-14T21:46:26.3937031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3937496Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3938005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3938525Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3939036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3939568Z return func(*args, **kwargs) 2025-08-14T21:46:26.3940139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3940723Z self_outputs = self.self( 2025-08-14T21:46:26.3941220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3941690Z return func(*args, **kwargs) 2025-08-14T21:46:26.3942189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.3942767Z value_layer = self.value(current_states) 2025-08-14T21:46:26.3942976Z 2025-08-14T21:46:26.3943081Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3943364Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.3943688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3944146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3944570Z return mod(**inputs) 2025-08-14T21:46:26.3945078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3945632Z outputs = self.electra( 2025-08-14T21:46:26.3946185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3946748Z hidden_states = self.encoder( 2025-08-14T21:46:26.3947255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3947810Z layer_outputs = layer_module( 2025-08-14T21:46:26.3948285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3948796Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3949329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3949877Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3950382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3950867Z return func(*args, **kwargs) 2025-08-14T21:46:26.3951346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.3951919Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.3952547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.3953092Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3953269Z 2025-08-14T21:46:26.3953398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3953870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3954313Z return mod(**inputs) 2025-08-14T21:46:26.3954880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3955452Z outputs = self.electra( 2025-08-14T21:46:26.3956091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3956655Z hidden_states = self.encoder( 2025-08-14T21:46:26.3957167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3957663Z layer_outputs = layer_module( 2025-08-14T21:46:26.3958097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3958552Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3959054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3959574Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3960086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3960608Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3961155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3961771Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3962342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.3962837Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3963017Z 2025-08-14T21:46:26.3963139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3963588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3963996Z return mod(**inputs) 2025-08-14T21:46:26.3964505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3965043Z outputs = self.electra( 2025-08-14T21:46:26.3965533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3966080Z hidden_states = self.encoder( 2025-08-14T21:46:26.3966578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3967093Z layer_outputs = layer_module( 2025-08-14T21:46:26.3967540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3968007Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3968536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3969086Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3969640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3970198Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3970809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.3971530Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.3972210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.3972768Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.3973340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.3973878Z return self.act(input) 2025-08-14T21:46:26.3974042Z 2025-08-14T21:46:26.3974187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3974740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3975216Z return mod(**inputs) 2025-08-14T21:46:26.3975757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3976396Z outputs = self.electra( 2025-08-14T21:46:26.3976963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3977567Z hidden_states = self.encoder( 2025-08-14T21:46:26.3978129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3978708Z layer_outputs = layer_module( 2025-08-14T21:46:26.3979205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3979699Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3980238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.3980819Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.3981381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.3981912Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.3982531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.3983173Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.3983778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.3984345Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.3984560Z 2025-08-14T21:46:26.3984701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3985218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3985721Z return mod(**inputs) 2025-08-14T21:46:26.3986269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3986848Z outputs = self.electra( 2025-08-14T21:46:26.3987398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3987974Z hidden_states = self.encoder( 2025-08-14T21:46:26.3988554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3989123Z layer_outputs = layer_module( 2025-08-14T21:46:26.3989646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3990157Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3990764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3991359Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3991948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3992517Z return func(*args, **kwargs) 2025-08-14T21:46:26.3993095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3993730Z self_outputs = self.self( 2025-08-14T21:46:26.3994139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3994233Z return func(*args, **kwargs) 2025-08-14T21:46:26.3994671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.3994776Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.3994782Z 2025-08-14T21:46:26.3994932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.3995221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.3995323Z return mod(**inputs) 2025-08-14T21:46:26.3995805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.3995921Z outputs = self.electra( 2025-08-14T21:46:26.3996386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.3996521Z hidden_states = self.encoder( 2025-08-14T21:46:26.3996958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.3997082Z layer_outputs = layer_module( 2025-08-14T21:46:26.3997482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.3997610Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.3998058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.3998185Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.3998593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3998703Z return func(*args, **kwargs) 2025-08-14T21:46:26.3999134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.3999252Z self_outputs = self.self( 2025-08-14T21:46:26.3999681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.3999838Z return func(*args, **kwargs) 2025-08-14T21:46:26.4000313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.4000430Z key_layer = self.key(current_states) 2025-08-14T21:46:26.4000435Z 2025-08-14T21:46:26.4000604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4000940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4001039Z return mod(**inputs) 2025-08-14T21:46:26.4001500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4001608Z outputs = self.electra( 2025-08-14T21:46:26.4002048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4002174Z hidden_states = self.encoder( 2025-08-14T21:46:26.4002619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4002760Z layer_outputs = layer_module( 2025-08-14T21:46:26.4003055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4003146Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4003423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4003510Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4003787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4003870Z return func(*args, **kwargs) 2025-08-14T21:46:26.4004161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4004248Z self_outputs = self.self( 2025-08-14T21:46:26.4004498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4004573Z return func(*args, **kwargs) 2025-08-14T21:46:26.4004852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.4004936Z value_layer = self.value(current_states) 2025-08-14T21:46:26.4004940Z 2025-08-14T21:46:26.4005035Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4005120Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4005232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4005450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4005519Z return mod(**inputs) 2025-08-14T21:46:26.4005796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4005877Z outputs = self.electra( 2025-08-14T21:46:26.4006151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4006232Z hidden_states = self.encoder( 2025-08-14T21:46:26.4006504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4006578Z layer_outputs = layer_module( 2025-08-14T21:46:26.4006816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4006900Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4007170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4007265Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4007533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4007614Z return func(*args, **kwargs) 2025-08-14T21:46:26.4007884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.4008020Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.4008297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.4008393Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4008396Z 2025-08-14T21:46:26.4008508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4008877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4008983Z return mod(**inputs) 2025-08-14T21:46:26.4009273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4009406Z outputs = self.electra( 2025-08-14T21:46:26.4009665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4009744Z hidden_states = self.encoder( 2025-08-14T21:46:26.4010001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4010079Z layer_outputs = layer_module( 2025-08-14T21:46:26.4010324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4010402Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4010696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4010785Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4011047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4011127Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4011423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4011555Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4011811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.4011894Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4011898Z 2025-08-14T21:46:26.4012008Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4012209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4012283Z return mod(**inputs) 2025-08-14T21:46:26.4012540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4012608Z outputs = self.electra( 2025-08-14T21:46:26.4012867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4012937Z hidden_states = self.encoder( 2025-08-14T21:46:26.4013198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4013266Z layer_outputs = layer_module( 2025-08-14T21:46:26.4013485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4013571Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4013830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4013938Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4014203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4014280Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4014574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4014694Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4014950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.4015074Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.4015284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.4015357Z return self.act(input) 2025-08-14T21:46:26.4015368Z 2025-08-14T21:46:26.4015469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4015683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4015755Z return mod(**inputs) 2025-08-14T21:46:26.4016015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4016082Z outputs = self.electra( 2025-08-14T21:46:26.4016345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4016431Z hidden_states = self.encoder( 2025-08-14T21:46:26.4016692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4016777Z layer_outputs = layer_module( 2025-08-14T21:46:26.4016996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4017088Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4017357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4017439Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4017701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4017776Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4018082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.4018216Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.4018481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.4018573Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4018578Z 2025-08-14T21:46:26.4018683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4018892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4018957Z return mod(**inputs) 2025-08-14T21:46:26.4019225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4019300Z outputs = self.electra( 2025-08-14T21:46:26.4019566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4019637Z hidden_states = self.encoder( 2025-08-14T21:46:26.4019907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4019995Z layer_outputs = layer_module( 2025-08-14T21:46:26.4020223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4020301Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4020551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4020636Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4020869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4020944Z return func(*args, **kwargs) 2025-08-14T21:46:26.4021197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4021265Z self_outputs = self.self( 2025-08-14T21:46:26.4021507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4021575Z return func(*args, **kwargs) 2025-08-14T21:46:26.4021840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.4021925Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.4021929Z 2025-08-14T21:46:26.4022028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4022226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4022288Z return mod(**inputs) 2025-08-14T21:46:26.4022555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4022628Z outputs = self.electra( 2025-08-14T21:46:26.4022893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4022964Z hidden_states = self.encoder( 2025-08-14T21:46:26.4023222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4023289Z layer_outputs = layer_module( 2025-08-14T21:46:26.4023527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4023603Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4023856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4023947Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4024186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4024264Z return func(*args, **kwargs) 2025-08-14T21:46:26.4024531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4024600Z self_outputs = self.self( 2025-08-14T21:46:26.4024839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4024906Z return func(*args, **kwargs) 2025-08-14T21:46:26.4025160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.4025243Z key_layer = self.key(current_states) 2025-08-14T21:46:26.4025246Z 2025-08-14T21:46:26.4025345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4025547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4025609Z return mod(**inputs) 2025-08-14T21:46:26.4025867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4025966Z outputs = self.electra( 2025-08-14T21:46:26.4026220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4026299Z hidden_states = self.encoder( 2025-08-14T21:46:26.4026553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4026622Z layer_outputs = layer_module( 2025-08-14T21:46:26.4026858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4026935Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4027188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4027273Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4027505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4027596Z return func(*args, **kwargs) 2025-08-14T21:46:26.4027846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4027911Z self_outputs = self.self( 2025-08-14T21:46:26.4028149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4028215Z return func(*args, **kwargs) 2025-08-14T21:46:26.4028468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.4028568Z value_layer = self.value(current_states) 2025-08-14T21:46:26.4028572Z 2025-08-14T21:46:26.4028652Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4028737Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4028852Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4029043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4029117Z return mod(**inputs) 2025-08-14T21:46:26.4029368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4029432Z outputs = self.electra( 2025-08-14T21:46:26.4029685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4029753Z hidden_states = self.encoder( 2025-08-14T21:46:26.4030007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4030078Z layer_outputs = layer_module( 2025-08-14T21:46:26.4030287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4030368Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4030614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4030700Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4030926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4031005Z return func(*args, **kwargs) 2025-08-14T21:46:26.4031259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.4031386Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.4031634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.4031724Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4031751Z 2025-08-14T21:46:26.4031854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4032056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4032120Z return mod(**inputs) 2025-08-14T21:46:26.4032378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4032454Z outputs = self.electra( 2025-08-14T21:46:26.4032709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4032787Z hidden_states = self.encoder( 2025-08-14T21:46:26.4033056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4033129Z layer_outputs = layer_module( 2025-08-14T21:46:26.4033369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4033452Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4033739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4033833Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4034102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4034189Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4034508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4034634Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4034926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.4035013Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4035017Z 2025-08-14T21:46:26.4035130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4035338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4035406Z return mod(**inputs) 2025-08-14T21:46:26.4035790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4035903Z outputs = self.electra( 2025-08-14T21:46:26.4036184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4036274Z hidden_states = self.encoder( 2025-08-14T21:46:26.4036554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4036639Z layer_outputs = layer_module( 2025-08-14T21:46:26.4036878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4036964Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4037250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4037341Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4037632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4037711Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4038019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4038153Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4038437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.4038588Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.4038820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.4038894Z return self.act(input) 2025-08-14T21:46:26.4038898Z 2025-08-14T21:46:26.4039009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4039223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4039290Z return mod(**inputs) 2025-08-14T21:46:26.4039574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4039646Z outputs = self.electra( 2025-08-14T21:46:26.4039927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4040009Z hidden_states = self.encoder( 2025-08-14T21:46:26.4040286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4040387Z layer_outputs = layer_module( 2025-08-14T21:46:26.4040624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4040706Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4040994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4041080Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4041367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4041448Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4041767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.4041918Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.4042200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.4042286Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4042297Z 2025-08-14T21:46:26.4042405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4042612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4042686Z return mod(**inputs) 2025-08-14T21:46:26.4042962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4043033Z outputs = self.electra( 2025-08-14T21:46:26.4043323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4043398Z hidden_states = self.encoder( 2025-08-14T21:46:26.4043688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4043761Z layer_outputs = layer_module( 2025-08-14T21:46:26.4044000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4044091Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4044370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4044454Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4044701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4044773Z return func(*args, **kwargs) 2025-08-14T21:46:26.4045053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4045124Z self_outputs = self.self( 2025-08-14T21:46:26.4045370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4045447Z return func(*args, **kwargs) 2025-08-14T21:46:26.4045705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.4045791Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.4045795Z 2025-08-14T21:46:26.4045897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4046093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4046164Z return mod(**inputs) 2025-08-14T21:46:26.4046423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4046506Z outputs = self.electra( 2025-08-14T21:46:26.4046779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4046849Z hidden_states = self.encoder( 2025-08-14T21:46:26.4047120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4047189Z layer_outputs = layer_module( 2025-08-14T21:46:26.4047412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4047514Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4047769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4047864Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4048114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4048184Z return func(*args, **kwargs) 2025-08-14T21:46:26.4048445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4048513Z self_outputs = self.self( 2025-08-14T21:46:26.4048748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4048824Z return func(*args, **kwargs) 2025-08-14T21:46:26.4049080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.4049164Z key_layer = self.key(current_states) 2025-08-14T21:46:26.4049168Z 2025-08-14T21:46:26.4049272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4049477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4049550Z return mod(**inputs) 2025-08-14T21:46:26.4049802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4049868Z outputs = self.electra( 2025-08-14T21:46:26.4050128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4050195Z hidden_states = self.encoder( 2025-08-14T21:46:26.4050452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4050519Z layer_outputs = layer_module( 2025-08-14T21:46:26.4050732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4050816Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4051087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4051167Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4051404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4051472Z return func(*args, **kwargs) 2025-08-14T21:46:26.4051727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4051793Z self_outputs = self.self( 2025-08-14T21:46:26.4052024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4052097Z return func(*args, **kwargs) 2025-08-14T21:46:26.4052344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.4052431Z value_layer = self.value(current_states) 2025-08-14T21:46:26.4052450Z 2025-08-14T21:46:26.4052529Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4052604Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4052711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4052898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4052959Z return mod(**inputs) 2025-08-14T21:46:26.4053220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4053284Z outputs = self.electra( 2025-08-14T21:46:26.4053550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4053620Z hidden_states = self.encoder( 2025-08-14T21:46:26.4053889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4053968Z layer_outputs = layer_module( 2025-08-14T21:46:26.4054182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4054259Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4054518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4054598Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4054848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4054913Z return func(*args, **kwargs) 2025-08-14T21:46:26.4055159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.4055294Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.4055543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.4055630Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4055633Z 2025-08-14T21:46:26.4055731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4055918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4055989Z return mod(**inputs) 2025-08-14T21:46:26.4056238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4056306Z outputs = self.electra( 2025-08-14T21:46:26.4056559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4056627Z hidden_states = self.encoder( 2025-08-14T21:46:26.4056901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4056970Z layer_outputs = layer_module( 2025-08-14T21:46:26.4057180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4057261Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4057508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4057596Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4057844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4057917Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4058206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4058322Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4058588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.4058674Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4058677Z 2025-08-14T21:46:26.4058775Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4058974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4059038Z return mod(**inputs) 2025-08-14T21:46:26.4059305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4059381Z outputs = self.electra( 2025-08-14T21:46:26.4059648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4059727Z hidden_states = self.encoder( 2025-08-14T21:46:26.4059978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4060046Z layer_outputs = layer_module( 2025-08-14T21:46:26.4060268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4060343Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4060594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4060680Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4060929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4061009Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4061296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4061414Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4061671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.4061779Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.4061991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.4062059Z return self.act(input) 2025-08-14T21:46:26.4062062Z 2025-08-14T21:46:26.4062164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4062360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4062423Z return mod(**inputs) 2025-08-14T21:46:26.4062678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4062792Z outputs = self.electra( 2025-08-14T21:46:26.4063042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4063117Z hidden_states = self.encoder( 2025-08-14T21:46:26.4063366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4063433Z layer_outputs = layer_module( 2025-08-14T21:46:26.4063651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4063728Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4063974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4064061Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4064306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4064408Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4064692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.4064825Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.4065089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.4065170Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4065190Z 2025-08-14T21:46:26.4065299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4065498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4065583Z return mod(**inputs) 2025-08-14T21:46:26.4065855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4065926Z outputs = self.electra( 2025-08-14T21:46:26.4066184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4066262Z hidden_states = self.encoder( 2025-08-14T21:46:26.4066520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4066597Z layer_outputs = layer_module( 2025-08-14T21:46:26.4066834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4066915Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4067208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4067296Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4067559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4067632Z return func(*args, **kwargs) 2025-08-14T21:46:26.4067922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4068002Z self_outputs = self.self( 2025-08-14T21:46:26.4068262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4068333Z return func(*args, **kwargs) 2025-08-14T21:46:26.4068626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.4068712Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.4068717Z 2025-08-14T21:46:26.4068857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4069051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4069116Z return mod(**inputs) 2025-08-14T21:46:26.4069382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4069449Z outputs = self.electra( 2025-08-14T21:46:26.4069725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4069799Z hidden_states = self.encoder( 2025-08-14T21:46:26.4070071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4070150Z layer_outputs = layer_module( 2025-08-14T21:46:26.4070381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4070463Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4070765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4070851Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4071110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4071182Z return func(*args, **kwargs) 2025-08-14T21:46:26.4071456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4071557Z self_outputs = self.self( 2025-08-14T21:46:26.4071810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4071883Z return func(*args, **kwargs) 2025-08-14T21:46:26.4072183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.4072271Z key_layer = self.key(current_states) 2025-08-14T21:46:26.4072275Z 2025-08-14T21:46:26.4072393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4072601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4072668Z return mod(**inputs) 2025-08-14T21:46:26.4072954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4073026Z outputs = self.electra( 2025-08-14T21:46:26.4073308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4073382Z hidden_states = self.encoder( 2025-08-14T21:46:26.4073656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4073739Z layer_outputs = layer_module( 2025-08-14T21:46:26.4073973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4074055Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4074335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4074420Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4074679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4074755Z return func(*args, **kwargs) 2025-08-14T21:46:26.4075025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4075106Z self_outputs = self.self( 2025-08-14T21:46:26.4075364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4075458Z return func(*args, **kwargs) 2025-08-14T21:46:26.4075874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.4075971Z value_layer = self.value(current_states) 2025-08-14T21:46:26.4075975Z 2025-08-14T21:46:26.4076070Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4076154Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4076262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4076481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4076551Z return mod(**inputs) 2025-08-14T21:46:26.4076834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4076907Z outputs = self.electra( 2025-08-14T21:46:26.4077186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4077296Z hidden_states = self.encoder( 2025-08-14T21:46:26.4077570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4077643Z layer_outputs = layer_module( 2025-08-14T21:46:26.4077887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4077965Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4078264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4078345Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4078596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4078675Z return func(*args, **kwargs) 2025-08-14T21:46:26.4078933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.4079064Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.4079328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.4079409Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4079413Z 2025-08-14T21:46:26.4079523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4079720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4079784Z return mod(**inputs) 2025-08-14T21:46:26.4080053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4080124Z outputs = self.electra( 2025-08-14T21:46:26.4080394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4080464Z hidden_states = self.encoder( 2025-08-14T21:46:26.4080717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4080795Z layer_outputs = layer_module( 2025-08-14T21:46:26.4081013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4081090Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4081358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4081440Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4081703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4081799Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4082087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4082215Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4082469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.4082556Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4082559Z 2025-08-14T21:46:26.4082660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4082853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4082924Z return mod(**inputs) 2025-08-14T21:46:26.4083182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4083263Z outputs = self.electra( 2025-08-14T21:46:26.4083529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4083598Z hidden_states = self.encoder( 2025-08-14T21:46:26.4083865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4083935Z layer_outputs = layer_module( 2025-08-14T21:46:26.4084155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4084250Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4084508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4084611Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4084863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4084939Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4085233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4085351Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4085603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.4085724Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.4085933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.4086010Z return self.act(input) 2025-08-14T21:46:26.4086015Z 2025-08-14T21:46:26.4086115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4086312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4086387Z return mod(**inputs) 2025-08-14T21:46:26.4086646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4086719Z outputs = self.electra( 2025-08-14T21:46:26.4086972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4087044Z hidden_states = self.encoder( 2025-08-14T21:46:26.4087307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4087375Z layer_outputs = layer_module( 2025-08-14T21:46:26.4087593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4087700Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4087962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4088054Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4088319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4088401Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4088715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.4088865Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.4089130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.4089216Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4089221Z 2025-08-14T21:46:26.4089326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4089549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4089613Z return mod(**inputs) 2025-08-14T21:46:26.4089873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4089948Z outputs = self.electra( 2025-08-14T21:46:26.4090201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4090295Z hidden_states = self.encoder( 2025-08-14T21:46:26.4090550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4090619Z layer_outputs = layer_module( 2025-08-14T21:46:26.4090859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4090941Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4091198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4091285Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4091527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4091602Z return func(*args, **kwargs) 2025-08-14T21:46:26.4091860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4091929Z self_outputs = self.self( 2025-08-14T21:46:26.4092175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4092245Z return func(*args, **kwargs) 2025-08-14T21:46:26.4092524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:26.4092607Z query_layer = self.query(hidden_states) 2025-08-14T21:46:26.4092610Z 2025-08-14T21:46:26.4092712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4092914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4092977Z return mod(**inputs) 2025-08-14T21:46:26.4093241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4093320Z outputs = self.electra( 2025-08-14T21:46:26.4093577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4093653Z hidden_states = self.encoder( 2025-08-14T21:46:26.4093910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4093997Z layer_outputs = layer_module( 2025-08-14T21:46:26.4094220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4094296Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4094552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4094639Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4094876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4094954Z return func(*args, **kwargs) 2025-08-14T21:46:26.4095211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4095281Z self_outputs = self.self( 2025-08-14T21:46:26.4095529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4095614Z return func(*args, **kwargs) 2025-08-14T21:46:26.4095879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:26.4095957Z key_layer = self.key(current_states) 2025-08-14T21:46:26.4095960Z 2025-08-14T21:46:26.4096064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4096267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4096346Z return mod(**inputs) 2025-08-14T21:46:26.4096604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4096696Z outputs = self.electra( 2025-08-14T21:46:26.4096956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4097037Z hidden_states = self.encoder( 2025-08-14T21:46:26.4097294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4097362Z layer_outputs = layer_module( 2025-08-14T21:46:26.4097588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4097665Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4097928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4098007Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4098245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4098321Z return func(*args, **kwargs) 2025-08-14T21:46:26.4098576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:26.4098645Z self_outputs = self.self( 2025-08-14T21:46:26.4098887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4098953Z return func(*args, **kwargs) 2025-08-14T21:46:26.4099217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:26.4099295Z value_layer = self.value(current_states) 2025-08-14T21:46:26.4099299Z 2025-08-14T21:46:26.4099381Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4099466Z cudagraph partition due to non gpu ops 2025-08-14T21:46:26.4099568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4099764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4099852Z return mod(**inputs) 2025-08-14T21:46:26.4100111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4100184Z outputs = self.electra( 2025-08-14T21:46:26.4100440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4100508Z hidden_states = self.encoder( 2025-08-14T21:46:26.4100769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4100839Z layer_outputs = layer_module( 2025-08-14T21:46:26.4101057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4101148Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4101434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:26.4101560Z self_attention_outputs = self.attention( 2025-08-14T21:46:26.4101821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:26.4101893Z return func(*args, **kwargs) 2025-08-14T21:46:26.4102190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:26.4102325Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:26.4102629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:26.4102719Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4102723Z 2025-08-14T21:46:26.4102845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4103062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4103131Z return mod(**inputs) 2025-08-14T21:46:26.4103415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4103494Z outputs = self.electra( 2025-08-14T21:46:26.4103775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4103856Z hidden_states = self.encoder( 2025-08-14T21:46:26.4104142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4104215Z layer_outputs = layer_module( 2025-08-14T21:46:26.4104456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4104537Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4104831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4104921Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4105187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4105273Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4105577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4105702Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4105991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:26.4106077Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4106082Z 2025-08-14T21:46:26.4106216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4106430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4106501Z return mod(**inputs) 2025-08-14T21:46:26.4106796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4106868Z outputs = self.electra( 2025-08-14T21:46:26.4107158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4107231Z hidden_states = self.encoder( 2025-08-14T21:46:26.4107513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4107594Z layer_outputs = layer_module( 2025-08-14T21:46:26.4107835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4107916Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4108223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4108308Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4108585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4108833Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4109312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:26.4109496Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:26.4109788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:26.4109942Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:26.4110168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:26.4110242Z return self.act(input) 2025-08-14T21:46:26.4110246Z 2025-08-14T21:46:26.4110360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4110571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4110641Z return mod(**inputs) 2025-08-14T21:46:26.4110923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:46:26.4110998Z outputs = self.electra( 2025-08-14T21:46:26.4111279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:26.4111356Z hidden_states = self.encoder( 2025-08-14T21:46:26.4111632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:26.4111722Z layer_outputs = layer_module( 2025-08-14T21:46:26.4111958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:26.4112042Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:26.4112350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:26.4112440Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:26.4112728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:26.4112811Z return forward_fn(*input_tensors) 2025-08-14T21:46:26.4113128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:26.4113303Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:26.4113577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:26.4113670Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:26.4113673Z 2025-08-14T21:46:26.4113779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4113985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4114063Z return mod(**inputs) 2025-08-14T21:46:26.4114366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1560, in forward 2025-08-14T21:46:26.4114562Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-08-14T21:46:26.4114837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 640, in forward 2025-08-14T21:46:26.4114948Z hidden_states = self.dense(generator_hidden_states) 2025-08-14T21:46:26.4114977Z 2025-08-14T21:46:26.4115092Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4115295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4115364Z return mod(**inputs) 2025-08-14T21:46:26.4115719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1560, in forward 2025-08-14T21:46:26.4115920Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-08-14T21:46:26.4115924Z 2025-08-14T21:46:26.4116358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:26.4116577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:26.4116699Z return mod(**inputs) 2025-08-14T21:46:26.4116990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1564, in forward 2025-08-14T21:46:26.4117070Z lm_loss = self.loss_function( 2025-08-14T21:46:26.4117334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:46:26.4117520Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:46:26.4117784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:46:26.4117998Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:46:26.4118002Z 2025-08-14T21:46:34.6670236Z Compilation time (from dynamo_timed): 15.413001886 2025-08-14T21:46:34.6756262Z pass 2025-08-14T21:46:34.6756763Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:34.6757617Z TIMING: _recursive_pre_grad_passes:0.00763 _recursive_joint_graph_passes:0.46343 _recursive_post_grad_passes:0.08214 async_compile.wait:0.77297 code_gen:7.65492 inductor_compile:8.85349 backend_compile:12.50493 gc:0.00017 entire_frame_compile:15.413 total_wall_time:15.413 2025-08-14T21:46:34.6758623Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:15041 | FakeTensor.__torch_dispatch__:4687 | ProxyTorchDispatchMode.__torch_dispatch__:5671 2025-08-14T21:46:34.6760505Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2025-08-14T21:46:39.9796142Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:46:39.9797180Z from pkg_resources import resource_filename 2025-08-14T21:46:40.5677068Z 2025-08-14T21:46:40.9435520Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:46:40.9436128Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:46:40.9449972Z cpu eval ElectraForQuestionAnswering 2025-08-14T21:46:41.0644365Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:41.1232544Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:41.1824073Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:49.3543712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3544564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3544940Z return mod(**inputs) 2025-08-14T21:46:49.3545369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3545815Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3546586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 797, in forward 2025-08-14T21:46:49.3547029Z hidden_states = self.embeddings_project(hidden_states) 2025-08-14T21:46:49.3547215Z 2025-08-14T21:46:49.3547327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3547701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3548038Z return mod(**inputs) 2025-08-14T21:46:49.3548476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3548897Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3549370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3549776Z hidden_states = self.encoder( 2025-08-14T21:46:49.3550208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3550610Z layer_outputs = layer_module( 2025-08-14T21:46:49.3550965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3551325Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3551740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3552295Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3552717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3553116Z return func(*args, **kwargs) 2025-08-14T21:46:49.3553545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3553982Z self_outputs = self.self( 2025-08-14T21:46:49.3554390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3554800Z return func(*args, **kwargs) 2025-08-14T21:46:49.3555221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.3555858Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.3556032Z 2025-08-14T21:46:49.3556151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3556568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3556930Z return mod(**inputs) 2025-08-14T21:46:49.3557352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3557820Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3558221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3558610Z hidden_states = self.encoder( 2025-08-14T21:46:49.3558996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3559392Z layer_outputs = layer_module( 2025-08-14T21:46:49.3559735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3560107Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3560518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3560949Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3561350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3561754Z return func(*args, **kwargs) 2025-08-14T21:46:49.3562138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3562533Z self_outputs = self.self( 2025-08-14T21:46:49.3562901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3563295Z return func(*args, **kwargs) 2025-08-14T21:46:49.3563728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.3564145Z key_layer = self.key(current_states) 2025-08-14T21:46:49.3564305Z 2025-08-14T21:46:49.3564417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3564825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3565167Z return mod(**inputs) 2025-08-14T21:46:49.3565569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3566011Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3566441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3566843Z hidden_states = self.encoder( 2025-08-14T21:46:49.3567254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3567676Z layer_outputs = layer_module( 2025-08-14T21:46:49.3568044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3568430Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3568852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3569275Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3569676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3570066Z return func(*args, **kwargs) 2025-08-14T21:46:49.3570464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3570881Z self_outputs = self.self( 2025-08-14T21:46:49.3571254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3571645Z return func(*args, **kwargs) 2025-08-14T21:46:49.3572070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.3572509Z value_layer = self.value(current_states) 2025-08-14T21:46:49.3572658Z 2025-08-14T21:46:49.3572748Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3572984Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3574199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3574684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3575068Z return mod(**inputs) 2025-08-14T21:46:49.3575523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3575991Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3576505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3576992Z hidden_states = self.encoder( 2025-08-14T21:46:49.3577610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3578165Z layer_outputs = layer_module( 2025-08-14T21:46:49.3578906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3579485Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3580101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3580545Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3581388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3581917Z return func(*args, **kwargs) 2025-08-14T21:46:49.3582598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.3583123Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.3583581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.3583987Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3584132Z 2025-08-14T21:46:49.3584245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3584615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3584969Z return mod(**inputs) 2025-08-14T21:46:49.3585432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3585883Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3586400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3587024Z hidden_states = self.encoder( 2025-08-14T21:46:49.3587463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3587864Z layer_outputs = layer_module( 2025-08-14T21:46:49.3588223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3588623Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3589060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3589616Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3590047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3590465Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3590914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3591512Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3591983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.3592463Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3592629Z 2025-08-14T21:46:49.3592753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3593154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3593514Z return mod(**inputs) 2025-08-14T21:46:49.3593922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3594370Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3594818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3595261Z hidden_states = self.encoder( 2025-08-14T21:46:49.3595988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3596450Z layer_outputs = layer_module( 2025-08-14T21:46:49.3596844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3597258Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3597714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3598199Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3598626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3599066Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3599535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3600043Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3600527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.3600985Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.3601398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.3601783Z return self.act(input) 2025-08-14T21:46:49.3601907Z 2025-08-14T21:46:49.3602023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3602410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3602755Z return mod(**inputs) 2025-08-14T21:46:49.3603156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3603589Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3604046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3604464Z hidden_states = self.encoder( 2025-08-14T21:46:49.3604842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3605258Z layer_outputs = layer_module( 2025-08-14T21:46:49.3605633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3606015Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3606414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3606860Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3607284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3607711Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3608132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.3608624Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.3609389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.3609814Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3609968Z 2025-08-14T21:46:49.3610076Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3610443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3610778Z return mod(**inputs) 2025-08-14T21:46:49.3611180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3611686Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3612102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3612496Z hidden_states = self.encoder( 2025-08-14T21:46:49.3612881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3613280Z layer_outputs = layer_module( 2025-08-14T21:46:49.3613715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3614073Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3614506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3614924Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3615317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3615688Z return func(*args, **kwargs) 2025-08-14T21:46:49.3616076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3616489Z self_outputs = self.self( 2025-08-14T21:46:49.3616867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3617273Z return func(*args, **kwargs) 2025-08-14T21:46:49.3617675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.3618106Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.3618263Z 2025-08-14T21:46:49.3618368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3618738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3619075Z return mod(**inputs) 2025-08-14T21:46:49.3619470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3619891Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3620298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3620691Z hidden_states = self.encoder( 2025-08-14T21:46:49.3621073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3621469Z layer_outputs = layer_module( 2025-08-14T21:46:49.3621819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3622230Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3622628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3623036Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3623424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3623792Z return func(*args, **kwargs) 2025-08-14T21:46:49.3624195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3624625Z self_outputs = self.self( 2025-08-14T21:46:49.3625012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3625415Z return func(*args, **kwargs) 2025-08-14T21:46:49.3625824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.3626281Z key_layer = self.key(current_states) 2025-08-14T21:46:49.3626427Z 2025-08-14T21:46:49.3626546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3626927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3627272Z return mod(**inputs) 2025-08-14T21:46:49.3627671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3628142Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3628578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3629019Z hidden_states = self.encoder( 2025-08-14T21:46:49.3629430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3629844Z layer_outputs = layer_module( 2025-08-14T21:46:49.3630218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3630603Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3631021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3631451Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3631862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3632256Z return func(*args, **kwargs) 2025-08-14T21:46:49.3632654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3633068Z self_outputs = self.self( 2025-08-14T21:46:49.3633453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3633845Z return func(*args, **kwargs) 2025-08-14T21:46:49.3634239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.3634659Z value_layer = self.value(current_states) 2025-08-14T21:46:49.3634801Z 2025-08-14T21:46:49.3634896Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3635120Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3635374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3635868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3636243Z return mod(**inputs) 2025-08-14T21:46:49.3636684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3637208Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3637647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3638057Z hidden_states = self.encoder( 2025-08-14T21:46:49.3638468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3638882Z layer_outputs = layer_module( 2025-08-14T21:46:49.3639252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3639633Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3640055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3640492Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3640896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3641303Z return func(*args, **kwargs) 2025-08-14T21:46:49.3641685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.3642224Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.3642668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.3643088Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3643235Z 2025-08-14T21:46:49.3643382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3643774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3644153Z return mod(**inputs) 2025-08-14T21:46:49.3644557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3644993Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3645395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3645796Z hidden_states = self.encoder( 2025-08-14T21:46:49.3646182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3646580Z layer_outputs = layer_module( 2025-08-14T21:46:49.3646926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3647296Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3647718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3648155Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3648582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3648976Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3649424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3649897Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3650337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.3650752Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3650892Z 2025-08-14T21:46:49.3651005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3651367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3651741Z return mod(**inputs) 2025-08-14T21:46:49.3652121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3652536Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3652951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3653360Z hidden_states = self.encoder( 2025-08-14T21:46:49.3653761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3654175Z layer_outputs = layer_module( 2025-08-14T21:46:49.3654556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3654952Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3655409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3655870Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3656270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3656663Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3657076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3657570Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3658065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.3658594Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.3658997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.3659372Z return self.act(input) 2025-08-14T21:46:49.3659492Z 2025-08-14T21:46:49.3659605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3659996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3660318Z return mod(**inputs) 2025-08-14T21:46:49.3660702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3661150Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3661561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3661985Z hidden_states = self.encoder( 2025-08-14T21:46:49.3662385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3662799Z layer_outputs = layer_module( 2025-08-14T21:46:49.3663140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3663506Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3663911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3664323Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3664721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3665134Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3665585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.3666068Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.3666648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.3667105Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3667248Z 2025-08-14T21:46:49.3667365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3667728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3668079Z return mod(**inputs) 2025-08-14T21:46:49.3668477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3668919Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3669357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3669775Z hidden_states = self.encoder( 2025-08-14T21:46:49.3670203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3670633Z layer_outputs = layer_module( 2025-08-14T21:46:49.3671002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3671382Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3671801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3672223Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3672656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3673074Z return func(*args, **kwargs) 2025-08-14T21:46:49.3673476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3673915Z self_outputs = self.self( 2025-08-14T21:46:49.3674305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3674702Z return func(*args, **kwargs) 2025-08-14T21:46:49.3675103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.3675533Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.3675750Z 2025-08-14T21:46:49.3675910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3676303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3676652Z return mod(**inputs) 2025-08-14T21:46:49.3677034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3677448Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3677858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3678261Z hidden_states = self.encoder( 2025-08-14T21:46:49.3678647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3679043Z layer_outputs = layer_module( 2025-08-14T21:46:49.3679385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3679750Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3680157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3680562Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3680952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3681408Z return func(*args, **kwargs) 2025-08-14T21:46:49.3681797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3682187Z self_outputs = self.self( 2025-08-14T21:46:49.3682555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3682937Z return func(*args, **kwargs) 2025-08-14T21:46:49.3683313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.3683718Z key_layer = self.key(current_states) 2025-08-14T21:46:49.3683860Z 2025-08-14T21:46:49.3683969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3684340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3684663Z return mod(**inputs) 2025-08-14T21:46:49.3685039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3685488Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3685900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3686290Z hidden_states = self.encoder( 2025-08-14T21:46:49.3686683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3687089Z layer_outputs = layer_module( 2025-08-14T21:46:49.3687462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3687843Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3688307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3688747Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3689122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3689499Z return func(*args, **kwargs) 2025-08-14T21:46:49.3689881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3690297Z self_outputs = self.self( 2025-08-14T21:46:49.3690684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3691148Z return func(*args, **kwargs) 2025-08-14T21:46:49.3691554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.3691972Z value_layer = self.value(current_states) 2025-08-14T21:46:49.3692124Z 2025-08-14T21:46:49.3692214Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3692449Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3692701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3693077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3693422Z return mod(**inputs) 2025-08-14T21:46:49.3693814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3694245Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3694678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3695090Z hidden_states = self.encoder( 2025-08-14T21:46:49.3695500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3695906Z layer_outputs = layer_module( 2025-08-14T21:46:49.3696303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3696690Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3697108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3697541Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3697949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3698352Z return func(*args, **kwargs) 2025-08-14T21:46:49.3698750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.3699205Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.3699657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.3700067Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3700236Z 2025-08-14T21:46:49.3700340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3700704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3701030Z return mod(**inputs) 2025-08-14T21:46:49.3701399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3701813Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3702254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3702648Z hidden_states = self.encoder( 2025-08-14T21:46:49.3703038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3703436Z layer_outputs = layer_module( 2025-08-14T21:46:49.3703788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3704142Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3704542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3704949Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3705357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3705745Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3706172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3706652Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3707095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.3707503Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3707658Z 2025-08-14T21:46:49.3707769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3708154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3708475Z return mod(**inputs) 2025-08-14T21:46:49.3708995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3709420Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3709829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3710218Z hidden_states = self.encoder( 2025-08-14T21:46:49.3710607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3711069Z layer_outputs = layer_module( 2025-08-14T21:46:49.3711432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3711811Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3712230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3712658Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3713077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3713495Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3713947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3714449Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3714935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.3715396Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.3715864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.3716250Z return self.act(input) 2025-08-14T21:46:49.3716377Z 2025-08-14T21:46:49.3716489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3716917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3717274Z return mod(**inputs) 2025-08-14T21:46:49.3717694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3718105Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3718504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3718885Z hidden_states = self.encoder( 2025-08-14T21:46:49.3719255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3719641Z layer_outputs = layer_module( 2025-08-14T21:46:49.3719979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3720324Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3720715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3721113Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3721505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3721881Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3722303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.3722775Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.3723221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.3723615Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3723760Z 2025-08-14T21:46:49.3723876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3724235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3724551Z return mod(**inputs) 2025-08-14T21:46:49.3724942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3725377Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3725777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3726154Z hidden_states = self.encoder( 2025-08-14T21:46:49.3726539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3726923Z layer_outputs = layer_module( 2025-08-14T21:46:49.3727269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3727608Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3727987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3728377Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3728740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3729124Z return func(*args, **kwargs) 2025-08-14T21:46:49.3729486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3729862Z self_outputs = self.self( 2025-08-14T21:46:49.3730198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3730554Z return func(*args, **kwargs) 2025-08-14T21:46:49.3730934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.3731312Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.3731452Z 2025-08-14T21:46:49.3731568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3731909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3732218Z return mod(**inputs) 2025-08-14T21:46:49.3732567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3732964Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3733361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3733735Z hidden_states = self.encoder( 2025-08-14T21:46:49.3734096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3734464Z layer_outputs = layer_module( 2025-08-14T21:46:49.3734795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3735134Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3735516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3735905Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3736272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3736618Z return func(*args, **kwargs) 2025-08-14T21:46:49.3736984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3737357Z self_outputs = self.self( 2025-08-14T21:46:49.3737696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3738047Z return func(*args, **kwargs) 2025-08-14T21:46:49.3738418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.3738827Z key_layer = self.key(current_states) 2025-08-14T21:46:49.3739188Z 2025-08-14T21:46:49.3739293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3739645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3739961Z return mod(**inputs) 2025-08-14T21:46:49.3740327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3740727Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3741138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3741511Z hidden_states = self.encoder( 2025-08-14T21:46:49.3741873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3742254Z layer_outputs = layer_module( 2025-08-14T21:46:49.3742590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3742967Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3743352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3743755Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3744136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3744494Z return func(*args, **kwargs) 2025-08-14T21:46:49.3744868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3745244Z self_outputs = self.self( 2025-08-14T21:46:49.3745610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3745965Z return func(*args, **kwargs) 2025-08-14T21:46:49.3746344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.3746742Z value_layer = self.value(current_states) 2025-08-14T21:46:49.3746873Z 2025-08-14T21:46:49.3746964Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3747171Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3747404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3747764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3748077Z return mod(**inputs) 2025-08-14T21:46:49.3748444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3748849Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3749255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3749649Z hidden_states = self.encoder( 2025-08-14T21:46:49.3750080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3750509Z layer_outputs = layer_module( 2025-08-14T21:46:49.3750871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3751263Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3751710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3752138Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3752541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3752956Z return func(*args, **kwargs) 2025-08-14T21:46:49.3753366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.3753852Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.3754312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.3754738Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3754886Z 2025-08-14T21:46:49.3755004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3755380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3755817Z return mod(**inputs) 2025-08-14T21:46:49.3756246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3756718Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3757194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3757599Z hidden_states = self.encoder( 2025-08-14T21:46:49.3757979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3758355Z layer_outputs = layer_module( 2025-08-14T21:46:49.3758697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3759082Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3759476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3759886Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3760278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3760667Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3761081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3761541Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3761984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.3762419Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3762565Z 2025-08-14T21:46:49.3762686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3763061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3763404Z return mod(**inputs) 2025-08-14T21:46:49.3763804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3764238Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3764653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3765077Z hidden_states = self.encoder( 2025-08-14T21:46:49.3765485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3765906Z layer_outputs = layer_module( 2025-08-14T21:46:49.3766274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3766658Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3767090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3767552Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3767966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3768376Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3768812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3769321Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3769782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.3770253Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.3770655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.3771024Z return self.act(input) 2025-08-14T21:46:49.3771146Z 2025-08-14T21:46:49.3771270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3771604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3771940Z return mod(**inputs) 2025-08-14T21:46:49.3772307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3772711Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3773103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3773489Z hidden_states = self.encoder( 2025-08-14T21:46:49.3773896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3774284Z layer_outputs = layer_module( 2025-08-14T21:46:49.3774670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3775053Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3775474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3775891Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3776290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3776679Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3777108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.3777570Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.3778010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.3778402Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3778535Z 2025-08-14T21:46:49.3778647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3779000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3779320Z return mod(**inputs) 2025-08-14T21:46:49.3779694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3780097Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3780512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3780905Z hidden_states = self.encoder( 2025-08-14T21:46:49.3781304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3781679Z layer_outputs = layer_module( 2025-08-14T21:46:49.3782046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3782402Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3782795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3783199Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3783586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3783961Z return func(*args, **kwargs) 2025-08-14T21:46:49.3784348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3784773Z self_outputs = self.self( 2025-08-14T21:46:49.3785159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3785548Z return func(*args, **kwargs) 2025-08-14T21:46:49.3785975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.3786401Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.3786545Z 2025-08-14T21:46:49.3786660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3787034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3787373Z return mod(**inputs) 2025-08-14T21:46:49.3787805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3788243Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3788685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3789103Z hidden_states = self.encoder( 2025-08-14T21:46:49.3789530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3789940Z layer_outputs = layer_module( 2025-08-14T21:46:49.3790309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3790701Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3791141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3791573Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3791986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3792385Z return func(*args, **kwargs) 2025-08-14T21:46:49.3792805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3793223Z self_outputs = self.self( 2025-08-14T21:46:49.3793612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3794015Z return func(*args, **kwargs) 2025-08-14T21:46:49.3794432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.3794866Z key_layer = self.key(current_states) 2025-08-14T21:46:49.3795011Z 2025-08-14T21:46:49.3795130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3795527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3795960Z return mod(**inputs) 2025-08-14T21:46:49.3796391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3796857Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3797284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3797706Z hidden_states = self.encoder( 2025-08-14T21:46:49.3798126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3798546Z layer_outputs = layer_module( 2025-08-14T21:46:49.3798912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3799303Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3799732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3800156Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3800570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3800991Z return func(*args, **kwargs) 2025-08-14T21:46:49.3801389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3801793Z self_outputs = self.self( 2025-08-14T21:46:49.3802174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3802569Z return func(*args, **kwargs) 2025-08-14T21:46:49.3802972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.3803410Z value_layer = self.value(current_states) 2025-08-14T21:46:49.3803563Z 2025-08-14T21:46:49.3803650Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3803899Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3804146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3804535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3804882Z return mod(**inputs) 2025-08-14T21:46:49.3805281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3805714Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3806149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3806568Z hidden_states = self.encoder( 2025-08-14T21:46:49.3806975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3807393Z layer_outputs = layer_module( 2025-08-14T21:46:49.3807768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3808155Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3808576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3809151Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3809567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3809956Z return func(*args, **kwargs) 2025-08-14T21:46:49.3810346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.3810998Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.3811683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.3812177Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3812421Z 2025-08-14T21:46:49.3812536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3812931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3813309Z return mod(**inputs) 2025-08-14T21:46:49.3813718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3814160Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3814593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3815013Z hidden_states = self.encoder( 2025-08-14T21:46:49.3815424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3815859Z layer_outputs = layer_module( 2025-08-14T21:46:49.3831344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3831953Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3832413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3832878Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3833329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3833767Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3834269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3834799Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3835317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.3835895Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3836074Z 2025-08-14T21:46:49.3836196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3836600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3836968Z return mod(**inputs) 2025-08-14T21:46:49.3837379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3837841Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3838281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3838698Z hidden_states = self.encoder( 2025-08-14T21:46:49.3839115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3839534Z layer_outputs = layer_module( 2025-08-14T21:46:49.3839910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3840291Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3840713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3841138Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3841558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3841963Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3842408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3842912Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3843382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.3843822Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.3844200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.3844538Z return self.act(input) 2025-08-14T21:46:49.3844651Z 2025-08-14T21:46:49.3844757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3845126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3845456Z return mod(**inputs) 2025-08-14T21:46:49.3845840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3846253Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3846669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3847070Z hidden_states = self.encoder( 2025-08-14T21:46:49.3847487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3847907Z layer_outputs = layer_module( 2025-08-14T21:46:49.3848254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3848616Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3849003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3849423Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3849820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3850208Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3850621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.3851091Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.3851525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.3851906Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3852050Z 2025-08-14T21:46:49.3852152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3852504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3852824Z return mod(**inputs) 2025-08-14T21:46:49.3853178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3853581Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3853974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3854377Z hidden_states = self.encoder( 2025-08-14T21:46:49.3854789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3855182Z layer_outputs = layer_module( 2025-08-14T21:46:49.3855529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3855884Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3856283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3856675Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3857055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3857441Z return func(*args, **kwargs) 2025-08-14T21:46:49.3857828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3858223Z self_outputs = self.self( 2025-08-14T21:46:49.3858577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3858954Z return func(*args, **kwargs) 2025-08-14T21:46:49.3859329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.3859729Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.3859866Z 2025-08-14T21:46:49.3859971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3860331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3860654Z return mod(**inputs) 2025-08-14T21:46:49.3861022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3861451Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3861858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3862250Z hidden_states = self.encoder( 2025-08-14T21:46:49.3862632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3863022Z layer_outputs = layer_module( 2025-08-14T21:46:49.3863435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3863799Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3864206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3864610Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3864992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3865356Z return func(*args, **kwargs) 2025-08-14T21:46:49.3865735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3866125Z self_outputs = self.self( 2025-08-14T21:46:49.3866487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3866850Z return func(*args, **kwargs) 2025-08-14T21:46:49.3867235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.3867637Z key_layer = self.key(current_states) 2025-08-14T21:46:49.3867773Z 2025-08-14T21:46:49.3867878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3868255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3868596Z return mod(**inputs) 2025-08-14T21:46:49.3868987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3869393Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3869798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3870213Z hidden_states = self.encoder( 2025-08-14T21:46:49.3870620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3871021Z layer_outputs = layer_module( 2025-08-14T21:46:49.3871389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3871796Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3872216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3872655Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3873062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3873474Z return func(*args, **kwargs) 2025-08-14T21:46:49.3873904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3874329Z self_outputs = self.self( 2025-08-14T21:46:49.3874723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3875128Z return func(*args, **kwargs) 2025-08-14T21:46:49.3875542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.3876114Z value_layer = self.value(current_states) 2025-08-14T21:46:49.3876266Z 2025-08-14T21:46:49.3876367Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3876598Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3876861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3877251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3877617Z return mod(**inputs) 2025-08-14T21:46:49.3878070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3878524Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3879004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3879433Z hidden_states = self.encoder( 2025-08-14T21:46:49.3879849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3880355Z layer_outputs = layer_module( 2025-08-14T21:46:49.3880733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3881140Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3881567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3882005Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3882422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3882825Z return func(*args, **kwargs) 2025-08-14T21:46:49.3883234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.3883727Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.3884215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.3884645Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3884803Z 2025-08-14T21:46:49.3884916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3885308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3885674Z return mod(**inputs) 2025-08-14T21:46:49.3886062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3886499Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3886929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3887361Z hidden_states = self.encoder( 2025-08-14T21:46:49.3887762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3888249Z layer_outputs = layer_module( 2025-08-14T21:46:49.3888628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3888999Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3889417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3889841Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3890262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3890672Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3891122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3891648Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3892106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.3892543Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3892697Z 2025-08-14T21:46:49.3892809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3893205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3893541Z return mod(**inputs) 2025-08-14T21:46:49.3893972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3894401Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3894810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3895217Z hidden_states = self.encoder( 2025-08-14T21:46:49.3895620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3896044Z layer_outputs = layer_module( 2025-08-14T21:46:49.3896399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3896787Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3897180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3897605Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3898016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3898430Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3898874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3899370Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3899823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.3900286Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.3900680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.3901009Z return self.act(input) 2025-08-14T21:46:49.3901129Z 2025-08-14T21:46:49.3901234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3901594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3901976Z return mod(**inputs) 2025-08-14T21:46:49.3902369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3902805Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3903237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3903657Z hidden_states = self.encoder( 2025-08-14T21:46:49.3904054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3904469Z layer_outputs = layer_module( 2025-08-14T21:46:49.3904834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3905211Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3905633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3906057Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3906450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3906850Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3907298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.3907806Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.3908296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.3908956Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3909233Z 2025-08-14T21:46:49.3909349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3909730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3910080Z return mod(**inputs) 2025-08-14T21:46:49.3910486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3910932Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3911370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3911774Z hidden_states = self.encoder( 2025-08-14T21:46:49.3912183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3912608Z layer_outputs = layer_module( 2025-08-14T21:46:49.3912976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3913373Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3913800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3914249Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3914658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3915077Z return func(*args, **kwargs) 2025-08-14T21:46:49.3915495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3915990Z self_outputs = self.self( 2025-08-14T21:46:49.3916384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3916786Z return func(*args, **kwargs) 2025-08-14T21:46:49.3917224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.3917677Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.3917833Z 2025-08-14T21:46:49.3917945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3918327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3918670Z return mod(**inputs) 2025-08-14T21:46:49.3919055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3919535Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3919971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3920378Z hidden_states = self.encoder( 2025-08-14T21:46:49.3920787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3921212Z layer_outputs = layer_module( 2025-08-14T21:46:49.3921671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3922057Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3922473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3922907Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3923306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3923731Z return func(*args, **kwargs) 2025-08-14T21:46:49.3924108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3924523Z self_outputs = self.self( 2025-08-14T21:46:49.3924909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3925306Z return func(*args, **kwargs) 2025-08-14T21:46:49.3925692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.3926093Z key_layer = self.key(current_states) 2025-08-14T21:46:49.3926233Z 2025-08-14T21:46:49.3926348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3926735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3927082Z return mod(**inputs) 2025-08-14T21:46:49.3927455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3927874Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3928293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3928686Z hidden_states = self.encoder( 2025-08-14T21:46:49.3929059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3929445Z layer_outputs = layer_module( 2025-08-14T21:46:49.3929796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3930172Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3930558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3930956Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3931337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3931708Z return func(*args, **kwargs) 2025-08-14T21:46:49.3933205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3933598Z self_outputs = self.self( 2025-08-14T21:46:49.3933962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3934322Z return func(*args, **kwargs) 2025-08-14T21:46:49.3934706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.3935116Z value_layer = self.value(current_states) 2025-08-14T21:46:49.3935254Z 2025-08-14T21:46:49.3935339Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3935562Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3935809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3936171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3936489Z return mod(**inputs) 2025-08-14T21:46:49.3936891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3937298Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3937700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3938079Z hidden_states = self.encoder( 2025-08-14T21:46:49.3938447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3938843Z layer_outputs = layer_module( 2025-08-14T21:46:49.3939174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3939542Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3939939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3940342Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3940712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3941104Z return func(*args, **kwargs) 2025-08-14T21:46:49.3941502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.3941952Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.3942386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.3942787Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3942931Z 2025-08-14T21:46:49.3943043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3943433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3943795Z return mod(**inputs) 2025-08-14T21:46:49.3944191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3944637Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3945073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3945499Z hidden_states = self.encoder( 2025-08-14T21:46:49.3945910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3946334Z layer_outputs = layer_module( 2025-08-14T21:46:49.3946705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3947114Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3947534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3947969Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3948384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3948789Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3949234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3949710Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3950137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.3950522Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3950668Z 2025-08-14T21:46:49.3950773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3951158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3951485Z return mod(**inputs) 2025-08-14T21:46:49.3951851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3952293Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3952723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3953133Z hidden_states = self.encoder( 2025-08-14T21:46:49.3953570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3953993Z layer_outputs = layer_module( 2025-08-14T21:46:49.3954372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3954745Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3955163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3955594Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3956102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3956536Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3957001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3957484Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3957903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.3958324Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.3958699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.3959035Z return self.act(input) 2025-08-14T21:46:49.3959149Z 2025-08-14T21:46:49.3959257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3959621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3959946Z return mod(**inputs) 2025-08-14T21:46:49.3960313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3960728Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3961137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3961544Z hidden_states = self.encoder( 2025-08-14T21:46:49.3961973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3962349Z layer_outputs = layer_module( 2025-08-14T21:46:49.3962679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3963042Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3963436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3963841Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3964242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3964635Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3965090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.3965592Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.3966069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.3966486Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3966639Z 2025-08-14T21:46:49.3966753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3967133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3967477Z return mod(**inputs) 2025-08-14T21:46:49.3967889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3968321Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3968760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3969191Z hidden_states = self.encoder( 2025-08-14T21:46:49.3969592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3970007Z layer_outputs = layer_module( 2025-08-14T21:46:49.3970372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3970742Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3971162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3971591Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3971999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3972388Z return func(*args, **kwargs) 2025-08-14T21:46:49.3972791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3973205Z self_outputs = self.self( 2025-08-14T21:46:49.3973461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3973536Z return func(*args, **kwargs) 2025-08-14T21:46:49.3973816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.3973903Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.3973906Z 2025-08-14T21:46:49.3974026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3974233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3974301Z return mod(**inputs) 2025-08-14T21:46:49.3974587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3974711Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3974990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3975064Z hidden_states = self.encoder( 2025-08-14T21:46:49.3975332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3975416Z layer_outputs = layer_module( 2025-08-14T21:46:49.3975646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3975730Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3976010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3976097Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3976354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3976449Z return func(*args, **kwargs) 2025-08-14T21:46:49.3976730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3976810Z self_outputs = self.self( 2025-08-14T21:46:49.3977071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3977149Z return func(*args, **kwargs) 2025-08-14T21:46:49.3977465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.3977550Z key_layer = self.key(current_states) 2025-08-14T21:46:49.3977554Z 2025-08-14T21:46:49.3977688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3977898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3977969Z return mod(**inputs) 2025-08-14T21:46:49.3978251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3978342Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3978615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3978688Z hidden_states = self.encoder( 2025-08-14T21:46:49.3978960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3979041Z layer_outputs = layer_module( 2025-08-14T21:46:49.3979270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3979355Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3979632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3979720Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3979979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3980049Z return func(*args, **kwargs) 2025-08-14T21:46:49.3980305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3980381Z self_outputs = self.self( 2025-08-14T21:46:49.3980627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3980706Z return func(*args, **kwargs) 2025-08-14T21:46:49.3980979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.3981085Z value_layer = self.value(current_states) 2025-08-14T21:46:49.3981091Z 2025-08-14T21:46:49.3981185Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3981268Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.3981378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3981592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3981660Z return mod(**inputs) 2025-08-14T21:46:49.3981963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3982056Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3982330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3982413Z hidden_states = self.encoder( 2025-08-14T21:46:49.3982687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3982805Z layer_outputs = layer_module( 2025-08-14T21:46:49.3983059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3983136Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3983407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3983489Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3983755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3983830Z return func(*args, **kwargs) 2025-08-14T21:46:49.3984101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.3984241Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.3984506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.3984593Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3984597Z 2025-08-14T21:46:49.3984713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3984918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3984987Z return mod(**inputs) 2025-08-14T21:46:49.3985272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3985363Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3985647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3985724Z hidden_states = self.encoder( 2025-08-14T21:46:49.3986004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3986092Z layer_outputs = layer_module( 2025-08-14T21:46:49.3986328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3986413Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3986669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3986751Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3987013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3987089Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3987380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3987537Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3987817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.3987910Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3987914Z 2025-08-14T21:46:49.3988021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3988238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3988314Z return mod(**inputs) 2025-08-14T21:46:49.3988603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3988702Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3988983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3989084Z hidden_states = self.encoder( 2025-08-14T21:46:49.3989368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3989442Z layer_outputs = layer_module( 2025-08-14T21:46:49.3989682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3989771Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3990071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3990167Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3990436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3990536Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3990855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.3990983Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.3991277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.3991395Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.3991627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.3991707Z return self.act(input) 2025-08-14T21:46:49.3991712Z 2025-08-14T21:46:49.3991823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3992044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3992121Z return mod(**inputs) 2025-08-14T21:46:49.3992413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3992514Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3992796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3992869Z hidden_states = self.encoder( 2025-08-14T21:46:49.3993164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3993237Z layer_outputs = layer_module( 2025-08-14T21:46:49.3993483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3993565Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3993846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.3993964Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.3994235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.3994319Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.3994635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.3994778Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.3995098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.3995191Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.3995194Z 2025-08-14T21:46:49.3995309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3995536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3995612Z return mod(**inputs) 2025-08-14T21:46:49.3996011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3996108Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.3996407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.3996493Z hidden_states = self.encoder( 2025-08-14T21:46:49.3996790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.3996893Z layer_outputs = layer_module( 2025-08-14T21:46:49.3997150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.3997260Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.3997543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.3997632Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.3997885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3997969Z return func(*args, **kwargs) 2025-08-14T21:46:49.3998243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.3998325Z self_outputs = self.self( 2025-08-14T21:46:49.3998576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.3998650Z return func(*args, **kwargs) 2025-08-14T21:46:49.3998928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.3999016Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.3999023Z 2025-08-14T21:46:49.3999134Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.3999348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.3999416Z return mod(**inputs) 2025-08-14T21:46:49.3999698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.3999789Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4000061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4000142Z hidden_states = self.encoder( 2025-08-14T21:46:49.4000412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4000487Z layer_outputs = layer_module( 2025-08-14T21:46:49.4000744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4000828Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4001106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4001191Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4001441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4001520Z return func(*args, **kwargs) 2025-08-14T21:46:49.4001793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4001871Z self_outputs = self.self( 2025-08-14T21:46:49.4002121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4002194Z return func(*args, **kwargs) 2025-08-14T21:46:49.4002470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.4002574Z key_layer = self.key(current_states) 2025-08-14T21:46:49.4002578Z 2025-08-14T21:46:49.4002684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4002899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4002966Z return mod(**inputs) 2025-08-14T21:46:49.4003265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4003356Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4003644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4003731Z hidden_states = self.encoder( 2025-08-14T21:46:49.4004006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4004090Z layer_outputs = layer_module( 2025-08-14T21:46:49.4004324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4004405Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4004704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4004788Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4005044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4005122Z return func(*args, **kwargs) 2025-08-14T21:46:49.4005400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4005480Z self_outputs = self.self( 2025-08-14T21:46:49.4005737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4005807Z return func(*args, **kwargs) 2025-08-14T21:46:49.4006090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.4006174Z value_layer = self.value(current_states) 2025-08-14T21:46:49.4006178Z 2025-08-14T21:46:49.4006263Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.4006352Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.4006462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4006679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4006749Z return mod(**inputs) 2025-08-14T21:46:49.4007026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4007154Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4007423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4007498Z hidden_states = self.encoder( 2025-08-14T21:46:49.4007775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4007849Z layer_outputs = layer_module( 2025-08-14T21:46:49.4008087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4008166Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4008437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4008531Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4008997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4009189Z return func(*args, **kwargs) 2025-08-14T21:46:49.4009486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.4009621Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.4009899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.4010014Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4010019Z 2025-08-14T21:46:49.4010135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4010369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4010442Z return mod(**inputs) 2025-08-14T21:46:49.4010721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4010814Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4011085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4011168Z hidden_states = self.encoder( 2025-08-14T21:46:49.4011436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4011515Z layer_outputs = layer_module( 2025-08-14T21:46:49.4011746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4011828Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4012106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4012194Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4012461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4012550Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4012863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.4012996Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.4013267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.4013353Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4013357Z 2025-08-14T21:46:49.4013472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4013680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4013797Z return mod(**inputs) 2025-08-14T21:46:49.4014051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4014135Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4014394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4014463Z hidden_states = self.encoder( 2025-08-14T21:46:49.4014713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4014788Z layer_outputs = layer_module( 2025-08-14T21:46:49.4015003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4015087Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4015337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4015434Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4015688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4015763Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4016052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.4016168Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.4016436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.4016557Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.4016773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.4016847Z return self.act(input) 2025-08-14T21:46:49.4016858Z 2025-08-14T21:46:49.4016958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4017148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4017219Z return mod(**inputs) 2025-08-14T21:46:49.4017468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4017551Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4017807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4017877Z hidden_states = self.encoder( 2025-08-14T21:46:49.4018131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4018201Z layer_outputs = layer_module( 2025-08-14T21:46:49.4018415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4018497Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4018745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4018825Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4019075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4019150Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4019434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.4019564Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.4019832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.4019921Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4019924Z 2025-08-14T21:46:49.4020024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4020219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4020282Z return mod(**inputs) 2025-08-14T21:46:49.4020533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4020625Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4020873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4020943Z hidden_states = self.encoder( 2025-08-14T21:46:49.4021199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4021287Z layer_outputs = layer_module( 2025-08-14T21:46:49.4021504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4021580Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4021831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4021917Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4022182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4022260Z return func(*args, **kwargs) 2025-08-14T21:46:49.4022532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4022602Z self_outputs = self.self( 2025-08-14T21:46:49.4022839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4022907Z return func(*args, **kwargs) 2025-08-14T21:46:49.4023154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.4023241Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.4023244Z 2025-08-14T21:46:49.4023343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4023538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4023604Z return mod(**inputs) 2025-08-14T21:46:49.4023854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4023947Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4024198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4024266Z hidden_states = self.encoder( 2025-08-14T21:46:49.4024521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4024589Z layer_outputs = layer_module( 2025-08-14T21:46:49.4024807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4024882Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4025132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4025217Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4025447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4025537Z return func(*args, **kwargs) 2025-08-14T21:46:49.4025789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4025857Z self_outputs = self.self( 2025-08-14T21:46:49.4026096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4026161Z return func(*args, **kwargs) 2025-08-14T21:46:49.4026411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.4026497Z key_layer = self.key(current_states) 2025-08-14T21:46:49.4026500Z 2025-08-14T21:46:49.4026599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4026797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4026861Z return mod(**inputs) 2025-08-14T21:46:49.4027115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4027252Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4027512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4027590Z hidden_states = self.encoder( 2025-08-14T21:46:49.4027847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4027917Z layer_outputs = layer_module( 2025-08-14T21:46:49.4028157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4028236Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4028508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4028600Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4028838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4028914Z return func(*args, **kwargs) 2025-08-14T21:46:49.4029174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4029243Z self_outputs = self.self( 2025-08-14T21:46:49.4029487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4029557Z return func(*args, **kwargs) 2025-08-14T21:46:49.4029822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.4029909Z value_layer = self.value(current_states) 2025-08-14T21:46:49.4029914Z 2025-08-14T21:46:49.4029992Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.4030081Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.4030186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4030384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4030456Z return mod(**inputs) 2025-08-14T21:46:49.4030716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4030803Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4031088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4031159Z hidden_states = self.encoder( 2025-08-14T21:46:49.4031423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4031514Z layer_outputs = layer_module( 2025-08-14T21:46:49.4031731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4031817Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4032072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4032158Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4032405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4032473Z return func(*args, **kwargs) 2025-08-14T21:46:49.4032736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.4032866Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.4033151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.4033264Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4033268Z 2025-08-14T21:46:49.4033374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4033590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4033657Z return mod(**inputs) 2025-08-14T21:46:49.4033944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4034043Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4034350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4034432Z hidden_states = self.encoder( 2025-08-14T21:46:49.4034731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4034810Z layer_outputs = layer_module( 2025-08-14T21:46:49.4035052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4035133Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4035418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4035514Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4035868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4035961Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4036275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.4036403Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.4036689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.4036778Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4036782Z 2025-08-14T21:46:49.4036897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4037109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4037179Z return mod(**inputs) 2025-08-14T21:46:49.4037461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4037548Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4037803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4037886Z hidden_states = self.encoder( 2025-08-14T21:46:49.4038165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4038245Z layer_outputs = layer_module( 2025-08-14T21:46:49.4038462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4038537Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4038797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4038879Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4039135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4039209Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4039495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.4039619Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.4039894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.4040006Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.4040219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.4040287Z return self.act(input) 2025-08-14T21:46:49.4040291Z 2025-08-14T21:46:49.4040399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4040608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4040674Z return mod(**inputs) 2025-08-14T21:46:49.4040958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4041046Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4041307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4041377Z hidden_states = self.encoder( 2025-08-14T21:46:49.4041630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4041708Z layer_outputs = layer_module( 2025-08-14T21:46:49.4041922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4042000Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4042260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4042340Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4042599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4042675Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4042958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.4043095Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.4043359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.4043451Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4043454Z 2025-08-14T21:46:49.4043561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4043767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4043844Z return mod(**inputs) 2025-08-14T21:46:49.4044166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4044259Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4044537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4044613Z hidden_states = self.encoder( 2025-08-14T21:46:49.4044888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4044964Z layer_outputs = layer_module( 2025-08-14T21:46:49.4045194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4045283Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4045552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4045648Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4045930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4046000Z return func(*args, **kwargs) 2025-08-14T21:46:49.4046260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4046329Z self_outputs = self.self( 2025-08-14T21:46:49.4046568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4046647Z return func(*args, **kwargs) 2025-08-14T21:46:49.4046925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.4047016Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.4047037Z 2025-08-14T21:46:49.4047139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4047334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4047408Z return mod(**inputs) 2025-08-14T21:46:49.4047666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4047751Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4048014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4048084Z hidden_states = self.encoder( 2025-08-14T21:46:49.4048346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4048415Z layer_outputs = layer_module( 2025-08-14T21:46:49.4048631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4048717Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4048980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4049071Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4049322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4049389Z return func(*args, **kwargs) 2025-08-14T21:46:49.4049649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4049719Z self_outputs = self.self( 2025-08-14T21:46:49.4049954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4050030Z return func(*args, **kwargs) 2025-08-14T21:46:49.4050288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.4050395Z key_layer = self.key(current_states) 2025-08-14T21:46:49.4050399Z 2025-08-14T21:46:49.4050499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4050693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4050766Z return mod(**inputs) 2025-08-14T21:46:49.4051020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4051114Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4051369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4051440Z hidden_states = self.encoder( 2025-08-14T21:46:49.4051702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4051772Z layer_outputs = layer_module( 2025-08-14T21:46:49.4052007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4052092Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4052344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4052485Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4052728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4052816Z return func(*args, **kwargs) 2025-08-14T21:46:49.4053084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4053173Z self_outputs = self.self( 2025-08-14T21:46:49.4053410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4053486Z return func(*args, **kwargs) 2025-08-14T21:46:49.4053741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.4053828Z value_layer = self.value(current_states) 2025-08-14T21:46:49.4053832Z 2025-08-14T21:46:49.4053909Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.4053990Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.4054096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4054292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4054360Z return mod(**inputs) 2025-08-14T21:46:49.4054647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4054741Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4055023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4055118Z hidden_states = self.encoder( 2025-08-14T21:46:49.4055391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4055474Z layer_outputs = layer_module( 2025-08-14T21:46:49.4055702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4055789Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4056062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4056157Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4056400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4056490Z return func(*args, **kwargs) 2025-08-14T21:46:49.4056748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.4056882Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.4057144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.4057231Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4057234Z 2025-08-14T21:46:49.4057336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4057538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4057614Z return mod(**inputs) 2025-08-14T21:46:49.4057893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4057991Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4058286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4058361Z hidden_states = self.encoder( 2025-08-14T21:46:49.4058660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4058733Z layer_outputs = layer_module( 2025-08-14T21:46:49.4058965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4059072Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4059347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4059452Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4059707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4059784Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4060078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.4060197Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.4060458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.4060537Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4060542Z 2025-08-14T21:46:49.4060646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4060848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4060915Z return mod(**inputs) 2025-08-14T21:46:49.4061177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4061272Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4061526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4061602Z hidden_states = self.encoder( 2025-08-14T21:46:49.4061857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4061927Z layer_outputs = layer_module( 2025-08-14T21:46:49.4062154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4062230Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4062492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4062591Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4062844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4062925Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4063213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.4063328Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.4063590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.4063701Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.4063918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.4063987Z return self.act(input) 2025-08-14T21:46:49.4063992Z 2025-08-14T21:46:49.4064091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4064323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4064385Z return mod(**inputs) 2025-08-14T21:46:49.4064644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4064729Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4064983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4065077Z hidden_states = self.encoder( 2025-08-14T21:46:49.4065334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4065419Z layer_outputs = layer_module( 2025-08-14T21:46:49.4065644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4065724Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4065984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4066066Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4066316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4066399Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4066691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.4066828Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.4067087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.4067170Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4067175Z 2025-08-14T21:46:49.4067282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4067476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4067541Z return mod(**inputs) 2025-08-14T21:46:49.4067807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4067892Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4068155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4068225Z hidden_states = self.encoder( 2025-08-14T21:46:49.4068478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4068591Z layer_outputs = layer_module( 2025-08-14T21:46:49.4068813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4068898Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4069160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4069241Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4069491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4069562Z return func(*args, **kwargs) 2025-08-14T21:46:49.4069823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4069901Z self_outputs = self.self( 2025-08-14T21:46:49.4070145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4070242Z return func(*args, **kwargs) 2025-08-14T21:46:49.4070519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:46:49.4070605Z query_layer = self.query(hidden_states) 2025-08-14T21:46:49.4070609Z 2025-08-14T21:46:49.4070721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4070938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4071004Z return mod(**inputs) 2025-08-14T21:46:49.4071323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4071414Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4071717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4071794Z hidden_states = self.encoder( 2025-08-14T21:46:49.4072080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4072160Z layer_outputs = layer_module( 2025-08-14T21:46:49.4072396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4072482Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4072762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4072848Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4073104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4073177Z return func(*args, **kwargs) 2025-08-14T21:46:49.4073460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4073542Z self_outputs = self.self( 2025-08-14T21:46:49.4073803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4073883Z return func(*args, **kwargs) 2025-08-14T21:46:49.4074166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:46:49.4074249Z key_layer = self.key(current_states) 2025-08-14T21:46:49.4074253Z 2025-08-14T21:46:49.4074365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4074581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4074657Z return mod(**inputs) 2025-08-14T21:46:49.4074943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4075053Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4075343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4075418Z hidden_states = self.encoder( 2025-08-14T21:46:49.4075752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4075843Z layer_outputs = layer_module( 2025-08-14T21:46:49.4076125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4076220Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4076510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4076599Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4076865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4076975Z return func(*args, **kwargs) 2025-08-14T21:46:49.4077233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:46:49.4077313Z self_outputs = self.self( 2025-08-14T21:46:49.4077552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4077628Z return func(*args, **kwargs) 2025-08-14T21:46:49.4077904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:46:49.4077986Z value_layer = self.value(current_states) 2025-08-14T21:46:49.4077990Z 2025-08-14T21:46:49.4078082Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.4078180Z cudagraph partition due to non gpu ops 2025-08-14T21:46:49.4078293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4078490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4078555Z return mod(**inputs) 2025-08-14T21:46:49.4078824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4078916Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4079189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4079270Z hidden_states = self.encoder( 2025-08-14T21:46:49.4079541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4079618Z layer_outputs = layer_module( 2025-08-14T21:46:49.4079837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4079917Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4080195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:46:49.4080281Z self_attention_outputs = self.attention( 2025-08-14T21:46:49.4080532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:49.4080625Z return func(*args, **kwargs) 2025-08-14T21:46:49.4080882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:46:49.4081017Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:49.4081274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:46:49.4081372Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4081376Z 2025-08-14T21:46:49.4081486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4081680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4081751Z return mod(**inputs) 2025-08-14T21:46:49.4082011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4082097Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4082364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4082435Z hidden_states = self.encoder( 2025-08-14T21:46:49.4082691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4082773Z layer_outputs = layer_module( 2025-08-14T21:46:49.4082993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4083097Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4083368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4083456Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4083729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4083808Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4084143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.4084271Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.4084559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:46:49.4084657Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4084660Z 2025-08-14T21:46:49.4084767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4084973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4085048Z return mod(**inputs) 2025-08-14T21:46:49.4085320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4085419Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4085695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4085770Z hidden_states = self.encoder( 2025-08-14T21:46:49.4086051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4086125Z layer_outputs = layer_module( 2025-08-14T21:46:49.4086363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4086444Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4086716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4086812Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4087078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4087158Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4087469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:46:49.4087593Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:49.4087891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:46:49.4088012Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:49.4088235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:49.4088315Z return self.act(input) 2025-08-14T21:46:49.4088319Z 2025-08-14T21:46:49.4088426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4088642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4088712Z return mod(**inputs) 2025-08-14T21:46:49.4088986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:46:49.4089086Z discriminator_hidden_states = self.electra( 2025-08-14T21:46:49.4089361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:46:49.4089455Z hidden_states = self.encoder( 2025-08-14T21:46:49.4089733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:46:49.4089806Z layer_outputs = layer_module( 2025-08-14T21:46:49.4090043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:49.4090126Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:49.4090410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:46:49.4090507Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:49.4090807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:49.4090896Z return forward_fn(*input_tensors) 2025-08-14T21:46:49.4091202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:46:49.4091344Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:49.4091624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:46:49.4091708Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:49.4091712Z 2025-08-14T21:46:49.4091817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4092033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4092101Z return mod(**inputs) 2025-08-14T21:46:49.4092385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1330, in forward 2025-08-14T21:46:49.4092473Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:46:49.4092476Z 2025-08-14T21:46:49.4092585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4092797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4092867Z return mod(**inputs) 2025-08-14T21:46:49.4093148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1348, in forward 2025-08-14T21:46:49.4093258Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:46:49.4093262Z 2025-08-14T21:46:49.4093366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:49.4093584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:49.4093653Z return mod(**inputs) 2025-08-14T21:46:49.4093939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1349, in forward 2025-08-14T21:46:49.4094069Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:46:49.4094075Z 2025-08-14T21:46:56.6605006Z Compilation time (from dynamo_timed): 14.35121329 2025-08-14T21:46:56.6605343Z pass 2025-08-14T21:46:56.6605646Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:56.6618767Z TIMING: _recursive_pre_grad_passes:0.00779 _recursive_joint_graph_passes:0.46155 _recursive_post_grad_passes:0.08804 async_compile.wait:0.00211 code_gen:6.70383 inductor_compile:7.89396 backend_compile:11.49422 gc:0.00029 entire_frame_compile:14.35121 total_wall_time:14.35121 2025-08-14T21:46:56.6626778Z STATS: call_* op count: 378 | FakeTensorMode.__torch_dispatch__:15006 | FakeTensor.__torch_dispatch__:4704 | ProxyTorchDispatchMode.__torch_dispatch__:5698 2025-08-14T21:46:56.6632439Z Dynamo produced 1 graphs covering 378 ops with 0 graph breaks (0 unique) 2025-08-14T21:47:01.9725845Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:47:01.9727215Z from pkg_resources import resource_filename 2025-08-14T21:47:02.5591231Z 2025-08-14T21:47:04.3715899Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:47:04.3736291Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:47:04.3736752Z cpu eval GPT2ForSequenceClassification 2025-08-14T21:47:05.1229660Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:05.4518326Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:05.7754298Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:12.6830686Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6831012Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6831275Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6831552Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6831767Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6831987Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6837586Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6837894Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6838139Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6838367Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6838598Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6838849Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6839119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6839567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6839957Z return mod(**inputs) 2025-08-14T21:47:12.6840409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1509, in forward 2025-08-14T21:47:12.6840921Z last_non_pad_token = (token_indices * non_pad_mask).argmax(-1) 2025-08-14T21:47:12.6841131Z 2025-08-14T21:47:12.6841250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6841658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6842064Z return mod(**inputs) 2025-08-14T21:47:12.6842479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6842932Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6843406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6843828Z outputs = block( 2025-08-14T21:47:12.6844555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6844970Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6845398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6845830Z return func(*args, **kwargs) 2025-08-14T21:47:12.6846258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6846752Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6847224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6847665Z return func(*args, **kwargs) 2025-08-14T21:47:12.6848112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.6848725Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.6849332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6849793Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6850096Z 2025-08-14T21:47:12.6850192Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6850423Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6850643Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6850855Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6851086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6851505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6851837Z return mod(**inputs) 2025-08-14T21:47:12.6852255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6852659Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6853081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6853485Z outputs = block( 2025-08-14T21:47:12.6853836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6854222Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6854629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6855045Z return func(*args, **kwargs) 2025-08-14T21:47:12.6855447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6855879Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6856296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6856710Z return func(*args, **kwargs) 2025-08-14T21:47:12.6857090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.6857523Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.6858037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.6858551Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.6858750Z 2025-08-14T21:47:12.6858865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6859261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6859612Z return mod(**inputs) 2025-08-14T21:47:12.6859999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6860467Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6860885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6861280Z outputs = block( 2025-08-14T21:47:12.6861625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6862015Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6862427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6862832Z return func(*args, **kwargs) 2025-08-14T21:47:12.6863234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6863657Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6864080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6864502Z return func(*args, **kwargs) 2025-08-14T21:47:12.6864894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.6865330Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.6865807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.6866315Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.6866513Z 2025-08-14T21:47:12.6866626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6867009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6867360Z return mod(**inputs) 2025-08-14T21:47:12.6867743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6868159Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6868569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6868949Z outputs = block( 2025-08-14T21:47:12.6869286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6869674Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6870066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6870453Z return func(*args, **kwargs) 2025-08-14T21:47:12.6870838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6871252Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6871651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6872042Z return func(*args, **kwargs) 2025-08-14T21:47:12.6872427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.6872831Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.6873217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6873668Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6873855Z 2025-08-14T21:47:12.6873974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6874347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6874692Z return mod(**inputs) 2025-08-14T21:47:12.6875102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6875534Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6876196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6876606Z outputs = block( 2025-08-14T21:47:12.6876960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6877374Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6877781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6878177Z return func(*args, **kwargs) 2025-08-14T21:47:12.6878570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.6879000Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.6879431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.6879875Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.6880225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6880626Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6880806Z 2025-08-14T21:47:12.6880910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6881290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6881609Z return mod(**inputs) 2025-08-14T21:47:12.6881969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6882376Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6882761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6883139Z outputs = block( 2025-08-14T21:47:12.6883476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6883857Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6884254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6884624Z return func(*args, **kwargs) 2025-08-14T21:47:12.6884988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.6885396Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.6885797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.6886180Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.6886529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.6886974Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.6887209Z 2025-08-14T21:47:12.6887314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6887674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6888023Z return mod(**inputs) 2025-08-14T21:47:12.6888373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6888761Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6889144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6889532Z outputs = block( 2025-08-14T21:47:12.6889847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6890212Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6890609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6890995Z return func(*args, **kwargs) 2025-08-14T21:47:12.6891358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.6891761Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.6892169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.6892560Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.6892947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6893392Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6893575Z 2025-08-14T21:47:12.6893693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6894064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6894403Z return mod(**inputs) 2025-08-14T21:47:12.6894773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6895191Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6895637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6896025Z outputs = block( 2025-08-14T21:47:12.6897127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6897535Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6897944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6898350Z return func(*args, **kwargs) 2025-08-14T21:47:12.6898728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6899150Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6899565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6899968Z return func(*args, **kwargs) 2025-08-14T21:47:12.6900347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.6900870Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.6901374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6901808Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6901992Z 2025-08-14T21:47:12.6902080Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6902315Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6902541Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6902752Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6903004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6903406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6903757Z return mod(**inputs) 2025-08-14T21:47:12.6904150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6904577Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6905012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6905479Z outputs = block( 2025-08-14T21:47:12.6905822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6906208Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6906611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6907008Z return func(*args, **kwargs) 2025-08-14T21:47:12.6907394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6907808Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6908208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6908602Z return func(*args, **kwargs) 2025-08-14T21:47:12.6909327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.6909763Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.6910245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.6910773Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.6910970Z 2025-08-14T21:47:12.6911092Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6911540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6911889Z return mod(**inputs) 2025-08-14T21:47:12.6912311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6912744Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6913165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6913567Z outputs = block( 2025-08-14T21:47:12.6913916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6914312Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6914720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6915124Z return func(*args, **kwargs) 2025-08-14T21:47:12.6915523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6916018Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6916469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6916874Z return func(*args, **kwargs) 2025-08-14T21:47:12.6917276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.6917707Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.6918202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.6918712Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.6918889Z 2025-08-14T21:47:12.6919012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6919400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6919755Z return mod(**inputs) 2025-08-14T21:47:12.6920151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6920609Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6921039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6921438Z outputs = block( 2025-08-14T21:47:12.6921788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6922170Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6922580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6922985Z return func(*args, **kwargs) 2025-08-14T21:47:12.6923374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6923798Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6924217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6924610Z return func(*args, **kwargs) 2025-08-14T21:47:12.6924968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.6925358Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.6925716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6926128Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6926320Z 2025-08-14T21:47:12.6926430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6926827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6927169Z return mod(**inputs) 2025-08-14T21:47:12.6927556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6927972Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6928362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6928730Z outputs = block( 2025-08-14T21:47:12.6929044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6929408Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6929787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6930154Z return func(*args, **kwargs) 2025-08-14T21:47:12.6930525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.6930937Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.6931344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.6931725Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.6932083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6932484Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6932655Z 2025-08-14T21:47:12.6932759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6933121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6933445Z return mod(**inputs) 2025-08-14T21:47:12.6933805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6934194Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6934584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6934969Z outputs = block( 2025-08-14T21:47:12.6935301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6935655Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6936033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6936403Z return func(*args, **kwargs) 2025-08-14T21:47:12.6936762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.6937169Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.6937575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.6937962Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.6938308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.6938784Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.6939020Z 2025-08-14T21:47:12.6939133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6939497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6939814Z return mod(**inputs) 2025-08-14T21:47:12.6940171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6940589Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6940960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6941332Z outputs = block( 2025-08-14T21:47:12.6941648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6942004Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6942374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6942746Z return func(*args, **kwargs) 2025-08-14T21:47:12.6943123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.6943531Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.6943947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.6944352Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.6944731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6945145Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6945335Z 2025-08-14T21:47:12.6945445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6945829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6946171Z return mod(**inputs) 2025-08-14T21:47:12.6946532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6946925Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6947317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6947692Z outputs = block( 2025-08-14T21:47:12.6948031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6948411Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6948836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6949223Z return func(*args, **kwargs) 2025-08-14T21:47:12.6949614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6950035Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6950440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6950991Z return func(*args, **kwargs) 2025-08-14T21:47:12.6951387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.6951905Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.6952393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6952819Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6953045Z 2025-08-14T21:47:12.6953143Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6953383Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6953608Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6953834Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.6954095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6954485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6954845Z return mod(**inputs) 2025-08-14T21:47:12.6955271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6955753Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6956211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6956623Z outputs = block( 2025-08-14T21:47:12.6956972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6957328Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6957719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6958125Z return func(*args, **kwargs) 2025-08-14T21:47:12.6958516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6958949Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6959372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6959774Z return func(*args, **kwargs) 2025-08-14T21:47:12.6960167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.6960608Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.6961099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.6961652Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.6961853Z 2025-08-14T21:47:12.6961969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6962373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6962753Z return mod(**inputs) 2025-08-14T21:47:12.6963201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6963651Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6964099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6964504Z outputs = block( 2025-08-14T21:47:12.6964855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6965260Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6965688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6966093Z return func(*args, **kwargs) 2025-08-14T21:47:12.6966507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6966942Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6967434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6967817Z return func(*args, **kwargs) 2025-08-14T21:47:12.6968182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.6968603Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.6969044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.6969490Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.6969656Z 2025-08-14T21:47:12.6969761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6970138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6970460Z return mod(**inputs) 2025-08-14T21:47:12.6970819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6971229Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6971616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6971978Z outputs = block( 2025-08-14T21:47:12.6972297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6972655Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6973022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6973393Z return func(*args, **kwargs) 2025-08-14T21:47:12.6973761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.6974152Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.6974532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6974901Z return func(*args, **kwargs) 2025-08-14T21:47:12.6975266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.6975651Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.6976001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6976400Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6976572Z 2025-08-14T21:47:12.6976685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6977040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6977368Z return mod(**inputs) 2025-08-14T21:47:12.6977729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6978127Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6978525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6978895Z outputs = block( 2025-08-14T21:47:12.6979212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6979567Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6979945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6980316Z return func(*args, **kwargs) 2025-08-14T21:47:12.6980685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.6981088Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.6981552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.6981948Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.6982336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6982749Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6982942Z 2025-08-14T21:47:12.6983048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6983405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6983723Z return mod(**inputs) 2025-08-14T21:47:12.6984095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6984488Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6984912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6985296Z outputs = block( 2025-08-14T21:47:12.6985636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6986021Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6986390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6986756Z return func(*args, **kwargs) 2025-08-14T21:47:12.6987119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.6987523Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.6987920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.6988306Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.6988681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.6989154Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.6989398Z 2025-08-14T21:47:12.6989507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6989885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6990226Z return mod(**inputs) 2025-08-14T21:47:12.6990597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6991012Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6991420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6991806Z outputs = block( 2025-08-14T21:47:12.6992139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6992536Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6992934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.6993333Z return func(*args, **kwargs) 2025-08-14T21:47:12.6993721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.6994215Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.6994646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.6995068Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.6995455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.6995965Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.6996154Z 2025-08-14T21:47:12.6996270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.6996668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.6997015Z return mod(**inputs) 2025-08-14T21:47:12.6997391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.6997810Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.6998222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.6998615Z outputs = block( 2025-08-14T21:47:12.6998976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.6999354Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.6999772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7000166Z return func(*args, **kwargs) 2025-08-14T21:47:12.7000546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:47:12.7001060Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:47:12.7001236Z 2025-08-14T21:47:12.7001346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7001725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7002059Z return mod(**inputs) 2025-08-14T21:47:12.7002440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7002852Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7003268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7003629Z outputs = block( 2025-08-14T21:47:12.7003949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7004305Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7004673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7005045Z return func(*args, **kwargs) 2025-08-14T21:47:12.7005409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7005803Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7006180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7006544Z return func(*args, **kwargs) 2025-08-14T21:47:12.7006909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.7007421Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.7007880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7008275Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7008441Z 2025-08-14T21:47:12.7008531Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7008929Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7009148Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7009361Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7009596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7009964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7010294Z return mod(**inputs) 2025-08-14T21:47:12.7010663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7011096Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7011483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7011851Z outputs = block( 2025-08-14T21:47:12.7012165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7012524Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7012930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7013307Z return func(*args, **kwargs) 2025-08-14T21:47:12.7013698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7014096Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7014483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7014852Z return func(*args, **kwargs) 2025-08-14T21:47:12.7015212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7015615Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7016056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.7016531Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.7016722Z 2025-08-14T21:47:12.7016825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7017187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7017515Z return mod(**inputs) 2025-08-14T21:47:12.7017868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7018264Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7018650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7019016Z outputs = block( 2025-08-14T21:47:12.7019330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7019694Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7020073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7020433Z return func(*args, **kwargs) 2025-08-14T21:47:12.7020802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7021224Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7021607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7021966Z return func(*args, **kwargs) 2025-08-14T21:47:12.7022325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7022724Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7023155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.7023613Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.7023780Z 2025-08-14T21:47:12.7023883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7024242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7024556Z return mod(**inputs) 2025-08-14T21:47:12.7024917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7025333Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7025717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7026080Z outputs = block( 2025-08-14T21:47:12.7026400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7026760Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7027148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7027526Z return func(*args, **kwargs) 2025-08-14T21:47:12.7027907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7028308Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7028691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7029061Z return func(*args, **kwargs) 2025-08-14T21:47:12.7029431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.7029815Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.7030180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7030586Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7030770Z 2025-08-14T21:47:12.7030887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7031275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7031633Z return mod(**inputs) 2025-08-14T21:47:12.7032026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7032450Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7032851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7033247Z outputs = block( 2025-08-14T21:47:12.7033593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7033973Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7034376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7034767Z return func(*args, **kwargs) 2025-08-14T21:47:12.7035158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7035603Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7036105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.7036540Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.7036914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7037336Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7037529Z 2025-08-14T21:47:12.7037641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7038029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7038370Z return mod(**inputs) 2025-08-14T21:47:12.7038749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7039172Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7039601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7039989Z outputs = block( 2025-08-14T21:47:12.7040327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7040713Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7041103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7041502Z return func(*args, **kwargs) 2025-08-14T21:47:12.7041908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7042334Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7042766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.7043179Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.7043552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.7044015Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.7044255Z 2025-08-14T21:47:12.7044360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7044718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7045042Z return mod(**inputs) 2025-08-14T21:47:12.7045390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7045781Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7046168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7046534Z outputs = block( 2025-08-14T21:47:12.7046849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7047207Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7047583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7047946Z return func(*args, **kwargs) 2025-08-14T21:47:12.7048311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7048718Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7049123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.7049508Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.7049890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7050314Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7050495Z 2025-08-14T21:47:12.7050611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7050983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7051325Z return mod(**inputs) 2025-08-14T21:47:12.7051699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7052104Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7052524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7052890Z outputs = block( 2025-08-14T21:47:12.7053212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7053583Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7053958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7054332Z return func(*args, **kwargs) 2025-08-14T21:47:12.7054690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7055083Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7055487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7055854Z return func(*args, **kwargs) 2025-08-14T21:47:12.7056243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.7056729Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.7057189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7057586Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7057759Z 2025-08-14T21:47:12.7057845Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7058066Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7058280Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7058481Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7058715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7059080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7059395Z return mod(**inputs) 2025-08-14T21:47:12.7059767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7060154Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7060533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7060940Z outputs = block( 2025-08-14T21:47:12.7061264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7061628Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7062000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7062372Z return func(*args, **kwargs) 2025-08-14T21:47:12.7062743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7063141Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7063513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7063897Z return func(*args, **kwargs) 2025-08-14T21:47:12.7064254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7064640Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7065065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.7065531Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.7065707Z 2025-08-14T21:47:12.7065820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7066163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7066481Z return mod(**inputs) 2025-08-14T21:47:12.7066828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7067234Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7067606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7067968Z outputs = block( 2025-08-14T21:47:12.7068283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7068635Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7069016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7069404Z return func(*args, **kwargs) 2025-08-14T21:47:12.7069768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7070192Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7070583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7070966Z return func(*args, **kwargs) 2025-08-14T21:47:12.7071356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7071773Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7072246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.7072703Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.7072864Z 2025-08-14T21:47:12.7072969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7073330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7073657Z return mod(**inputs) 2025-08-14T21:47:12.7074037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7074451Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7074861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7075251Z outputs = block( 2025-08-14T21:47:12.7075584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7076046Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7076458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7076848Z return func(*args, **kwargs) 2025-08-14T21:47:12.7077235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7077665Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7078104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7078494Z return func(*args, **kwargs) 2025-08-14T21:47:12.7078873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.7079289Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.7079670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7080087Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7080280Z 2025-08-14T21:47:12.7080394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7080778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7081128Z return mod(**inputs) 2025-08-14T21:47:12.7081506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7081955Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7082362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7082750Z outputs = block( 2025-08-14T21:47:12.7083088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7083474Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7083886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7084279Z return func(*args, **kwargs) 2025-08-14T21:47:12.7084721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7085163Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7085569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.7085949Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.7086304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7086700Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7086872Z 2025-08-14T21:47:12.7086978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7087341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7087662Z return mod(**inputs) 2025-08-14T21:47:12.7088018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7088407Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7088796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7089169Z outputs = block( 2025-08-14T21:47:12.7089479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7089839Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7090217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7090588Z return func(*args, **kwargs) 2025-08-14T21:47:12.7090952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7091358Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7091766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.7092164Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.7092504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.7092946Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.7093173Z 2025-08-14T21:47:12.7093285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7093634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7093954Z return mod(**inputs) 2025-08-14T21:47:12.7094309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7094699Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7095077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7095447Z outputs = block( 2025-08-14T21:47:12.7095763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7096130Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7096508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7096875Z return func(*args, **kwargs) 2025-08-14T21:47:12.7097240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7097640Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7098063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.7098456Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.7098833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7099229Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7099405Z 2025-08-14T21:47:12.7099713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7100073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7100405Z return mod(**inputs) 2025-08-14T21:47:12.7100781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7101198Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7101620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7101982Z outputs = block( 2025-08-14T21:47:12.7102314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7102704Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7103102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7103485Z return func(*args, **kwargs) 2025-08-14T21:47:12.7103871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:47:12.7104307Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:47:12.7104475Z 2025-08-14T21:47:12.7104582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7104964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7105302Z return mod(**inputs) 2025-08-14T21:47:12.7105682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7106092Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7106523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7106916Z outputs = block( 2025-08-14T21:47:12.7107247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7107625Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7108027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7108420Z return func(*args, **kwargs) 2025-08-14T21:47:12.7108944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7109368Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7109780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7110174Z return func(*args, **kwargs) 2025-08-14T21:47:12.7110608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.7111124Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.7111605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7112018Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7112205Z 2025-08-14T21:47:12.7112292Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7112564Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7112791Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7113005Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7113278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7113660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7113994Z return mod(**inputs) 2025-08-14T21:47:12.7114375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7114789Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7115200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7115581Z outputs = block( 2025-08-14T21:47:12.7115989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7116372Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7116766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7117158Z return func(*args, **kwargs) 2025-08-14T21:47:12.7117547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7117966Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7118373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7118761Z return func(*args, **kwargs) 2025-08-14T21:47:12.7119150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7119622Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7120069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.7120577Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.7120779Z 2025-08-14T21:47:12.7120893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7121280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7121604Z return mod(**inputs) 2025-08-14T21:47:12.7121960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7122362Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7122739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7123109Z outputs = block( 2025-08-14T21:47:12.7123438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7123812Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7124205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7124578Z return func(*args, **kwargs) 2025-08-14T21:47:12.7124945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7125354Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7125737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7126108Z return func(*args, **kwargs) 2025-08-14T21:47:12.7126463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7126870Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7127328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.7127780Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.7127957Z 2025-08-14T21:47:12.7128063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7128424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7128748Z return mod(**inputs) 2025-08-14T21:47:12.7129103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7129487Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7129872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7130235Z outputs = block( 2025-08-14T21:47:12.7130550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7130909Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7131287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7131654Z return func(*args, **kwargs) 2025-08-14T21:47:12.7132013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7132404Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7132813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7133194Z return func(*args, **kwargs) 2025-08-14T21:47:12.7133583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.7133993Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.7134375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7134789Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7134979Z 2025-08-14T21:47:12.7135116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7135501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7135849Z return mod(**inputs) 2025-08-14T21:47:12.7136201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7136617Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7137031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7137417Z outputs = block( 2025-08-14T21:47:12.7137758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7138138Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7138541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7138928Z return func(*args, **kwargs) 2025-08-14T21:47:12.7139337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7139770Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7140192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.7140610Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.7140986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7141435Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7141619Z 2025-08-14T21:47:12.7141729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7142129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7142477Z return mod(**inputs) 2025-08-14T21:47:12.7142856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7143266Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7143675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7144064Z outputs = block( 2025-08-14T21:47:12.7144392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7144781Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7145182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7145581Z return func(*args, **kwargs) 2025-08-14T21:47:12.7145960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7146393Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7146825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.7147235Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.7147607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.7148090Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.7148331Z 2025-08-14T21:47:12.7148454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7148821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7149145Z return mod(**inputs) 2025-08-14T21:47:12.7149503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7149920Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7150306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7150682Z outputs = block( 2025-08-14T21:47:12.7151009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7151369Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7151754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7152134Z return func(*args, **kwargs) 2025-08-14T21:47:12.7152507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7152915Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7153329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.7153741Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.7154099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7154501Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7154677Z 2025-08-14T21:47:12.7154781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7155146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7155479Z return mod(**inputs) 2025-08-14T21:47:12.7155954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7156387Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7156826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7157227Z outputs = block( 2025-08-14T21:47:12.7157578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7157968Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7158337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7158707Z return func(*args, **kwargs) 2025-08-14T21:47:12.7159075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7159468Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7159847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7160217Z return func(*args, **kwargs) 2025-08-14T21:47:12.7160585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.7161071Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.7161532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7161929Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7162099Z 2025-08-14T21:47:12.7162192Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7162406Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7162617Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7162823Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7163049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7163408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7163755Z return mod(**inputs) 2025-08-14T21:47:12.7164123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7164503Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7164882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7165243Z outputs = block( 2025-08-14T21:47:12.7165549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7165901Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7166271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7166637Z return func(*args, **kwargs) 2025-08-14T21:47:12.7167002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7167398Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7167818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7168199Z return func(*args, **kwargs) 2025-08-14T21:47:12.7168544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7168940Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7169369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.7169843Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.7170029Z 2025-08-14T21:47:12.7170130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7170493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7170812Z return mod(**inputs) 2025-08-14T21:47:12.7171158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7171552Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7171928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7172286Z outputs = block( 2025-08-14T21:47:12.7172592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7172943Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7173316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7173682Z return func(*args, **kwargs) 2025-08-14T21:47:12.7174044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7174428Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7174804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7175160Z return func(*args, **kwargs) 2025-08-14T21:47:12.7175515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7175907Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7176335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.7176788Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.7176956Z 2025-08-14T21:47:12.7177065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7177435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7177772Z return mod(**inputs) 2025-08-14T21:47:12.7178136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7178547Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7178933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7179287Z outputs = block( 2025-08-14T21:47:12.7179605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7179964Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7180326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7180691Z return func(*args, **kwargs) 2025-08-14T21:47:12.7181045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7181447Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7181820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7182184Z return func(*args, **kwargs) 2025-08-14T21:47:12.7182546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.7182922Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.7183288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7183684Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7183853Z 2025-08-14T21:47:12.7183977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7184322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7184638Z return mod(**inputs) 2025-08-14T21:47:12.7184986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7185370Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7185736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7186094Z outputs = block( 2025-08-14T21:47:12.7186406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7186748Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7187118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7187490Z return func(*args, **kwargs) 2025-08-14T21:47:12.7187861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7188261Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7188671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.7189052Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.7189391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7189785Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7189959Z 2025-08-14T21:47:12.7190064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7190426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7190744Z return mod(**inputs) 2025-08-14T21:47:12.7191102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7191518Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7191911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7192301Z outputs = block( 2025-08-14T21:47:12.7192641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7193030Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7193428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7193826Z return func(*args, **kwargs) 2025-08-14T21:47:12.7194222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7194666Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7195101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.7195535Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.7196011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.7196512Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.7196773Z 2025-08-14T21:47:12.7196890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7197326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7197696Z return mod(**inputs) 2025-08-14T21:47:12.7198107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7198556Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7198988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7199399Z outputs = block( 2025-08-14T21:47:12.7199743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7200132Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7200546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7200949Z return func(*args, **kwargs) 2025-08-14T21:47:12.7201347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7201824Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7202270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.7202699Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.7203102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7203540Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7203724Z 2025-08-14T21:47:12.7203843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7204228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7204582Z return mod(**inputs) 2025-08-14T21:47:12.7204975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7205399Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7205786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7206172Z outputs = block( 2025-08-14T21:47:12.7206494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7206843Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7207221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7207592Z return func(*args, **kwargs) 2025-08-14T21:47:12.7207961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:47:12.7208405Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:47:12.7208587Z 2025-08-14T21:47:12.7208860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7209237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7209558Z return mod(**inputs) 2025-08-14T21:47:12.7209928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7210369Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7210746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7211110Z outputs = block( 2025-08-14T21:47:12.7211432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7211790Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7212190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7212562Z return func(*args, **kwargs) 2025-08-14T21:47:12.7212949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7213346Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7213728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7214100Z return func(*args, **kwargs) 2025-08-14T21:47:12.7214469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.7214950Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.7215415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7215812Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7215983Z 2025-08-14T21:47:12.7216072Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7216281Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7216492Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7216701Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7216929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7217293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7217621Z return mod(**inputs) 2025-08-14T21:47:12.7217979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7218360Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7218745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7219118Z outputs = block( 2025-08-14T21:47:12.7219426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7219778Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7220150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7220553Z return func(*args, **kwargs) 2025-08-14T21:47:12.7220915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7221310Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7221698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7222058Z return func(*args, **kwargs) 2025-08-14T21:47:12.7222437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7222827Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7223265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.7223727Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.7223932Z 2025-08-14T21:47:12.7224036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7224392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7224711Z return mod(**inputs) 2025-08-14T21:47:12.7225054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7225439Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7225830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7226178Z outputs = block( 2025-08-14T21:47:12.7226489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7226850Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7227231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7227303Z return func(*args, **kwargs) 2025-08-14T21:47:12.7227549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7227644Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7227884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7227952Z return func(*args, **kwargs) 2025-08-14T21:47:12.7228207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7228303Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7228599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.7228710Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.7228716Z 2025-08-14T21:47:12.7228822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7229037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7229102Z return mod(**inputs) 2025-08-14T21:47:12.7229352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7229434Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7229675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7229745Z outputs = block( 2025-08-14T21:47:12.7229965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7230044Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7230304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7230375Z return func(*args, **kwargs) 2025-08-14T21:47:12.7230624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7230712Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7230950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7231026Z return func(*args, **kwargs) 2025-08-14T21:47:12.7231276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.7231355Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.7231570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7231688Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7231709Z 2025-08-14T21:47:12.7231821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7232014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7232078Z return mod(**inputs) 2025-08-14T21:47:12.7232326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7232408Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7232674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7232737Z outputs = block( 2025-08-14T21:47:12.7232971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7233059Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7233294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7233369Z return func(*args, **kwargs) 2025-08-14T21:47:12.7233622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7233728Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7233980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.7234064Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.7234284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7234411Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7234416Z 2025-08-14T21:47:12.7234522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7234735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7234806Z return mod(**inputs) 2025-08-14T21:47:12.7235080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7235176Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7235448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7235517Z outputs = block( 2025-08-14T21:47:12.7235826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7235912Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7236204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7236299Z return func(*args, **kwargs) 2025-08-14T21:47:12.7236572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7236694Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7236953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.7237032Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.7237249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.7237429Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.7237433Z 2025-08-14T21:47:12.7237545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7237748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7237816Z return mod(**inputs) 2025-08-14T21:47:12.7238090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7238174Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7238424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7238488Z outputs = block( 2025-08-14T21:47:12.7238708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7238797Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7239049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7239120Z return func(*args, **kwargs) 2025-08-14T21:47:12.7239414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7239518Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7239768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.7239856Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.7240070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7240197Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7240201Z 2025-08-14T21:47:12.7240302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7240505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7240571Z return mod(**inputs) 2025-08-14T21:47:12.7240844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7240940Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7241194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7241261Z outputs = block( 2025-08-14T21:47:12.7241506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7241582Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7241832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7241902Z return func(*args, **kwargs) 2025-08-14T21:47:12.7242143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7242238Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7242474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7242561Z return func(*args, **kwargs) 2025-08-14T21:47:12.7242809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.7242991Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.7243211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7243327Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7243330Z 2025-08-14T21:47:12.7243412Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7243500Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7243577Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7243660Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7243762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7243960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7244050Z return mod(**inputs) 2025-08-14T21:47:12.7244299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7244382Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7244638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7244701Z outputs = block( 2025-08-14T21:47:12.7244993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7245074Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7245341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7245420Z return func(*args, **kwargs) 2025-08-14T21:47:12.7245661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7245750Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7245995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7246062Z return func(*args, **kwargs) 2025-08-14T21:47:12.7246313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7246411Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7246702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.7246841Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.7246846Z 2025-08-14T21:47:12.7246948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7247150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7247215Z return mod(**inputs) 2025-08-14T21:47:12.7247464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7247555Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7247797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7247859Z outputs = block( 2025-08-14T21:47:12.7248084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7248161Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7248406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7248490Z return func(*args, **kwargs) 2025-08-14T21:47:12.7248733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7248827Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7249062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7249129Z return func(*args, **kwargs) 2025-08-14T21:47:12.7249375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7249471Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7249762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.7249872Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.7249877Z 2025-08-14T21:47:12.7249979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7250198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7250263Z return mod(**inputs) 2025-08-14T21:47:12.7250518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7250600Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7250840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7250909Z outputs = block( 2025-08-14T21:47:12.7251143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7251223Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7251489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7251563Z return func(*args, **kwargs) 2025-08-14T21:47:12.7251817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7251909Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7252157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7252235Z return func(*args, **kwargs) 2025-08-14T21:47:12.7252491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.7252585Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.7252811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7252938Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7252943Z 2025-08-14T21:47:12.7253069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7253266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7253331Z return mod(**inputs) 2025-08-14T21:47:12.7253585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7253677Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7253921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7253983Z outputs = block( 2025-08-14T21:47:12.7254197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7254283Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7254517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7254603Z return func(*args, **kwargs) 2025-08-14T21:47:12.7254850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7254953Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7255204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.7255285Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.7255503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7255627Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7255631Z 2025-08-14T21:47:12.7255733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7255940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7256006Z return mod(**inputs) 2025-08-14T21:47:12.7256275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7256365Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7256606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7256667Z outputs = block( 2025-08-14T21:47:12.7256892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7256985Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7257230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7257313Z return func(*args, **kwargs) 2025-08-14T21:47:12.7257570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7257688Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7257944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.7258026Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.7258257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.7258445Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.7258449Z 2025-08-14T21:47:12.7258565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7258772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7258842Z return mod(**inputs) 2025-08-14T21:47:12.7259111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7259200Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7259463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7259527Z outputs = block( 2025-08-14T21:47:12.7259744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7259828Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7260064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7260133Z return func(*args, **kwargs) 2025-08-14T21:47:12.7260385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7260508Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7260768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.7260861Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.7261086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7261216Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7261220Z 2025-08-14T21:47:12.7261326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7261540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7261610Z return mod(**inputs) 2025-08-14T21:47:12.7261868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7261967Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7262221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7262303Z outputs = block( 2025-08-14T21:47:12.7262541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7262620Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7262863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7262931Z return func(*args, **kwargs) 2025-08-14T21:47:12.7263233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:47:12.7263354Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:47:12.7263358Z 2025-08-14T21:47:12.7263478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7263692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7263763Z return mod(**inputs) 2025-08-14T21:47:12.7264020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7264118Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7264468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7264545Z outputs = block( 2025-08-14T21:47:12.7264783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7264868Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7265123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7265195Z return func(*args, **kwargs) 2025-08-14T21:47:12.7265438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7265536Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7265771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7265840Z return func(*args, **kwargs) 2025-08-14T21:47:12.7266085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.7266272Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.7266507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7266631Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7266636Z 2025-08-14T21:47:12.7266722Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7266839Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7266924Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7267012Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7267121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7267333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7267408Z return mod(**inputs) 2025-08-14T21:47:12.7267675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7267762Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7268031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7268097Z outputs = block( 2025-08-14T21:47:12.7268342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7268427Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7268691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7268770Z return func(*args, **kwargs) 2025-08-14T21:47:12.7269026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7269118Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7269374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7269475Z return func(*args, **kwargs) 2025-08-14T21:47:12.7269737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7269855Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7270161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.7270307Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.7270311Z 2025-08-14T21:47:12.7270417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7270630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7270700Z return mod(**inputs) 2025-08-14T21:47:12.7270963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7271060Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7271313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7271378Z outputs = block( 2025-08-14T21:47:12.7271613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7271697Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7271949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7272021Z return func(*args, **kwargs) 2025-08-14T21:47:12.7272275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7272374Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7272626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7272697Z return func(*args, **kwargs) 2025-08-14T21:47:12.7272957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7273058Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7273386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.7273504Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.7273507Z 2025-08-14T21:47:12.7273614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7273833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7273904Z return mod(**inputs) 2025-08-14T21:47:12.7274180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7274273Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7274537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7274615Z outputs = block( 2025-08-14T21:47:12.7274853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7274955Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7275225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7275302Z return func(*args, **kwargs) 2025-08-14T21:47:12.7275581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7275734Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7276024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7276109Z return func(*args, **kwargs) 2025-08-14T21:47:12.7276385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.7276478Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.7276714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7276843Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7276847Z 2025-08-14T21:47:12.7276963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7277179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7277257Z return mod(**inputs) 2025-08-14T21:47:12.7277523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7277611Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7277874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7277942Z outputs = block( 2025-08-14T21:47:12.7278170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7278263Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7278513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7278585Z return func(*args, **kwargs) 2025-08-14T21:47:12.7278850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7278960Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7279224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.7279309Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.7279534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7279683Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7279689Z 2025-08-14T21:47:12.7279796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7280009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7280077Z return mod(**inputs) 2025-08-14T21:47:12.7280337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7280430Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7280688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7280752Z outputs = block( 2025-08-14T21:47:12.7280990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7281073Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7281328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7281418Z return func(*args, **kwargs) 2025-08-14T21:47:12.7281675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7281789Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7282044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.7282128Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.7282373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.7282574Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.7282578Z 2025-08-14T21:47:12.7282696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7282903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7282973Z return mod(**inputs) 2025-08-14T21:47:12.7283240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7283325Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7283588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7283654Z outputs = block( 2025-08-14T21:47:12.7283884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7283984Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7284221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7284292Z return func(*args, **kwargs) 2025-08-14T21:47:12.7284541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7284643Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7284902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.7284993Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.7285221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7285353Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7285357Z 2025-08-14T21:47:12.7285463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7285681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7285767Z return mod(**inputs) 2025-08-14T21:47:12.7286029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7286124Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7286380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7286444Z outputs = block( 2025-08-14T21:47:12.7286682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7286763Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7287018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7287089Z return func(*args, **kwargs) 2025-08-14T21:47:12.7287343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7287443Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7287708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7287782Z return func(*args, **kwargs) 2025-08-14T21:47:12.7288046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.7288242Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.7288492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7288616Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7288620Z 2025-08-14T21:47:12.7288704Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7288813Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7288899Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7288987Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7289099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7289309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7289382Z return mod(**inputs) 2025-08-14T21:47:12.7289643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7289727Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7289993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7290058Z outputs = block( 2025-08-14T21:47:12.7290295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7290379Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7290632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7290714Z return func(*args, **kwargs) 2025-08-14T21:47:12.7290972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7291064Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7291324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7291395Z return func(*args, **kwargs) 2025-08-14T21:47:12.7291660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7291759Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7292065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.7292228Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.7292233Z 2025-08-14T21:47:12.7292342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7292561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7292628Z return mod(**inputs) 2025-08-14T21:47:12.7292887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7292979Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7293237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7293303Z outputs = block( 2025-08-14T21:47:12.7293542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7293627Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7293883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7293985Z return func(*args, **kwargs) 2025-08-14T21:47:12.7294245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7294343Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7294596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7294667Z return func(*args, **kwargs) 2025-08-14T21:47:12.7294951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7295052Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7295376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.7295503Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.7295506Z 2025-08-14T21:47:12.7295621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7295852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7295926Z return mod(**inputs) 2025-08-14T21:47:12.7296204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7296298Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7296566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7296644Z outputs = block( 2025-08-14T21:47:12.7296893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7296982Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7297245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7297320Z return func(*args, **kwargs) 2025-08-14T21:47:12.7297592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7297687Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7297947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7298034Z return func(*args, **kwargs) 2025-08-14T21:47:12.7298293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.7298384Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.7298621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7298766Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7298769Z 2025-08-14T21:47:12.7298886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7299105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7299175Z return mod(**inputs) 2025-08-14T21:47:12.7299455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7299542Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7299838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7299904Z outputs = block( 2025-08-14T21:47:12.7300143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7300237Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7300501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7300572Z return func(*args, **kwargs) 2025-08-14T21:47:12.7300844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7300953Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7301223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.7301327Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.7301556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7301702Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7301708Z 2025-08-14T21:47:12.7301815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7302033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7302099Z return mod(**inputs) 2025-08-14T21:47:12.7302367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7302467Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7302729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7302794Z outputs = block( 2025-08-14T21:47:12.7303043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7303126Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7303391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7303466Z return func(*args, **kwargs) 2025-08-14T21:47:12.7303730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7303845Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7304107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.7304192Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.7304429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.7304617Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.7304621Z 2025-08-14T21:47:12.7304738Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7304963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7305033Z return mod(**inputs) 2025-08-14T21:47:12.7305312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7305397Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7305665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7305729Z outputs = block( 2025-08-14T21:47:12.7305971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7306064Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7306320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7306395Z return func(*args, **kwargs) 2025-08-14T21:47:12.7306680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7306808Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7307076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.7307169Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.7307401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7307536Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7307540Z 2025-08-14T21:47:12.7307667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7307894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7307981Z return mod(**inputs) 2025-08-14T21:47:12.7308246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7308344Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7308599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7308844Z outputs = block( 2025-08-14T21:47:12.7309121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7309203Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7309472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7309548Z return func(*args, **kwargs) 2025-08-14T21:47:12.7309807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:47:12.7309928Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:47:12.7309934Z 2025-08-14T21:47:12.7310041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7310253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7310327Z return mod(**inputs) 2025-08-14T21:47:12.7310593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7310688Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7310947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7311015Z outputs = block( 2025-08-14T21:47:12.7311257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7311337Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7311600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7311724Z return func(*args, **kwargs) 2025-08-14T21:47:12.7311982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7312081Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7312333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7312404Z return func(*args, **kwargs) 2025-08-14T21:47:12.7312675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:47:12.7312871Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:47:12.7313109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7313234Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7313261Z 2025-08-14T21:47:12.7313350Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7313444Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7313529Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7313615Z cudagraph partition due to non gpu ops 2025-08-14T21:47:12.7313735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7313949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7314030Z return mod(**inputs) 2025-08-14T21:47:12.7314324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7314413Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7314703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7314774Z outputs = block( 2025-08-14T21:47:12.7315006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7315097Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7315349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7315429Z return func(*args, **kwargs) 2025-08-14T21:47:12.7315744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7315843Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7316117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7316191Z return func(*args, **kwargs) 2025-08-14T21:47:12.7316462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7316570Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7316883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:12.7317032Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:12.7317036Z 2025-08-14T21:47:12.7317146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7317374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7317449Z return mod(**inputs) 2025-08-14T21:47:12.7317712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7317805Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7318063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7318163Z outputs = block( 2025-08-14T21:47:12.7318405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7318488Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7318744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7318815Z return func(*args, **kwargs) 2025-08-14T21:47:12.7319068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7319168Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7319419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7319493Z return func(*args, **kwargs) 2025-08-14T21:47:12.7319756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:47:12.7319866Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:12.7320149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:12.7320254Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:12.7320258Z 2025-08-14T21:47:12.7320354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7320549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7320629Z return mod(**inputs) 2025-08-14T21:47:12.7320870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7320972Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7321214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7321286Z outputs = block( 2025-08-14T21:47:12.7321497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7321575Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7321814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7321882Z return func(*args, **kwargs) 2025-08-14T21:47:12.7322125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:47:12.7322209Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:47:12.7322441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7322517Z return func(*args, **kwargs) 2025-08-14T21:47:12.7322753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:47:12.7322834Z attn_output = self.c_proj(attn_output) 2025-08-14T21:47:12.7323047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7323160Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7323163Z 2025-08-14T21:47:12.7323267Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7323456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7323522Z return mod(**inputs) 2025-08-14T21:47:12.7323767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7323848Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7324108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7324170Z outputs = block( 2025-08-14T21:47:12.7324380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7324462Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7324694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7324759Z return func(*args, **kwargs) 2025-08-14T21:47:12.7325004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7325103Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7325345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:47:12.7325424Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:47:12.7325634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7325769Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7325772Z 2025-08-14T21:47:12.7325868Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7326059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7326128Z return mod(**inputs) 2025-08-14T21:47:12.7326363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7327238Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7327494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7327575Z outputs = block( 2025-08-14T21:47:12.7327799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7327876Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7328113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7328178Z return func(*args, **kwargs) 2025-08-14T21:47:12.7328411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7328518Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7328754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:47:12.7328831Z hidden_states = self.act(hidden_states) 2025-08-14T21:47:12.7329043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:12.7329215Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:12.7329220Z 2025-08-14T21:47:12.7329327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7329516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7329580Z return mod(**inputs) 2025-08-14T21:47:12.7329827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:47:12.7329906Z transformer_outputs = self.transformer( 2025-08-14T21:47:12.7330147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:47:12.7330209Z outputs = block( 2025-08-14T21:47:12.7330418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:12.7330519Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:12.7330750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:12.7330821Z return func(*args, **kwargs) 2025-08-14T21:47:12.7331065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:47:12.7331166Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:47:12.7331412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:47:12.7331499Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:47:12.7331713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:47:12.7331836Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:47:12.7331841Z 2025-08-14T21:47:12.7331942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7332138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7332234Z return mod(**inputs) 2025-08-14T21:47:12.7332480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1494, in forward 2025-08-14T21:47:12.7332563Z logits = self.score(hidden_states) 2025-08-14T21:47:12.7332567Z 2025-08-14T21:47:12.7332666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7332860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7332951Z return mod(**inputs) 2025-08-14T21:47:12.7333204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1537, in forward 2025-08-14T21:47:12.7333366Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:47:12.7333371Z 2025-08-14T21:47:12.7333469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:12.7333659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:12.7333729Z return mod(**inputs) 2025-08-14T21:47:12.7333967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1537, in forward 2025-08-14T21:47:12.7334104Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:47:12.7334114Z 2025-08-14T21:47:24.1156621Z Compilation time (from dynamo_timed): 16.874796777 2025-08-14T21:47:24.1156927Z pass 2025-08-14T21:47:24.1157272Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:24.1165525Z TIMING: _recursive_pre_grad_passes:0.01405 _recursive_joint_graph_passes:0.58851 _recursive_post_grad_passes:0.08389 async_compile.wait:0.71351 code_gen:8.37908 inductor_compile:9.52965 backend_compile:12.74098 gc:0.00082 entire_frame_compile:16.8748 total_wall_time:16.8748 2025-08-14T21:47:24.1167319Z STATS: call_* op count: 1138 | FakeTensorMode.__torch_dispatch__:12461 | FakeTensor.__torch_dispatch__:4654 | ProxyTorchDispatchMode.__torch_dispatch__:4144 2025-08-14T21:47:24.1167936Z Dynamo produced 2 graphs covering 1138 ops with 0 graph breaks (0 unique) 2025-08-14T21:47:29.3731856Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:47:29.3733051Z from pkg_resources import resource_filename 2025-08-14T21:47:29.9675123Z 2025-08-14T21:47:31.1709249Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:47:31.1709574Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:47:31.1715108Z cpu eval GoogleFnet 2025-08-14T21:47:31.6485927Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:31.8134790Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:31.9776513Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:37.5642929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5643438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5643807Z return mod(**inputs) 2025-08-14T21:47:37.5644269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5644690Z outputs = self.fnet( 2025-08-14T21:47:37.5645128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5645566Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5645981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5646719Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5647105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5647478Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5647879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5648360Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5648793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5649213Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5649690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5650149Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5650332Z 2025-08-14T21:47:37.5650442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5650826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5651164Z return mod(**inputs) 2025-08-14T21:47:37.5651515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5651894Z outputs = self.fnet( 2025-08-14T21:47:37.5652250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5652636Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5653010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5653405Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5653776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5654148Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5654525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5654927Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5655326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5655707Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5656097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5656538Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5656755Z 2025-08-14T21:47:37.5656876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5657255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5657593Z return mod(**inputs) 2025-08-14T21:47:37.5657966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5658335Z outputs = self.fnet( 2025-08-14T21:47:37.5658694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5659077Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5659456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5659851Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5660223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5660630Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5661010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5661407Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5661824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5662238Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5662687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5663122Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5663282Z 2025-08-14T21:47:37.5663389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5663785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5664143Z return mod(**inputs) 2025-08-14T21:47:37.5664528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5664962Z outputs = self.fnet( 2025-08-14T21:47:37.5665339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5665806Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5666199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5666619Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5666995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5667385Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5667786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5668207Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5668614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5669017Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5669417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5669851Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5670019Z 2025-08-14T21:47:37.5670133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5670513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5670856Z return mod(**inputs) 2025-08-14T21:47:37.5671216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5671630Z outputs = self.fnet( 2025-08-14T21:47:37.5672002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5672412Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5673066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5673501Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5673889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5674284Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5674701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5675141Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5675575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5676225Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5676641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5677086Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5677267Z 2025-08-14T21:47:37.5677382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5677774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5678162Z return mod(**inputs) 2025-08-14T21:47:37.5678529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5678946Z outputs = self.fnet( 2025-08-14T21:47:37.5679328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5679730Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5680135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5680558Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5680956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5681346Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5681758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5682173Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5682583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5682991Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5683405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5683859Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5684032Z 2025-08-14T21:47:37.5684159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5684545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5684894Z return mod(**inputs) 2025-08-14T21:47:37.5685271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5685672Z outputs = self.fnet( 2025-08-14T21:47:37.5686028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5686414Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5686806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5687197Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5687562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5687920Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5688291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5688694Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5689095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5689480Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5689859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5690291Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5690476Z 2025-08-14T21:47:37.5690592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5690965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5691310Z return mod(**inputs) 2025-08-14T21:47:37.5691684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5692079Z outputs = self.fnet( 2025-08-14T21:47:37.5692464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5692870Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5693973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5694413Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5694799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5695179Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5695583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5695996Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5696421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5696835Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5697242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5697662Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5697834Z 2025-08-14T21:47:37.5697947Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5698328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5698665Z return mod(**inputs) 2025-08-14T21:47:37.5699035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5699427Z outputs = self.fnet( 2025-08-14T21:47:37.5699796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5700190Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5700586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5701001Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5701383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5701792Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5702294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5702759Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5703184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5703605Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5704022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5704463Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5704626Z 2025-08-14T21:47:37.5704733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5705117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5705457Z return mod(**inputs) 2025-08-14T21:47:37.5705816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5706234Z outputs = self.fnet( 2025-08-14T21:47:37.5706600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5707004Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5707391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5707803Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5708212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5708585Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5709384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5709817Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5710257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5710680Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5711102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5711558Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5711731Z 2025-08-14T21:47:37.5711853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5712242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5712596Z return mod(**inputs) 2025-08-14T21:47:37.5712985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5713386Z outputs = self.fnet( 2025-08-14T21:47:37.5713771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5714195Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5714604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5715032Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5715431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5715902Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5716313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5716752Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5717229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5717653Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5718062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5718508Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5718688Z 2025-08-14T21:47:37.5718803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5719193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5719542Z return mod(**inputs) 2025-08-14T21:47:37.5719926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5720332Z outputs = self.fnet( 2025-08-14T21:47:37.5720709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5721122Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5721581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5722012Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5722405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5722797Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5723215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5723685Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5724124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5724574Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5724996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5725435Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5725602Z 2025-08-14T21:47:37.5725710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5726089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5726415Z return mod(**inputs) 2025-08-14T21:47:37.5726771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5727157Z outputs = self.fnet( 2025-08-14T21:47:37.5727518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 512, in forward 2025-08-14T21:47:37.5727911Z embedding_output = self.embeddings( 2025-08-14T21:47:37.5728308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 142, in forward 2025-08-14T21:47:37.5728712Z embeddings = self.projection(embeddings) 2025-08-14T21:47:37.5728858Z 2025-08-14T21:47:37.5728956Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.5729201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5729573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5729912Z return mod(**inputs) 2025-08-14T21:47:37.5730270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5730655Z outputs = self.fnet( 2025-08-14T21:47:37.5731009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5731388Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5731759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5732174Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5732542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5732902Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5733295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5733703Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5734103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5734483Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5734866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5735265Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5735417Z 2025-08-14T21:47:37.5735553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5735892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5736206Z return mod(**inputs) 2025-08-14T21:47:37.5736547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5736903Z outputs = self.fnet( 2025-08-14T21:47:37.5737250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5737649Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5738028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5738433Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5738808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5739161Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5739532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5739914Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5740302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5740679Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5741046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5741443Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5741600Z 2025-08-14T21:47:37.5741705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5742057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5742366Z return mod(**inputs) 2025-08-14T21:47:37.5742712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5743077Z outputs = self.fnet( 2025-08-14T21:47:37.5743413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5743842Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5744217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5744608Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5744965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5745330Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5745721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5746114Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5746495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5746870Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5747245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5747651Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5747817Z 2025-08-14T21:47:37.5747922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5748279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5748607Z return mod(**inputs) 2025-08-14T21:47:37.5748951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5749340Z outputs = self.fnet( 2025-08-14T21:47:37.5749692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5750060Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5750437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5750857Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5751263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5751637Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5752062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5752489Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5752920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5753335Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5753753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5770682Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5770875Z 2025-08-14T21:47:37.5770991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5771389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5771734Z return mod(**inputs) 2025-08-14T21:47:37.5772132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5772518Z outputs = self.fnet( 2025-08-14T21:47:37.5772877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5773271Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5773644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5774049Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5774431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5774809Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5775209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5775633Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5776067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5776559Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5776971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5777436Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5777849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.5778229Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5778377Z 2025-08-14T21:47:37.5778485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5778849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5779173Z return mod(**inputs) 2025-08-14T21:47:37.5779517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5779884Z outputs = self.fnet( 2025-08-14T21:47:37.5780231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5780635Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5781001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5781382Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5781740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5782084Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5782487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5782871Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5783307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5783696Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5784106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5784563Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5784969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.5785387Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.5785776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.5786236Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.5786467Z 2025-08-14T21:47:37.5786576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5786946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5787281Z return mod(**inputs) 2025-08-14T21:47:37.5787641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5788010Z outputs = self.fnet( 2025-08-14T21:47:37.5788364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5788778Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5789148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5789544Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5789912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5790292Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5790665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5791056Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5791458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5791852Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5792256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.5792823Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.5793282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.5793691Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5793850Z 2025-08-14T21:47:37.5793942Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.5794227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5794615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5794952Z return mod(**inputs) 2025-08-14T21:47:37.5795331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5795824Z outputs = self.fnet( 2025-08-14T21:47:37.5796196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5796626Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5797023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5797443Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5797806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5798171Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5798551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5798957Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5799350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5799736Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5800124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5800525Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5800690Z 2025-08-14T21:47:37.5800805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5801157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5801479Z return mod(**inputs) 2025-08-14T21:47:37.5801817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5802183Z outputs = self.fnet( 2025-08-14T21:47:37.5802533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5802919Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5803313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5803728Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5804113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5804489Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5804910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5805324Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5805751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5806148Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5806531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5806941Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5807096Z 2025-08-14T21:47:37.5807201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5807562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5807893Z return mod(**inputs) 2025-08-14T21:47:37.5808241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5808616Z outputs = self.fnet( 2025-08-14T21:47:37.5809161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5809540Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5809906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5810300Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5810735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5811103Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5811494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5811899Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5812292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5812684Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5813067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5813478Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5813636Z 2025-08-14T21:47:37.5813748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5814104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5814439Z return mod(**inputs) 2025-08-14T21:47:37.5814791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5815169Z outputs = self.fnet( 2025-08-14T21:47:37.5815519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5815922Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5816348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5816762Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5817150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5817535Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5817931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5818330Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5818726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5819177Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5819581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5820012Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5820176Z 2025-08-14T21:47:37.5820287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5820666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5821009Z return mod(**inputs) 2025-08-14T21:47:37.5821384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5821773Z outputs = self.fnet( 2025-08-14T21:47:37.5822148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5822554Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5822944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5823389Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5823780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5824161Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5824557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5824974Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5825431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5825839Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5826289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5826773Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5827192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.5827575Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5827719Z 2025-08-14T21:47:37.5827823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5828187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5828501Z return mod(**inputs) 2025-08-14T21:47:37.5828838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5829203Z outputs = self.fnet( 2025-08-14T21:47:37.5829549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5829915Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5830280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5830657Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5831013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5831366Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5831747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5832139Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5832532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5832937Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5833399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5833875Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5834309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.5834750Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.5835152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.5835728Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.5835986Z 2025-08-14T21:47:37.5836099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5836492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5836840Z return mod(**inputs) 2025-08-14T21:47:37.5837193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5837579Z outputs = self.fnet( 2025-08-14T21:47:37.5837923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5838300Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5838678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5839059Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5839433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5839783Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5840172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5840555Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5840948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5841324Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5841723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.5842176Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.5842605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.5842989Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5843139Z 2025-08-14T21:47:37.5843223Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.5843472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5843831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5844162Z return mod(**inputs) 2025-08-14T21:47:37.5844509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5844879Z outputs = self.fnet( 2025-08-14T21:47:37.5845223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5845605Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5845978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5846371Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5846737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5847100Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5847502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5847907Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5848318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5848711Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5849102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5849511Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5849680Z 2025-08-14T21:47:37.5849788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5850161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5850499Z return mod(**inputs) 2025-08-14T21:47:37.5850852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5851248Z outputs = self.fnet( 2025-08-14T21:47:37.5851597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5851962Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5852336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5852728Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5853114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5853472Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5853876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5854280Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5854690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5855098Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5855504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5855933Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5856100Z 2025-08-14T21:47:37.5856210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5856584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5856912Z return mod(**inputs) 2025-08-14T21:47:37.5857266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5857632Z outputs = self.fnet( 2025-08-14T21:47:37.5857985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5858368Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5858731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5859126Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5859496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5859859Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5860234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5860644Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5861029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5861417Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5861791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5862191Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5862345Z 2025-08-14T21:47:37.5862457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5862805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5863133Z return mod(**inputs) 2025-08-14T21:47:37.5863491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5863872Z outputs = self.fnet( 2025-08-14T21:47:37.5864215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5864592Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5864974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5865380Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5865740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5866090Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5866466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5866851Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5867256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5867638Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5868020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5868419Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5868579Z 2025-08-14T21:47:37.5868680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5869030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5869341Z return mod(**inputs) 2025-08-14T21:47:37.5869679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5870038Z outputs = self.fnet( 2025-08-14T21:47:37.5870376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5870735Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5871096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5871486Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5871842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5872203Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5872583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5872971Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5873383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5873801Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5874235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5874706Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5875155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.5875568Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5875812Z 2025-08-14T21:47:37.5875935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5876310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5876655Z return mod(**inputs) 2025-08-14T21:47:37.5877030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5877416Z outputs = self.fnet( 2025-08-14T21:47:37.5877770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5878171Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5878565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5878988Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5879419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5879806Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5880222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5880646Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5881076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5881524Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5881965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5882453Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5882899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.5883343Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.5883749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.5884222Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.5884477Z 2025-08-14T21:47:37.5884591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5884974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5885316Z return mod(**inputs) 2025-08-14T21:47:37.5885687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5886089Z outputs = self.fnet( 2025-08-14T21:47:37.5886465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5886864Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5887262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5887674Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5888066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5888442Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5888854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5889237Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5889640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5890047Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5890460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.5890940Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.5891355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.5891740Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5891882Z 2025-08-14T21:47:37.5891969Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.5892221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5892583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5892937Z return mod(**inputs) 2025-08-14T21:47:37.5893323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5893733Z outputs = self.fnet( 2025-08-14T21:47:37.5894114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5894521Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5894895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5895280Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5895671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5896036Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5896417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5896842Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5897251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5897633Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5898001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5898411Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5898575Z 2025-08-14T21:47:37.5898681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5899041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5899360Z return mod(**inputs) 2025-08-14T21:47:37.5899713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5900090Z outputs = self.fnet( 2025-08-14T21:47:37.5900445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5900816Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5901184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5901575Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5901934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5902294Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5902677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5903096Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5903515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5903946Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5904357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5904792Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5904970Z 2025-08-14T21:47:37.5905082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5905444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5905788Z return mod(**inputs) 2025-08-14T21:47:37.5906154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5906552Z outputs = self.fnet( 2025-08-14T21:47:37.5906923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5907340Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5907744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5908176Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5908556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5909148Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5909551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5910055Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5910530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5910933Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5911368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5911803Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5911970Z 2025-08-14T21:47:37.5912086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5912464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5912816Z return mod(**inputs) 2025-08-14T21:47:37.5913191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5913638Z outputs = self.fnet( 2025-08-14T21:47:37.5914010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5914416Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5914816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5915224Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5915666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5916132Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5916545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5916993Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5917423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5917841Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5918250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5918678Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5918844Z 2025-08-14T21:47:37.5919001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5919373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5919683Z return mod(**inputs) 2025-08-14T21:47:37.5920027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5920384Z outputs = self.fnet( 2025-08-14T21:47:37.5920718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5921090Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5921454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5921828Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5922182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5922540Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5922954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5923347Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5923755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5924160Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5924572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5925037Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5925447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.5925840Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5925977Z 2025-08-14T21:47:37.5926086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5926428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5926755Z return mod(**inputs) 2025-08-14T21:47:37.5927110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5927475Z outputs = self.fnet( 2025-08-14T21:47:37.5927829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5928215Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5928593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5928970Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5929334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5929689Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5930056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5930437Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5930829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5931213Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5931605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5932039Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5932446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.5932889Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.5933258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.5933711Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.5933940Z 2025-08-14T21:47:37.5934053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5934414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5934736Z return mod(**inputs) 2025-08-14T21:47:37.5935083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5935456Z outputs = self.fnet( 2025-08-14T21:47:37.5935805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5936184Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5936559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5936976Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5937335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5937693Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5938076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5938458Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5938879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5939275Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5939705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.5940167Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.5940597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.5940990Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5941132Z 2025-08-14T21:47:37.5941223Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.5941473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5941852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5942197Z return mod(**inputs) 2025-08-14T21:47:37.5942570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5942977Z outputs = self.fnet( 2025-08-14T21:47:37.5943358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5943772Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5944160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5944581Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5944972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5945360Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5945776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5946222Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5946644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5947070Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5947479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5947892Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5948048Z 2025-08-14T21:47:37.5948158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5948508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5948831Z return mod(**inputs) 2025-08-14T21:47:37.5949182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5949544Z outputs = self.fnet( 2025-08-14T21:47:37.5949893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5950264Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5950636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5951051Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5951438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5951818Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5952213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5952639Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5953084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5953494Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5953917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5954345Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5954513Z 2025-08-14T21:47:37.5954631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5955007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5955342Z return mod(**inputs) 2025-08-14T21:47:37.5955792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5956202Z outputs = self.fnet( 2025-08-14T21:47:37.5956582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5957008Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5957422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5957840Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5958223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5958611Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5959020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5959443Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5959873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5960285Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5960692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5961112Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5961286Z 2025-08-14T21:47:37.5961428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5961810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5962154Z return mod(**inputs) 2025-08-14T21:47:37.5962527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5962927Z outputs = self.fnet( 2025-08-14T21:47:37.5963301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5963705Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5964104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5964518Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5964909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5965287Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5965713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5966135Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5966547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5966954Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5967332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5967755Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5967910Z 2025-08-14T21:47:37.5968015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5968380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5968688Z return mod(**inputs) 2025-08-14T21:47:37.5969022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5969366Z outputs = self.fnet( 2025-08-14T21:47:37.5969697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5970062Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5970420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5970804Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5971162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5971512Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5971879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5972258Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5972652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5973028Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5973427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5973877Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5974300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.5974684Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5974830Z 2025-08-14T21:47:37.5974935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5975298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5975651Z return mod(**inputs) 2025-08-14T21:47:37.5975999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5976376Z outputs = self.fnet( 2025-08-14T21:47:37.5976729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5977114Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5977488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5977873Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5978233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5978578Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5978983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5979391Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5979794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5980192Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5980591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.5981027Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.5981443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.5981849Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.5982246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.5982690Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.5982922Z 2025-08-14T21:47:37.5983027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5983386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5983712Z return mod(**inputs) 2025-08-14T21:47:37.5984076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5984434Z outputs = self.fnet( 2025-08-14T21:47:37.5984779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5985145Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5985501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5985892Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5986264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5986627Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5987000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.5987391Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.5987790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.5988176Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.5988584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.5989046Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.5989490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.5989870Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.5990015Z 2025-08-14T21:47:37.5990097Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.5990338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5990703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5991024Z return mod(**inputs) 2025-08-14T21:47:37.5991376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5991750Z outputs = self.fnet( 2025-08-14T21:47:37.5992094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5992492Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5992893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.5993333Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.5993715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.5994097Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.5994501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.5994923Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.5995368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.5995863Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.5996300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.5996730Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.5996917Z 2025-08-14T21:47:37.5997031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.5997432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.5997778Z return mod(**inputs) 2025-08-14T21:47:37.5998148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.5998522Z outputs = self.fnet( 2025-08-14T21:47:37.5998879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.5999251Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.5999626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6000027Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6000397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6000772Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6001178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6001612Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6002025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6002444Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6002848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6003272Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6003435Z 2025-08-14T21:47:37.6003544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6003946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6004291Z return mod(**inputs) 2025-08-14T21:47:37.6004661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6005053Z outputs = self.fnet( 2025-08-14T21:47:37.6005402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6005780Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6006146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6006551Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6006937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6007315Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6007710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6008159Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6008580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6009130Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6009520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6009979Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6010150Z 2025-08-14T21:47:37.6010270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6010680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6011011Z return mod(**inputs) 2025-08-14T21:47:37.6011367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6011747Z outputs = self.fnet( 2025-08-14T21:47:37.6012095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6012482Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6012862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6013252Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6013628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6013989Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6014379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6014778Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6015181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6015568Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6015950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6016365Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6016529Z 2025-08-14T21:47:37.6016634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6016994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6017320Z return mod(**inputs) 2025-08-14T21:47:37.6017682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6018086Z outputs = self.fnet( 2025-08-14T21:47:37.6018440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6018813Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6019188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6019578Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6019937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6020301Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6020679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6021047Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6021422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6021823Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6022205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6022634Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6023029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.6023409Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6023542Z 2025-08-14T21:47:37.6023669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6024015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6024342Z return mod(**inputs) 2025-08-14T21:47:37.6024709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6025070Z outputs = self.fnet( 2025-08-14T21:47:37.6025408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6025778Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6026141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6026517Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6026874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6027227Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6027598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6027971Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6028365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6028746Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6029145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6029583Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6029996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.6030397Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.6030764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.6031219Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.6031457Z 2025-08-14T21:47:37.6031581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6031941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6032259Z return mod(**inputs) 2025-08-14T21:47:37.6032617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6032988Z outputs = self.fnet( 2025-08-14T21:47:37.6033339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6033727Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6034127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6034539Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6034919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6035302Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6035808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6036239Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6036670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6037101Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6037542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.6038056Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.6038516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.6038947Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6039097Z 2025-08-14T21:47:37.6039191Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.6039439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6039817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6040159Z return mod(**inputs) 2025-08-14T21:47:37.6040527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6040914Z outputs = self.fnet( 2025-08-14T21:47:37.6041288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6041690Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6042077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6042495Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6042884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6043266Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6043658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6044076Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6044467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6044834Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6045208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6045603Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6045756Z 2025-08-14T21:47:37.6045869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6046236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6046568Z return mod(**inputs) 2025-08-14T21:47:37.6046921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6047308Z outputs = self.fnet( 2025-08-14T21:47:37.6047680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6048058Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6048431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6048820Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6049194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6049544Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6049915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6050321Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6050707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6051080Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6051452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6051857Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6052015Z 2025-08-14T21:47:37.6052117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6052481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6052795Z return mod(**inputs) 2025-08-14T21:47:37.6053137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6053500Z outputs = self.fnet( 2025-08-14T21:47:37.6053840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6054205Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6054581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6054961Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6055181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6055263Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6055516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6055616Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6055866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6055948Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6056188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6056295Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6056299Z 2025-08-14T21:47:37.6056402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6056604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6056670Z return mod(**inputs) 2025-08-14T21:47:37.6056912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6057021Z outputs = self.fnet( 2025-08-14T21:47:37.6057256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6057328Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6057568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6057649Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6057868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6057946Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6058183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6058286Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6058530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6058642Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6058882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6058982Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6058986Z 2025-08-14T21:47:37.6059096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6059291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6059355Z return mod(**inputs) 2025-08-14T21:47:37.6059621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6059689Z outputs = self.fnet( 2025-08-14T21:47:37.6059950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6060027Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6060274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6060365Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6060581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6060661Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6060910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6060997Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6061261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6061340Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6061613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6061741Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6061988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.6062079Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6062082Z 2025-08-14T21:47:37.6062185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6062381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6062456Z return mod(**inputs) 2025-08-14T21:47:37.6062702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6062767Z outputs = self.fnet( 2025-08-14T21:47:37.6063016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6063110Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6063365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6063451Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6063672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6063759Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6064007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6064098Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6064360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6064439Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6064728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6064862Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6065101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.6065217Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.6065427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.6065634Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.6065638Z 2025-08-14T21:47:37.6065742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6065953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6066028Z return mod(**inputs) 2025-08-14T21:47:37.6066273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6066350Z outputs = self.fnet( 2025-08-14T21:47:37.6066592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6066665Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6066917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6067001Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6067223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6067312Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6067554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6067648Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6067907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6067986Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6068273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.6068403Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.6068655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.6068739Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6068743Z 2025-08-14T21:47:37.6068826Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.6068940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6069155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6069222Z return mod(**inputs) 2025-08-14T21:47:37.6069471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6069537Z outputs = self.fnet( 2025-08-14T21:47:37.6069958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6070036Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6070285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6070380Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6070602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6070680Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6070940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6071068Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6071317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6071400Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6071642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6071754Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6071774Z 2025-08-14T21:47:37.6071879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6072083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6072165Z return mod(**inputs) 2025-08-14T21:47:37.6072420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6072497Z outputs = self.fnet( 2025-08-14T21:47:37.6072749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6072827Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6073088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6073177Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6073416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6073500Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6073754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6073866Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6074123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6074208Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6074472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6074582Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6074586Z 2025-08-14T21:47:37.6074701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6074911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6074980Z return mod(**inputs) 2025-08-14T21:47:37.6075242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6075330Z outputs = self.fnet( 2025-08-14T21:47:37.6075658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6075749Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6076017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6076116Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6076354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6076439Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6076712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6076817Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6077092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6077184Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6077472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6077595Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6077599Z 2025-08-14T21:47:37.6077701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6077907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6077976Z return mod(**inputs) 2025-08-14T21:47:37.6078250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6078335Z outputs = self.fnet( 2025-08-14T21:47:37.6078612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6078695Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6078965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6079059Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6079304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6079389Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6079653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6079769Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6080035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6080128Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6080393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6080505Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6080508Z 2025-08-14T21:47:37.6080625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6080838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6080908Z return mod(**inputs) 2025-08-14T21:47:37.6081176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6081246Z outputs = self.fnet( 2025-08-14T21:47:37.6081518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6081597Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6081861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6081983Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6082224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6082310Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6082579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6082670Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6082958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6083043Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6083340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6083474Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6083738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.6083858Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6083862Z 2025-08-14T21:47:37.6083975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6084190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6084268Z return mod(**inputs) 2025-08-14T21:47:37.6084531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6084622Z outputs = self.fnet( 2025-08-14T21:47:37.6084896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6084974Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6085259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6085356Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6085593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6085687Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6085949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6086045Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6086325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6086408Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6086715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6086840Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6087105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.6087231Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.6087461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.6087659Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.6087663Z 2025-08-14T21:47:37.6087773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6087987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6088065Z return mod(**inputs) 2025-08-14T21:47:37.6088330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6088427Z outputs = self.fnet( 2025-08-14T21:47:37.6088688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6088769Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6089036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6089129Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6089365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6089456Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6089718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6089815Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6090101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6090205Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6090500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.6090634Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.6090897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.6090982Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6090985Z 2025-08-14T21:47:37.6091091Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.6091208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6091414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6091501Z return mod(**inputs) 2025-08-14T21:47:37.6091772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6091847Z outputs = self.fnet( 2025-08-14T21:47:37.6092110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6092188Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6092444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6092543Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6092775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6092859Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6093125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6093229Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6093500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6093587Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6093846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6093963Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6093967Z 2025-08-14T21:47:37.6094075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6094292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6094362Z return mod(**inputs) 2025-08-14T21:47:37.6094619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6094697Z outputs = self.fnet( 2025-08-14T21:47:37.6094983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6095062Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6095324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6095414Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6095658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6095740Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6095997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6096107Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6096363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6096449Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6096731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6096837Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6096841Z 2025-08-14T21:47:37.6096954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6097165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6097234Z return mod(**inputs) 2025-08-14T21:47:37.6097550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6097621Z outputs = self.fnet( 2025-08-14T21:47:37.6097898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6097978Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6098240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6098338Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6098570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6098651Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6098918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6099019Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6099284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6099371Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6099637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6099756Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6099760Z 2025-08-14T21:47:37.6099868Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6100088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6100157Z return mod(**inputs) 2025-08-14T21:47:37.6100416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6100493Z outputs = self.fnet( 2025-08-14T21:47:37.6100752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6100829Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6101099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6101209Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6101453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6101540Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6101797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6101909Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6102170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6102260Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6102529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6102643Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6102648Z 2025-08-14T21:47:37.6102766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6102992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6103060Z return mod(**inputs) 2025-08-14T21:47:37.6103319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6103387Z outputs = self.fnet( 2025-08-14T21:47:37.6103648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6103725Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6104000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6104099Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6104400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6104485Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6104748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6104835Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6105114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6105194Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6105485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6105614Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6105870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.6105964Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6105968Z 2025-08-14T21:47:37.6106077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6106285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6106363Z return mod(**inputs) 2025-08-14T21:47:37.6106618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6106687Z outputs = self.fnet( 2025-08-14T21:47:37.6106948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6107026Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6107288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6107379Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6107632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6107727Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6107983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6108078Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6108352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6108433Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6108886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6109007Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6109256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.6109378Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.6109642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.6109829Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.6109833Z 2025-08-14T21:47:37.6109937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6110145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6110225Z return mod(**inputs) 2025-08-14T21:47:37.6110504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6110584Z outputs = self.fnet( 2025-08-14T21:47:37.6110866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6110948Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6111214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6111304Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6111535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6111626Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6111880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6111985Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6112258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6112340Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6112643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.6112779Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.6113041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.6113128Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6113131Z 2025-08-14T21:47:37.6113216Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.6113331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6113539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6113607Z return mod(**inputs) 2025-08-14T21:47:37.6113866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6113937Z outputs = self.fnet( 2025-08-14T21:47:37.6114228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6114308Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6114564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6114661Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6114892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6114976Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6115241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6115344Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6115668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6115763Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6116039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6116157Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6116162Z 2025-08-14T21:47:37.6116272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6116506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6116576Z return mod(**inputs) 2025-08-14T21:47:37.6116901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6116976Z outputs = self.fnet( 2025-08-14T21:47:37.6117245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6117322Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6117572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6117657Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6117884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6117962Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6118204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6118309Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6118550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6118631Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6118876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6118979Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6118983Z 2025-08-14T21:47:37.6119089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6119285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6119351Z return mod(**inputs) 2025-08-14T21:47:37.6119598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6119663Z outputs = self.fnet( 2025-08-14T21:47:37.6119912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6119985Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6120226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6120336Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6120679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6120763Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6121004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6121102Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6121352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6121433Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6121716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6121821Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6121826Z 2025-08-14T21:47:37.6121922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6122136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6122200Z return mod(**inputs) 2025-08-14T21:47:37.6122431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6122500Z outputs = self.fnet( 2025-08-14T21:47:37.6122731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6122801Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6123054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6123136Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6123371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6123448Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6123678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6123778Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6124006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6124088Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6124323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6124426Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6124430Z 2025-08-14T21:47:37.6124540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6124738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6124804Z return mod(**inputs) 2025-08-14T21:47:37.6125053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6125118Z outputs = self.fnet( 2025-08-14T21:47:37.6125365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6125438Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6125680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6125773Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6125987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6126074Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6126311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6126415Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6126682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6126758Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6127032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6127154Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6127407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.6127493Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6127497Z 2025-08-14T21:47:37.6127596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6127790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6127885Z return mod(**inputs) 2025-08-14T21:47:37.6128125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6128196Z outputs = self.fnet( 2025-08-14T21:47:37.6128428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6128499Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6128738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6128838Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6129053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6129153Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6129388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6129478Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6129725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6129800Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6130068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6130176Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6130413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.6130529Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.6130738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.6130934Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.6130939Z 2025-08-14T21:47:37.6131038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6131228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6131299Z return mod(**inputs) 2025-08-14T21:47:37.6131533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6131605Z outputs = self.fnet( 2025-08-14T21:47:37.6131841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6131911Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6132164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6132263Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6132468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6132551Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6132783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6132868Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6133113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6133189Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6133468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.6133607Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.6133867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.6133973Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6133976Z 2025-08-14T21:47:37.6134063Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.6134177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6134386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6134457Z return mod(**inputs) 2025-08-14T21:47:37.6134748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6134819Z outputs = self.fnet( 2025-08-14T21:47:37.6135098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6135176Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6135441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6135535Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6135746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6135831Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6136065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6136158Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6136399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6136478Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6136713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6136826Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6136831Z 2025-08-14T21:47:37.6136931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6137134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6137198Z return mod(**inputs) 2025-08-14T21:47:37.6137446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6137521Z outputs = self.fnet( 2025-08-14T21:47:37.6137757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6137830Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6138071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6138175Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6138398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6138474Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6138706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6138806Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6139039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6139127Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6139362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6139462Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6139467Z 2025-08-14T21:47:37.6139576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6139788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6139853Z return mod(**inputs) 2025-08-14T21:47:37.6140095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6140158Z outputs = self.fnet( 2025-08-14T21:47:37.6140400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6140470Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6140732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6140825Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6141055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6141142Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6141376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6141468Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6141711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6141791Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6142025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6142133Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6142136Z 2025-08-14T21:47:37.6142236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6142442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6142509Z return mod(**inputs) 2025-08-14T21:47:37.6142750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6142825Z outputs = self.fnet( 2025-08-14T21:47:37.6143064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6143137Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6143382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6143465Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6143691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6143770Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6144011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6144136Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6144377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6144465Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6144715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6144812Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6144815Z 2025-08-14T21:47:37.6144922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6145111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6145174Z return mod(**inputs) 2025-08-14T21:47:37.6145414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6145479Z outputs = self.fnet( 2025-08-14T21:47:37.6145739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6145810Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6146044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6146133Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6146345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6147303Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6147556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6147661Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6147922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6147999Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6148265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6148386Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6148621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.6148708Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6148712Z 2025-08-14T21:47:37.6148811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6149002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6149074Z return mod(**inputs) 2025-08-14T21:47:37.6149313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6149387Z outputs = self.fnet( 2025-08-14T21:47:37.6149625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6149698Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6149940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6150023Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6150239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6150324Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6150564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6150656Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6150931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6151009Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6151288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6151403Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6151656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.6151779Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.6152000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.6152197Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.6152202Z 2025-08-14T21:47:37.6152310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6152541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6152621Z return mod(**inputs) 2025-08-14T21:47:37.6152878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6152956Z outputs = self.fnet( 2025-08-14T21:47:37.6153213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6153291Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6153577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6153668Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6153920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6154014Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6154270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6154363Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6154631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6154710Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6155005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.6155140Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.6155403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.6155491Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6155495Z 2025-08-14T21:47:37.6155583Z cudagraph partition due to non gpu ops 2025-08-14T21:47:37.6155773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6155992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6156062Z return mod(**inputs) 2025-08-14T21:47:37.6156329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6156400Z outputs = self.fnet( 2025-08-14T21:47:37.6156675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6156754Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6157021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6157147Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6157383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6157477Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6157744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6157850Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6158125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6158211Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6158469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6158588Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6158594Z 2025-08-14T21:47:37.6158702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6161580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6161709Z return mod(**inputs) 2025-08-14T21:47:37.6161992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6162073Z outputs = self.fnet( 2025-08-14T21:47:37.6162343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6162424Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6162705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6162796Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6163049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6163144Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6163439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6163542Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6163813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6163899Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6164162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6164276Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6164280Z 2025-08-14T21:47:37.6164388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6164622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6164692Z return mod(**inputs) 2025-08-14T21:47:37.6164949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6165028Z outputs = self.fnet( 2025-08-14T21:47:37.6165297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6165374Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6165646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6165735Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6165988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6166071Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6166339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6166470Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6166728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6166821Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6167074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6167181Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6167185Z 2025-08-14T21:47:37.6167306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6167503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6167570Z return mod(**inputs) 2025-08-14T21:47:37.6167819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6167886Z outputs = self.fnet( 2025-08-14T21:47:37.6168234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6168312Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6168553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6168646Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6168866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6168951Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6169195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:47:37.6169316Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:47:37.6169579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:47:37.6169668Z self_outputs = self.self(hidden_states) 2025-08-14T21:47:37.6169925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:47:37.6170037Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:47:37.6170041Z 2025-08-14T21:47:37.6170150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6170364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6170432Z return mod(**inputs) 2025-08-14T21:47:37.6170690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6170762Z outputs = self.fnet( 2025-08-14T21:47:37.6171002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6171075Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6171333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6171414Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6171631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6171707Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6171937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6172024Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6172274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6172357Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6172650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6172766Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6173014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:47:37.6173097Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6173100Z 2025-08-14T21:47:37.6173201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6173405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6173470Z return mod(**inputs) 2025-08-14T21:47:37.6173720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6173786Z outputs = self.fnet( 2025-08-14T21:47:37.6174025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6174125Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6174412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6174511Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6174740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6174835Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6175083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6175165Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6175440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6175527Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6175799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:47:37.6175929Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:47:37.6176170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:47:37.6176286Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:37.6176499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:47:37.6176680Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:47:37.6176685Z 2025-08-14T21:47:37.6176794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6176993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6177069Z return mod(**inputs) 2025-08-14T21:47:37.6177315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:47:37.6177381Z outputs = self.fnet( 2025-08-14T21:47:37.6177629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:47:37.6177701Z encoder_outputs = self.encoder( 2025-08-14T21:47:37.6177941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:47:37.6178033Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:47:37.6178253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:37.6178340Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:37.6178581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:47:37.6178686Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:37.6178948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:37.6179024Z return forward_fn(*input_tensors) 2025-08-14T21:47:37.6179307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:47:37.6179449Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:47:37.6179705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:47:37.6179796Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6179800Z 2025-08-14T21:47:37.6179909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6180118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6180221Z return mod(**inputs) 2025-08-14T21:47:37.6180504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-08-14T21:47:37.6180617Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:47:37.6180873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-08-14T21:47:37.6180991Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:47:37.6181252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 340, in forward 2025-08-14T21:47:37.6181348Z hidden_states = self.transform(hidden_states) 2025-08-14T21:47:37.6181618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 321, in forward 2025-08-14T21:47:37.6181721Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:37.6181725Z 2025-08-14T21:47:37.6181828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6182033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6182098Z return mod(**inputs) 2025-08-14T21:47:37.6182340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-08-14T21:47:37.6182439Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:47:37.6182681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-08-14T21:47:37.6182795Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:47:37.6183043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 341, in forward 2025-08-14T21:47:37.6183137Z hidden_states = self.decoder(hidden_states) 2025-08-14T21:47:37.6183140Z 2025-08-14T21:47:37.6183253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:37.6183464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:37.6183533Z return mod(**inputs) 2025-08-14T21:47:37.6183797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 686, in forward 2025-08-14T21:47:37.6183994Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:47:37.6183998Z 2025-08-14T21:47:45.7643691Z Compilation time (from dynamo_timed): 12.497892071 2025-08-14T21:47:45.7702706Z pass 2025-08-14T21:47:45.7703258Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:45.7704204Z TIMING: _recursive_pre_grad_passes:0.00593 _recursive_joint_graph_passes:0.20919 _recursive_post_grad_passes:0.07773 async_compile.wait:0.78037 code_gen:7.74584 inductor_compile:8.90994 backend_compile:10.76074 gc:0.00018 entire_frame_compile:12.49789 total_wall_time:12.49789 2025-08-14T21:47:45.7705408Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:7521 | FakeTensor.__torch_dispatch__:3660 | ProxyTorchDispatchMode.__torch_dispatch__:2859 2025-08-14T21:47:45.7705910Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2025-08-14T21:47:51.0521530Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:47:51.0527886Z from pkg_resources import resource_filename 2025-08-14T21:47:51.6908019Z 2025-08-14T21:47:53.1061883Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:47:53.1063684Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:47:53.1075442Z cpu eval LayoutLMForMaskedLM 2025-08-14T21:47:53.6808216Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:53.9096655Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:54.1364885Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:02.6019746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6022827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6023226Z return mod(**inputs) 2025-08-14T21:48:02.6023658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6024082Z return func(*args, **kwargs) 2025-08-14T21:48:02.6024821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6025249Z return func(*args, **kwargs) 2025-08-14T21:48:02.6025632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6026011Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6028616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6029095Z outputs = self.layoutlm( 2025-08-14T21:48:02.6029508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6029913Z return func(*args, **kwargs) 2025-08-14T21:48:02.6030302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6030710Z return func(*args, **kwargs) 2025-08-14T21:48:02.6031075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6031464Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6031936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6032384Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6032786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6033180Z return func(*args, **kwargs) 2025-08-14T21:48:02.6033574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6033982Z return func(*args, **kwargs) 2025-08-14T21:48:02.6034376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6034772Z return func(*args, **kwargs) 2025-08-14T21:48:02.6035145Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6035553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6036184Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6036673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6037110Z layer_outputs = layer_module( 2025-08-14T21:48:02.6037480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6037860Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6038287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6038680Z return func(*args, **kwargs) 2025-08-14T21:48:02.6039059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6039525Z return func(*args, **kwargs) 2025-08-14T21:48:02.6039949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6040338Z return func(*args, **kwargs) 2025-08-14T21:48:02.6040741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6041200Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6041613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6042007Z return func(*args, **kwargs) 2025-08-14T21:48:02.6042374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6042796Z return func(*args, **kwargs) 2025-08-14T21:48:02.6043182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6043576Z return func(*args, **kwargs) 2025-08-14T21:48:02.6043988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6044468Z self_outputs = self.self( 2025-08-14T21:48:02.6044867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6045271Z return func(*args, **kwargs) 2025-08-14T21:48:02.6045663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6046075Z return func(*args, **kwargs) 2025-08-14T21:48:02.6046449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6046840Z return func(*args, **kwargs) 2025-08-14T21:48:02.6047274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6047787Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6048012Z 2025-08-14T21:48:02.6048132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6048528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6048877Z return mod(**inputs) 2025-08-14T21:48:02.6049251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6049643Z return func(*args, **kwargs) 2025-08-14T21:48:02.6050022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6050423Z return func(*args, **kwargs) 2025-08-14T21:48:02.6050790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6051167Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6051589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6052012Z outputs = self.layoutlm( 2025-08-14T21:48:02.6052382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6052763Z return func(*args, **kwargs) 2025-08-14T21:48:02.6053141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6053518Z return func(*args, **kwargs) 2025-08-14T21:48:02.6053870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6054243Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6054690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6055130Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6055520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6055906Z return func(*args, **kwargs) 2025-08-14T21:48:02.6056269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6056658Z return func(*args, **kwargs) 2025-08-14T21:48:02.6057031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6057413Z return func(*args, **kwargs) 2025-08-14T21:48:02.6057611Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6058004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6058379Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6058800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6059214Z layer_outputs = layer_module( 2025-08-14T21:48:02.6059580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6059963Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6060351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6060744Z return func(*args, **kwargs) 2025-08-14T21:48:02.6061119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6061504Z return func(*args, **kwargs) 2025-08-14T21:48:02.6061872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6062259Z return func(*args, **kwargs) 2025-08-14T21:48:02.6062663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6063099Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6063502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6063887Z return func(*args, **kwargs) 2025-08-14T21:48:02.6064263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6064620Z return func(*args, **kwargs) 2025-08-14T21:48:02.6064981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6065374Z return func(*args, **kwargs) 2025-08-14T21:48:02.6065797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6066659Z self_outputs = self.self( 2025-08-14T21:48:02.6067044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6067443Z return func(*args, **kwargs) 2025-08-14T21:48:02.6067820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6068221Z return func(*args, **kwargs) 2025-08-14T21:48:02.6068603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6068998Z return func(*args, **kwargs) 2025-08-14T21:48:02.6069418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6069918Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6070143Z 2025-08-14T21:48:02.6070289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6070685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6071056Z return mod(**inputs) 2025-08-14T21:48:02.6071445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6071859Z return func(*args, **kwargs) 2025-08-14T21:48:02.6072245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6072659Z return func(*args, **kwargs) 2025-08-14T21:48:02.6073055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6073451Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6073886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6074328Z outputs = self.layoutlm( 2025-08-14T21:48:02.6074717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6075111Z return func(*args, **kwargs) 2025-08-14T21:48:02.6075516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6076013Z return func(*args, **kwargs) 2025-08-14T21:48:02.6076382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6076761Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6077199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6077635Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6078033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6078431Z return func(*args, **kwargs) 2025-08-14T21:48:02.6078813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6079212Z return func(*args, **kwargs) 2025-08-14T21:48:02.6079591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6079992Z return func(*args, **kwargs) 2025-08-14T21:48:02.6080203Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6080581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6080961Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6081429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6081876Z layer_outputs = layer_module( 2025-08-14T21:48:02.6082261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6082661Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6083092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6083498Z return func(*args, **kwargs) 2025-08-14T21:48:02.6083891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6084297Z return func(*args, **kwargs) 2025-08-14T21:48:02.6084698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6085103Z return func(*args, **kwargs) 2025-08-14T21:48:02.6085589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6086057Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6086466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6086862Z return func(*args, **kwargs) 2025-08-14T21:48:02.6087247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6087652Z return func(*args, **kwargs) 2025-08-14T21:48:02.6088029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6088446Z return func(*args, **kwargs) 2025-08-14T21:48:02.6088900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6089337Z self_outputs = self.self( 2025-08-14T21:48:02.6089728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6090127Z return func(*args, **kwargs) 2025-08-14T21:48:02.6090515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6090916Z return func(*args, **kwargs) 2025-08-14T21:48:02.6091303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6091690Z return func(*args, **kwargs) 2025-08-14T21:48:02.6092110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6092614Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6092837Z 2025-08-14T21:48:02.6092925Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6093154Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6093411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6093792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6094138Z return mod(**inputs) 2025-08-14T21:48:02.6094520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6094921Z return func(*args, **kwargs) 2025-08-14T21:48:02.6095296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6095683Z return func(*args, **kwargs) 2025-08-14T21:48:02.6096038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6096428Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6096853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6097274Z outputs = self.layoutlm( 2025-08-14T21:48:02.6097646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6098035Z return func(*args, **kwargs) 2025-08-14T21:48:02.6098411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6098800Z return func(*args, **kwargs) 2025-08-14T21:48:02.6099149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6099525Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6099955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6100379Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6100809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6101199Z return func(*args, **kwargs) 2025-08-14T21:48:02.6101572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6101952Z return func(*args, **kwargs) 2025-08-14T21:48:02.6102338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6102735Z return func(*args, **kwargs) 2025-08-14T21:48:02.6102939Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6103322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6103728Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6104171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6104585Z layer_outputs = layer_module( 2025-08-14T21:48:02.6104951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6105395Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6105785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6106181Z return func(*args, **kwargs) 2025-08-14T21:48:02.6106558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6106957Z return func(*args, **kwargs) 2025-08-14T21:48:02.6107326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6107712Z return func(*args, **kwargs) 2025-08-14T21:48:02.6108118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6108548Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6109156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6109557Z return func(*args, **kwargs) 2025-08-14T21:48:02.6109936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6110327Z return func(*args, **kwargs) 2025-08-14T21:48:02.6110705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6111102Z return func(*args, **kwargs) 2025-08-14T21:48:02.6111513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6112054Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6112535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6112969Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6113119Z 2025-08-14T21:48:02.6113240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6113613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6113959Z return mod(**inputs) 2025-08-14T21:48:02.6114327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6114717Z return func(*args, **kwargs) 2025-08-14T21:48:02.6115093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6115485Z return func(*args, **kwargs) 2025-08-14T21:48:02.6116960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6117351Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6117778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6118198Z outputs = self.layoutlm( 2025-08-14T21:48:02.6118577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6118979Z return func(*args, **kwargs) 2025-08-14T21:48:02.6119370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6119768Z return func(*args, **kwargs) 2025-08-14T21:48:02.6120157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6120541Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6120978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6121401Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6121804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6122200Z return func(*args, **kwargs) 2025-08-14T21:48:02.6122582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6122968Z return func(*args, **kwargs) 2025-08-14T21:48:02.6123352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6123750Z return func(*args, **kwargs) 2025-08-14T21:48:02.6123955Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6124337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6124718Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6125148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6125559Z layer_outputs = layer_module( 2025-08-14T21:48:02.6125906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6126267Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6126638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6127003Z return func(*args, **kwargs) 2025-08-14T21:48:02.6127358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6127744Z return func(*args, **kwargs) 2025-08-14T21:48:02.6128095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6128463Z return func(*args, **kwargs) 2025-08-14T21:48:02.6128846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6129261Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6129665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6130065Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6130496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6130972Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6131417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6131869Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6132012Z 2025-08-14T21:48:02.6132128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6132482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6132809Z return mod(**inputs) 2025-08-14T21:48:02.6133164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6133525Z return func(*args, **kwargs) 2025-08-14T21:48:02.6133878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6134244Z return func(*args, **kwargs) 2025-08-14T21:48:02.6134595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6134939Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6135342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6135739Z outputs = self.layoutlm( 2025-08-14T21:48:02.6136096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6136451Z return func(*args, **kwargs) 2025-08-14T21:48:02.6136805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6137171Z return func(*args, **kwargs) 2025-08-14T21:48:02.6137498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6137851Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6138249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6138656Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6139023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6139388Z return func(*args, **kwargs) 2025-08-14T21:48:02.6139741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6140095Z return func(*args, **kwargs) 2025-08-14T21:48:02.6140447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6140810Z return func(*args, **kwargs) 2025-08-14T21:48:02.6141007Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6141352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6141732Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6142132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6142533Z layer_outputs = layer_module( 2025-08-14T21:48:02.6142874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6143229Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6143604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6143960Z return func(*args, **kwargs) 2025-08-14T21:48:02.6144317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6144686Z return func(*args, **kwargs) 2025-08-14T21:48:02.6145032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6145420Z return func(*args, **kwargs) 2025-08-14T21:48:02.6145863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6146277Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6146674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6147074Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6147489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6147956Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6148402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6148834Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6149223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6149565Z return self.act(input) 2025-08-14T21:48:02.6149692Z 2025-08-14T21:48:02.6149798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6150169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6150501Z return mod(**inputs) 2025-08-14T21:48:02.6150859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6151228Z return func(*args, **kwargs) 2025-08-14T21:48:02.6151580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6151941Z return func(*args, **kwargs) 2025-08-14T21:48:02.6152275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6152637Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6153056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6153460Z outputs = self.layoutlm( 2025-08-14T21:48:02.6153825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6154205Z return func(*args, **kwargs) 2025-08-14T21:48:02.6154570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6154936Z return func(*args, **kwargs) 2025-08-14T21:48:02.6155275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6155723Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6156158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6156599Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6156994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6157362Z return func(*args, **kwargs) 2025-08-14T21:48:02.6157717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6158085Z return func(*args, **kwargs) 2025-08-14T21:48:02.6158448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6158869Z return func(*args, **kwargs) 2025-08-14T21:48:02.6159066Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6159412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6159759Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6160188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6160660Z layer_outputs = layer_module( 2025-08-14T21:48:02.6161008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6161365Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6161747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6162117Z return func(*args, **kwargs) 2025-08-14T21:48:02.6162477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6162853Z return func(*args, **kwargs) 2025-08-14T21:48:02.6163209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6163582Z return func(*args, **kwargs) 2025-08-14T21:48:02.6163960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6164374Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6164774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6165163Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6165584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6166072Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6166527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6166935Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6167077Z 2025-08-14T21:48:02.6167183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6167543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6167867Z return mod(**inputs) 2025-08-14T21:48:02.6168209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6168576Z return func(*args, **kwargs) 2025-08-14T21:48:02.6168929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6169291Z return func(*args, **kwargs) 2025-08-14T21:48:02.6169619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6169971Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6170391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6170786Z outputs = self.layoutlm( 2025-08-14T21:48:02.6171134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6171499Z return func(*args, **kwargs) 2025-08-14T21:48:02.6171853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6172209Z return func(*args, **kwargs) 2025-08-14T21:48:02.6172542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6172890Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6173290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6173689Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6174093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6174450Z return func(*args, **kwargs) 2025-08-14T21:48:02.6174789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6175147Z return func(*args, **kwargs) 2025-08-14T21:48:02.6175491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6175845Z return func(*args, **kwargs) 2025-08-14T21:48:02.6176027Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6176383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6176743Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6177128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6177517Z layer_outputs = layer_module( 2025-08-14T21:48:02.6177854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6178204Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6178560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6178911Z return func(*args, **kwargs) 2025-08-14T21:48:02.6179258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6179604Z return func(*args, **kwargs) 2025-08-14T21:48:02.6179954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6180312Z return func(*args, **kwargs) 2025-08-14T21:48:02.6180683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6181083Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6181452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6181807Z return func(*args, **kwargs) 2025-08-14T21:48:02.6182149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6182495Z return func(*args, **kwargs) 2025-08-14T21:48:02.6182838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6183190Z return func(*args, **kwargs) 2025-08-14T21:48:02.6183555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6183963Z self_outputs = self.self( 2025-08-14T21:48:02.6184318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6184671Z return func(*args, **kwargs) 2025-08-14T21:48:02.6185009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6185361Z return func(*args, **kwargs) 2025-08-14T21:48:02.6185715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6186076Z return func(*args, **kwargs) 2025-08-14T21:48:02.6186461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6186936Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6187138Z 2025-08-14T21:48:02.6187253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6187645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6187964Z return mod(**inputs) 2025-08-14T21:48:02.6188309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6188671Z return func(*args, **kwargs) 2025-08-14T21:48:02.6189027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6189395Z return func(*args, **kwargs) 2025-08-14T21:48:02.6189728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6190077Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6190513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6190913Z outputs = self.layoutlm( 2025-08-14T21:48:02.6191273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6191630Z return func(*args, **kwargs) 2025-08-14T21:48:02.6191982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6192348Z return func(*args, **kwargs) 2025-08-14T21:48:02.6192671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6193022Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6193460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6193893Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6194276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6194665Z return func(*args, **kwargs) 2025-08-14T21:48:02.6195042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6195423Z return func(*args, **kwargs) 2025-08-14T21:48:02.6195876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6196279Z return func(*args, **kwargs) 2025-08-14T21:48:02.6196493Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6196887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6197247Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6197662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6198085Z layer_outputs = layer_module( 2025-08-14T21:48:02.6198436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6198800Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6199177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6199536Z return func(*args, **kwargs) 2025-08-14T21:48:02.6199889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6200252Z return func(*args, **kwargs) 2025-08-14T21:48:02.6200594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6200959Z return func(*args, **kwargs) 2025-08-14T21:48:02.6201338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6201746Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6202164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6202530Z return func(*args, **kwargs) 2025-08-14T21:48:02.6202880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6203241Z return func(*args, **kwargs) 2025-08-14T21:48:02.6203589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6203952Z return func(*args, **kwargs) 2025-08-14T21:48:02.6204335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6204721Z self_outputs = self.self( 2025-08-14T21:48:02.6205102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6205470Z return func(*args, **kwargs) 2025-08-14T21:48:02.6205821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6206183Z return func(*args, **kwargs) 2025-08-14T21:48:02.6206531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6206886Z return func(*args, **kwargs) 2025-08-14T21:48:02.6207261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6207728Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6207923Z 2025-08-14T21:48:02.6208028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6208378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6208808Z return mod(**inputs) 2025-08-14T21:48:02.6209180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6209548Z return func(*args, **kwargs) 2025-08-14T21:48:02.6209906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6210265Z return func(*args, **kwargs) 2025-08-14T21:48:02.6210599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6210934Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6211324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6211718Z outputs = self.layoutlm( 2025-08-14T21:48:02.6212073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6212490Z return func(*args, **kwargs) 2025-08-14T21:48:02.6212830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6213183Z return func(*args, **kwargs) 2025-08-14T21:48:02.6213513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6213849Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6214242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6214633Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6215000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6215343Z return func(*args, **kwargs) 2025-08-14T21:48:02.6215683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6216119Z return func(*args, **kwargs) 2025-08-14T21:48:02.6216470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6216820Z return func(*args, **kwargs) 2025-08-14T21:48:02.6217009Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6217341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6217668Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6218048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6218426Z layer_outputs = layer_module( 2025-08-14T21:48:02.6218783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6219132Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6219533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6219889Z return func(*args, **kwargs) 2025-08-14T21:48:02.6220231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6220583Z return func(*args, **kwargs) 2025-08-14T21:48:02.6220929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6221278Z return func(*args, **kwargs) 2025-08-14T21:48:02.6221657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6222048Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6222409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6222749Z return func(*args, **kwargs) 2025-08-14T21:48:02.6223089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6223437Z return func(*args, **kwargs) 2025-08-14T21:48:02.6223774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6224116Z return func(*args, **kwargs) 2025-08-14T21:48:02.6224482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6224862Z self_outputs = self.self( 2025-08-14T21:48:02.6225203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6225572Z return func(*args, **kwargs) 2025-08-14T21:48:02.6225954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6226328Z return func(*args, **kwargs) 2025-08-14T21:48:02.6226676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6227050Z return func(*args, **kwargs) 2025-08-14T21:48:02.6227491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6227959Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6228166Z 2025-08-14T21:48:02.6228248Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6228466Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6228703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6229062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6229389Z return mod(**inputs) 2025-08-14T21:48:02.6229786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6230145Z return func(*args, **kwargs) 2025-08-14T21:48:02.6230501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6230868Z return func(*args, **kwargs) 2025-08-14T21:48:02.6231198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6231548Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6231957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6232388Z outputs = self.layoutlm( 2025-08-14T21:48:02.6232779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6233171Z return func(*args, **kwargs) 2025-08-14T21:48:02.6233549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6233932Z return func(*args, **kwargs) 2025-08-14T21:48:02.6234282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6234663Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6235104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6235533Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6235988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6236379Z return func(*args, **kwargs) 2025-08-14T21:48:02.6236772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6237164Z return func(*args, **kwargs) 2025-08-14T21:48:02.6237557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6237923Z return func(*args, **kwargs) 2025-08-14T21:48:02.6238125Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6238469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6238819Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6239222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6239614Z layer_outputs = layer_module( 2025-08-14T21:48:02.6239964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6240353Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6240730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6241090Z return func(*args, **kwargs) 2025-08-14T21:48:02.6241447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6241815Z return func(*args, **kwargs) 2025-08-14T21:48:02.6242163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6242540Z return func(*args, **kwargs) 2025-08-14T21:48:02.6242911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6243305Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6243664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6244039Z return func(*args, **kwargs) 2025-08-14T21:48:02.6244412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6244771Z return func(*args, **kwargs) 2025-08-14T21:48:02.6245122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6245484Z return func(*args, **kwargs) 2025-08-14T21:48:02.6245864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6246309Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6246777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6247191Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6247330Z 2025-08-14T21:48:02.6247445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6247797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6248121Z return mod(**inputs) 2025-08-14T21:48:02.6248468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6248823Z return func(*args, **kwargs) 2025-08-14T21:48:02.6249182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6249536Z return func(*args, **kwargs) 2025-08-14T21:48:02.6249859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6250196Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6250585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6250977Z outputs = self.layoutlm( 2025-08-14T21:48:02.6251321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6251673Z return func(*args, **kwargs) 2025-08-14T21:48:02.6252017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6252370Z return func(*args, **kwargs) 2025-08-14T21:48:02.6252687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6253037Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6253423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6253859Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6254250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6254630Z return func(*args, **kwargs) 2025-08-14T21:48:02.6254983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6255340Z return func(*args, **kwargs) 2025-08-14T21:48:02.6255691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6256056Z return func(*args, **kwargs) 2025-08-14T21:48:02.6256250Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6256589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6256939Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6257338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6257726Z layer_outputs = layer_module( 2025-08-14T21:48:02.6258107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6258461Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6258828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6259178Z return func(*args, **kwargs) 2025-08-14T21:48:02.6259522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6259880Z return func(*args, **kwargs) 2025-08-14T21:48:02.6260217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6260605Z return func(*args, **kwargs) 2025-08-14T21:48:02.6260980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6261389Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6261778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6262163Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6262582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6263048Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6263477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6263875Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6264010Z 2025-08-14T21:48:02.6264124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6264467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6264786Z return mod(**inputs) 2025-08-14T21:48:02.6265130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6265492Z return func(*args, **kwargs) 2025-08-14T21:48:02.6265840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6266209Z return func(*args, **kwargs) 2025-08-14T21:48:02.6266546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6266896Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6267298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6267693Z outputs = self.layoutlm( 2025-08-14T21:48:02.6268073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6268430Z return func(*args, **kwargs) 2025-08-14T21:48:02.6268783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6269144Z return func(*args, **kwargs) 2025-08-14T21:48:02.6269468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6269815Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6270212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6270635Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6271018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6271406Z return func(*args, **kwargs) 2025-08-14T21:48:02.6271808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6272262Z return func(*args, **kwargs) 2025-08-14T21:48:02.6272626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6273014Z return func(*args, **kwargs) 2025-08-14T21:48:02.6273218Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6273583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6273965Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6274405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6274849Z layer_outputs = layer_module( 2025-08-14T21:48:02.6275220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6275685Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6276108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6276499Z return func(*args, **kwargs) 2025-08-14T21:48:02.6276890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6277298Z return func(*args, **kwargs) 2025-08-14T21:48:02.6277649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6278002Z return func(*args, **kwargs) 2025-08-14T21:48:02.6278370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6278764Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6279143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6279522Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6279934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6280398Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6280821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6281248Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6281620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6281950Z return self.act(input) 2025-08-14T21:48:02.6282065Z 2025-08-14T21:48:02.6282206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6282584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6282902Z return mod(**inputs) 2025-08-14T21:48:02.6283244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6283619Z return func(*args, **kwargs) 2025-08-14T21:48:02.6283980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6284348Z return func(*args, **kwargs) 2025-08-14T21:48:02.6284675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6285030Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6285440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6285841Z outputs = self.layoutlm( 2025-08-14T21:48:02.6286216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6286592Z return func(*args, **kwargs) 2025-08-14T21:48:02.6286934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6287289Z return func(*args, **kwargs) 2025-08-14T21:48:02.6287624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6287991Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6288401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6288833Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6289217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6289591Z return func(*args, **kwargs) 2025-08-14T21:48:02.6289932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6290290Z return func(*args, **kwargs) 2025-08-14T21:48:02.6290647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6291015Z return func(*args, **kwargs) 2025-08-14T21:48:02.6291203Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6291553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6291904Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6292299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6292699Z layer_outputs = layer_module( 2025-08-14T21:48:02.6293049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6293415Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6293788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6294156Z return func(*args, **kwargs) 2025-08-14T21:48:02.6294510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6294870Z return func(*args, **kwargs) 2025-08-14T21:48:02.6295227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6295593Z return func(*args, **kwargs) 2025-08-14T21:48:02.6295989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6296409Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6296808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6297197Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6297613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6298095Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6298547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6298947Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6299087Z 2025-08-14T21:48:02.6299194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6299554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6299874Z return mod(**inputs) 2025-08-14T21:48:02.6300272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6300628Z return func(*args, **kwargs) 2025-08-14T21:48:02.6300990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6301374Z return func(*args, **kwargs) 2025-08-14T21:48:02.6301704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6302090Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6302528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6302933Z outputs = self.layoutlm( 2025-08-14T21:48:02.6303306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6303678Z return func(*args, **kwargs) 2025-08-14T21:48:02.6304038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6304394Z return func(*args, **kwargs) 2025-08-14T21:48:02.6304732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6305087Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6305488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6305881Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6306252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6306618Z return func(*args, **kwargs) 2025-08-14T21:48:02.6306970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6307336Z return func(*args, **kwargs) 2025-08-14T21:48:02.6307694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6308061Z return func(*args, **kwargs) 2025-08-14T21:48:02.6308247Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6308598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6309144Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6309540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6309944Z layer_outputs = layer_module( 2025-08-14T21:48:02.6310297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6310714Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6311092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6311458Z return func(*args, **kwargs) 2025-08-14T21:48:02.6311818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6312180Z return func(*args, **kwargs) 2025-08-14T21:48:02.6312529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6312895Z return func(*args, **kwargs) 2025-08-14T21:48:02.6313282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6313711Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6314117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6314553Z return func(*args, **kwargs) 2025-08-14T21:48:02.6314958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6315340Z return func(*args, **kwargs) 2025-08-14T21:48:02.6315800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6316213Z return func(*args, **kwargs) 2025-08-14T21:48:02.6316638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6317083Z self_outputs = self.self( 2025-08-14T21:48:02.6317476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6317879Z return func(*args, **kwargs) 2025-08-14T21:48:02.6318229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6318597Z return func(*args, **kwargs) 2025-08-14T21:48:02.6318954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6319312Z return func(*args, **kwargs) 2025-08-14T21:48:02.6319693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6320168Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6320367Z 2025-08-14T21:48:02.6320481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6320831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6321153Z return mod(**inputs) 2025-08-14T21:48:02.6321502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6321869Z return func(*args, **kwargs) 2025-08-14T21:48:02.6322217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6322600Z return func(*args, **kwargs) 2025-08-14T21:48:02.6322951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6323326Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6323746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6324182Z outputs = self.layoutlm( 2025-08-14T21:48:02.6324577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6324959Z return func(*args, **kwargs) 2025-08-14T21:48:02.6325354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6325746Z return func(*args, **kwargs) 2025-08-14T21:48:02.6326100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6326487Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6326895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6327310Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6327702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6328090Z return func(*args, **kwargs) 2025-08-14T21:48:02.6328452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6328826Z return func(*args, **kwargs) 2025-08-14T21:48:02.6329185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6329616Z return func(*args, **kwargs) 2025-08-14T21:48:02.6329815Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6330167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6330525Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6330932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6331366Z layer_outputs = layer_module( 2025-08-14T21:48:02.6331739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6332153Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6332596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6332990Z return func(*args, **kwargs) 2025-08-14T21:48:02.6333380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6333768Z return func(*args, **kwargs) 2025-08-14T21:48:02.6334124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6334480Z return func(*args, **kwargs) 2025-08-14T21:48:02.6334861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6335287Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6335677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6336065Z return func(*args, **kwargs) 2025-08-14T21:48:02.6336444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6336848Z return func(*args, **kwargs) 2025-08-14T21:48:02.6337184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6337538Z return func(*args, **kwargs) 2025-08-14T21:48:02.6337912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6338298Z self_outputs = self.self( 2025-08-14T21:48:02.6338658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6339023Z return func(*args, **kwargs) 2025-08-14T21:48:02.6339375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6339757Z return func(*args, **kwargs) 2025-08-14T21:48:02.6340112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6340475Z return func(*args, **kwargs) 2025-08-14T21:48:02.6340849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6341319Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6341516Z 2025-08-14T21:48:02.6341626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6341990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6342308Z return mod(**inputs) 2025-08-14T21:48:02.6342662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6343034Z return func(*args, **kwargs) 2025-08-14T21:48:02.6343391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6343791Z return func(*args, **kwargs) 2025-08-14T21:48:02.6344128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6344480Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6344869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6345266Z outputs = self.layoutlm( 2025-08-14T21:48:02.6345623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6346020Z return func(*args, **kwargs) 2025-08-14T21:48:02.6346416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6346787Z return func(*args, **kwargs) 2025-08-14T21:48:02.6347132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6347491Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6347903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6348318Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6348698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6349067Z return func(*args, **kwargs) 2025-08-14T21:48:02.6349434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6349810Z return func(*args, **kwargs) 2025-08-14T21:48:02.6350174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6350543Z return func(*args, **kwargs) 2025-08-14T21:48:02.6350741Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6351105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6351457Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6351918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6352328Z layer_outputs = layer_module( 2025-08-14T21:48:02.6352680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6353044Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6353458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6353863Z return func(*args, **kwargs) 2025-08-14T21:48:02.6354248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6354621Z return func(*args, **kwargs) 2025-08-14T21:48:02.6354978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6355343Z return func(*args, **kwargs) 2025-08-14T21:48:02.6355821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6356282Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6356701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6357101Z return func(*args, **kwargs) 2025-08-14T21:48:02.6357473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6357839Z return func(*args, **kwargs) 2025-08-14T21:48:02.6358215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6358630Z return func(*args, **kwargs) 2025-08-14T21:48:02.6359067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6359515Z self_outputs = self.self( 2025-08-14T21:48:02.6359897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6360299Z return func(*args, **kwargs) 2025-08-14T21:48:02.6360681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6361087Z return func(*args, **kwargs) 2025-08-14T21:48:02.6361478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6361879Z return func(*args, **kwargs) 2025-08-14T21:48:02.6362300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6362817Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6363034Z 2025-08-14T21:48:02.6363123Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6363357Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6363621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6364012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6364389Z return mod(**inputs) 2025-08-14T21:48:02.6364772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6365176Z return func(*args, **kwargs) 2025-08-14T21:48:02.6365523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6365892Z return func(*args, **kwargs) 2025-08-14T21:48:02.6366225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6366570Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6366971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6367365Z outputs = self.layoutlm( 2025-08-14T21:48:02.6367720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6368079Z return func(*args, **kwargs) 2025-08-14T21:48:02.6368433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6368814Z return func(*args, **kwargs) 2025-08-14T21:48:02.6369137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6369489Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6369886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6370281Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6370638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6371002Z return func(*args, **kwargs) 2025-08-14T21:48:02.6371359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6371716Z return func(*args, **kwargs) 2025-08-14T21:48:02.6372072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6372436Z return func(*args, **kwargs) 2025-08-14T21:48:02.6372652Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6373032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6373417Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6373820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6374212Z layer_outputs = layer_module( 2025-08-14T21:48:02.6374559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6374919Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6375293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6375670Z return func(*args, **kwargs) 2025-08-14T21:48:02.6376028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6376394Z return func(*args, **kwargs) 2025-08-14T21:48:02.6376749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6377104Z return func(*args, **kwargs) 2025-08-14T21:48:02.6377484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6377894Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6378264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6378628Z return func(*args, **kwargs) 2025-08-14T21:48:02.6378982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6379345Z return func(*args, **kwargs) 2025-08-14T21:48:02.6379696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6380052Z return func(*args, **kwargs) 2025-08-14T21:48:02.6380422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6380859Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6381302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6381701Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6381838Z 2025-08-14T21:48:02.6381954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6382311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6382655Z return mod(**inputs) 2025-08-14T21:48:02.6383004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6383371Z return func(*args, **kwargs) 2025-08-14T21:48:02.6383719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6384085Z return func(*args, **kwargs) 2025-08-14T21:48:02.6384413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6384758Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6385144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6385525Z outputs = self.layoutlm( 2025-08-14T21:48:02.6385875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6386225Z return func(*args, **kwargs) 2025-08-14T21:48:02.6386588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6386969Z return func(*args, **kwargs) 2025-08-14T21:48:02.6387288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6387631Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6388022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6388411Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6388770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6389141Z return func(*args, **kwargs) 2025-08-14T21:48:02.6389515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6389874Z return func(*args, **kwargs) 2025-08-14T21:48:02.6390229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6390592Z return func(*args, **kwargs) 2025-08-14T21:48:02.6390787Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6391126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6391478Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6391897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6392308Z layer_outputs = layer_module( 2025-08-14T21:48:02.6392678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6393061Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6393456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6393836Z return func(*args, **kwargs) 2025-08-14T21:48:02.6394210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6394595Z return func(*args, **kwargs) 2025-08-14T21:48:02.6394963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6395343Z return func(*args, **kwargs) 2025-08-14T21:48:02.6395830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6396277Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6396701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6397152Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6397613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6398123Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6398594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6399030Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6399178Z 2025-08-14T21:48:02.6399300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6399687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6400025Z return mod(**inputs) 2025-08-14T21:48:02.6400405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6400795Z return func(*args, **kwargs) 2025-08-14T21:48:02.6401202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6401595Z return func(*args, **kwargs) 2025-08-14T21:48:02.6401953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6402332Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6402766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6403191Z outputs = self.layoutlm( 2025-08-14T21:48:02.6403569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6403959Z return func(*args, **kwargs) 2025-08-14T21:48:02.6404368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6404756Z return func(*args, **kwargs) 2025-08-14T21:48:02.6405112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6405473Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6405912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6406351Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6406733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6407128Z return func(*args, **kwargs) 2025-08-14T21:48:02.6407500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6407888Z return func(*args, **kwargs) 2025-08-14T21:48:02.6408255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6408796Z return func(*args, **kwargs) 2025-08-14T21:48:02.6409025Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6409394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6410725Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6411280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6411736Z layer_outputs = layer_module( 2025-08-14T21:48:02.6412131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6412523Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6413008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6413687Z return func(*args, **kwargs) 2025-08-14T21:48:02.6414112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6414515Z return func(*args, **kwargs) 2025-08-14T21:48:02.6414897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6415317Z return func(*args, **kwargs) 2025-08-14T21:48:02.6415700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6415789Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6416054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6416131Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6416707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6416960Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6417247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6417371Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6417588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6417670Z return self.act(input) 2025-08-14T21:48:02.6417680Z 2025-08-14T21:48:02.6417794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6417999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6418073Z return mod(**inputs) 2025-08-14T21:48:02.6418350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6418425Z return func(*args, **kwargs) 2025-08-14T21:48:02.6418670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6418739Z return func(*args, **kwargs) 2025-08-14T21:48:02.6418962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6419039Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6419307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6419389Z outputs = self.layoutlm( 2025-08-14T21:48:02.6419622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6419692Z return func(*args, **kwargs) 2025-08-14T21:48:02.6419930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6419998Z return func(*args, **kwargs) 2025-08-14T21:48:02.6420216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6420289Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6420554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6420636Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6420869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6420942Z return func(*args, **kwargs) 2025-08-14T21:48:02.6421179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6421245Z return func(*args, **kwargs) 2025-08-14T21:48:02.6421498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6421567Z return func(*args, **kwargs) 2025-08-14T21:48:02.6421647Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6421864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6421937Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6422207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6422281Z layer_outputs = layer_module( 2025-08-14T21:48:02.6422498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6422588Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6422824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6422892Z return func(*args, **kwargs) 2025-08-14T21:48:02.6423172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6423241Z return func(*args, **kwargs) 2025-08-14T21:48:02.6423482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6423548Z return func(*args, **kwargs) 2025-08-14T21:48:02.6423817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6423921Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6424175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6424267Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6424564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6424701Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6424966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6425049Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6425053Z 2025-08-14T21:48:02.6425162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6425373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6425440Z return mod(**inputs) 2025-08-14T21:48:02.6425707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6425774Z return func(*args, **kwargs) 2025-08-14T21:48:02.6426005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6426080Z return func(*args, **kwargs) 2025-08-14T21:48:02.6426291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6426363Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6426631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6426702Z outputs = self.layoutlm( 2025-08-14T21:48:02.6426938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6427003Z return func(*args, **kwargs) 2025-08-14T21:48:02.6427231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6427302Z return func(*args, **kwargs) 2025-08-14T21:48:02.6427523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6427598Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6427862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6427934Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6428168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6428234Z return func(*args, **kwargs) 2025-08-14T21:48:02.6428462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6428532Z return func(*args, **kwargs) 2025-08-14T21:48:02.6428766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6428840Z return func(*args, **kwargs) 2025-08-14T21:48:02.6428917Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6429168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6429247Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6429512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6429582Z layer_outputs = layer_module( 2025-08-14T21:48:02.6429807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6429891Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6430152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6430237Z return func(*args, **kwargs) 2025-08-14T21:48:02.6430488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6430569Z return func(*args, **kwargs) 2025-08-14T21:48:02.6430825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6430896Z return func(*args, **kwargs) 2025-08-14T21:48:02.6431195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6431283Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6431543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6431612Z return func(*args, **kwargs) 2025-08-14T21:48:02.6431869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6431949Z return func(*args, **kwargs) 2025-08-14T21:48:02.6432202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6432274Z return func(*args, **kwargs) 2025-08-14T21:48:02.6432572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6432647Z self_outputs = self.self( 2025-08-14T21:48:02.6432908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6432979Z return func(*args, **kwargs) 2025-08-14T21:48:02.6433233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6433311Z return func(*args, **kwargs) 2025-08-14T21:48:02.6433569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6433664Z return func(*args, **kwargs) 2025-08-14T21:48:02.6433961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6434120Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6434125Z 2025-08-14T21:48:02.6434245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6434467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6434538Z return mod(**inputs) 2025-08-14T21:48:02.6434802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6434874Z return func(*args, **kwargs) 2025-08-14T21:48:02.6435140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6435212Z return func(*args, **kwargs) 2025-08-14T21:48:02.6435444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6435766Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6436080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6436156Z outputs = self.layoutlm( 2025-08-14T21:48:02.6436426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6436496Z return func(*args, **kwargs) 2025-08-14T21:48:02.6436762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6436832Z return func(*args, **kwargs) 2025-08-14T21:48:02.6437063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6437152Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6437427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6437522Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6437754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6437819Z return func(*args, **kwargs) 2025-08-14T21:48:02.6438062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6438129Z return func(*args, **kwargs) 2025-08-14T21:48:02.6438355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6438431Z return func(*args, **kwargs) 2025-08-14T21:48:02.6438511Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6438733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6438808Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6439072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6439150Z layer_outputs = layer_module( 2025-08-14T21:48:02.6439374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6439453Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6439688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6439753Z return func(*args, **kwargs) 2025-08-14T21:48:02.6439987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6440070Z return func(*args, **kwargs) 2025-08-14T21:48:02.6440300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6440379Z return func(*args, **kwargs) 2025-08-14T21:48:02.6440635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6440717Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6440955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6441020Z return func(*args, **kwargs) 2025-08-14T21:48:02.6441258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6441322Z return func(*args, **kwargs) 2025-08-14T21:48:02.6441550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6441626Z return func(*args, **kwargs) 2025-08-14T21:48:02.6441917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6441999Z self_outputs = self.self( 2025-08-14T21:48:02.6442228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6442294Z return func(*args, **kwargs) 2025-08-14T21:48:02.6442535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6442599Z return func(*args, **kwargs) 2025-08-14T21:48:02.6442826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6442899Z return func(*args, **kwargs) 2025-08-14T21:48:02.6443174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6443321Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6443326Z 2025-08-14T21:48:02.6443430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6443627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6443699Z return mod(**inputs) 2025-08-14T21:48:02.6443937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6444003Z return func(*args, **kwargs) 2025-08-14T21:48:02.6444245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6444310Z return func(*args, **kwargs) 2025-08-14T21:48:02.6444532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6444608Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6444878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6444957Z outputs = self.layoutlm( 2025-08-14T21:48:02.6445209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6445285Z return func(*args, **kwargs) 2025-08-14T21:48:02.6445537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6445615Z return func(*args, **kwargs) 2025-08-14T21:48:02.6445838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6445915Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6446180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6446284Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6446523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6446596Z return func(*args, **kwargs) 2025-08-14T21:48:02.6446832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6446898Z return func(*args, **kwargs) 2025-08-14T21:48:02.6447140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6447207Z return func(*args, **kwargs) 2025-08-14T21:48:02.6447283Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6447508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6447583Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6447877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6447977Z layer_outputs = layer_module( 2025-08-14T21:48:02.6448200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6448286Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6448525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6448592Z return func(*args, **kwargs) 2025-08-14T21:48:02.6448838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6448904Z return func(*args, **kwargs) 2025-08-14T21:48:02.6449172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6449242Z return func(*args, **kwargs) 2025-08-14T21:48:02.6449507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6449598Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6449837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6449903Z return func(*args, **kwargs) 2025-08-14T21:48:02.6450146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6450213Z return func(*args, **kwargs) 2025-08-14T21:48:02.6450456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6450522Z return func(*args, **kwargs) 2025-08-14T21:48:02.6450791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6450875Z self_outputs = self.self( 2025-08-14T21:48:02.6451114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6451191Z return func(*args, **kwargs) 2025-08-14T21:48:02.6451425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6451491Z return func(*args, **kwargs) 2025-08-14T21:48:02.6451758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6451825Z return func(*args, **kwargs) 2025-08-14T21:48:02.6452090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6452245Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6452265Z 2025-08-14T21:48:02.6452354Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6452444Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6452549Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6452746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6452821Z return mod(**inputs) 2025-08-14T21:48:02.6453056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6453123Z return func(*args, **kwargs) 2025-08-14T21:48:02.6453362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6453427Z return func(*args, **kwargs) 2025-08-14T21:48:02.6453650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6453725Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6454025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6454104Z outputs = self.layoutlm( 2025-08-14T21:48:02.6454342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6454416Z return func(*args, **kwargs) 2025-08-14T21:48:02.6454653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6454721Z return func(*args, **kwargs) 2025-08-14T21:48:02.6454943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6455019Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6455301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6455388Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6455628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6455705Z return func(*args, **kwargs) 2025-08-14T21:48:02.6455943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6456012Z return func(*args, **kwargs) 2025-08-14T21:48:02.6456261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6456331Z return func(*args, **kwargs) 2025-08-14T21:48:02.6456407Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6456638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6456713Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6456993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6457068Z layer_outputs = layer_module( 2025-08-14T21:48:02.6457292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6457380Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6457622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6457690Z return func(*args, **kwargs) 2025-08-14T21:48:02.6457937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6458005Z return func(*args, **kwargs) 2025-08-14T21:48:02.6458255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6458343Z return func(*args, **kwargs) 2025-08-14T21:48:02.6458614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6458705Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6458959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6459026Z return func(*args, **kwargs) 2025-08-14T21:48:02.6459268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6459332Z return func(*args, **kwargs) 2025-08-14T21:48:02.6459572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6459637Z return func(*args, **kwargs) 2025-08-14T21:48:02.6459901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6460084Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6460347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6460435Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6460439Z 2025-08-14T21:48:02.6460541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6460729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6460798Z return mod(**inputs) 2025-08-14T21:48:02.6461019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6461082Z return func(*args, **kwargs) 2025-08-14T21:48:02.6461329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6461396Z return func(*args, **kwargs) 2025-08-14T21:48:02.6461609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6461680Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6461930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6462004Z outputs = self.layoutlm( 2025-08-14T21:48:02.6462227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6462290Z return func(*args, **kwargs) 2025-08-14T21:48:02.6462522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6462585Z return func(*args, **kwargs) 2025-08-14T21:48:02.6462796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6462869Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6463130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6463210Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6463443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6463515Z return func(*args, **kwargs) 2025-08-14T21:48:02.6463746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6463810Z return func(*args, **kwargs) 2025-08-14T21:48:02.6464046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6464113Z return func(*args, **kwargs) 2025-08-14T21:48:02.6464210Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6464427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6464499Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6464762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6464831Z layer_outputs = layer_module( 2025-08-14T21:48:02.6465051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6465134Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6465357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6465420Z return func(*args, **kwargs) 2025-08-14T21:48:02.6487950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6488165Z return func(*args, **kwargs) 2025-08-14T21:48:02.6488599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6488680Z return func(*args, **kwargs) 2025-08-14T21:48:02.6488978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6489077Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6489344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6489438Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6489738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6489898Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6490180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6490273Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6490281Z 2025-08-14T21:48:02.6490401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6490614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6490693Z return mod(**inputs) 2025-08-14T21:48:02.6490942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6491015Z return func(*args, **kwargs) 2025-08-14T21:48:02.6491263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6491336Z return func(*args, **kwargs) 2025-08-14T21:48:02.6491556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6491647Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6491919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6492007Z outputs = self.layoutlm( 2025-08-14T21:48:02.6492258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6492330Z return func(*args, **kwargs) 2025-08-14T21:48:02.6492595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6492662Z return func(*args, **kwargs) 2025-08-14T21:48:02.6492887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6492964Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6493261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6493353Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6493591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6493660Z return func(*args, **kwargs) 2025-08-14T21:48:02.6493903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6493977Z return func(*args, **kwargs) 2025-08-14T21:48:02.6494233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6494304Z return func(*args, **kwargs) 2025-08-14T21:48:02.6494387Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6494633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6494709Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6495016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6495100Z layer_outputs = layer_module( 2025-08-14T21:48:02.6495326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6495418Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6495660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6495728Z return func(*args, **kwargs) 2025-08-14T21:48:02.6495972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6496085Z return func(*args, **kwargs) 2025-08-14T21:48:02.6496320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6496396Z return func(*args, **kwargs) 2025-08-14T21:48:02.6496661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6496756Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6497016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6497093Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6497396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6497520Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6497795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6497914Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6498128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6498211Z return self.act(input) 2025-08-14T21:48:02.6498215Z 2025-08-14T21:48:02.6498325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6498532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6498607Z return mod(**inputs) 2025-08-14T21:48:02.6498843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6498920Z return func(*args, **kwargs) 2025-08-14T21:48:02.6499154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6499223Z return func(*args, **kwargs) 2025-08-14T21:48:02.6499465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6499546Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6499810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6499893Z outputs = self.layoutlm( 2025-08-14T21:48:02.6500133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6500208Z return func(*args, **kwargs) 2025-08-14T21:48:02.6500441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6500508Z return func(*args, **kwargs) 2025-08-14T21:48:02.6500730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6500806Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6501095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6501190Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6501431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6501507Z return func(*args, **kwargs) 2025-08-14T21:48:02.6501747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6501815Z return func(*args, **kwargs) 2025-08-14T21:48:02.6502062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6502129Z return func(*args, **kwargs) 2025-08-14T21:48:02.6502231Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6502446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6502523Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6502795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6502867Z layer_outputs = layer_module( 2025-08-14T21:48:02.6503089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6503181Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6503417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6503493Z return func(*args, **kwargs) 2025-08-14T21:48:02.6503727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6503797Z return func(*args, **kwargs) 2025-08-14T21:48:02.6504040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6504109Z return func(*args, **kwargs) 2025-08-14T21:48:02.6504376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6504473Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6504732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6504817Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6505113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6505249Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6505525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6505630Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6505635Z 2025-08-14T21:48:02.6505752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6505958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6506030Z return mod(**inputs) 2025-08-14T21:48:02.6506278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6506350Z return func(*args, **kwargs) 2025-08-14T21:48:02.6506589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6506667Z return func(*args, **kwargs) 2025-08-14T21:48:02.6506888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6506977Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6507267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6507357Z outputs = self.layoutlm( 2025-08-14T21:48:02.6507606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6507671Z return func(*args, **kwargs) 2025-08-14T21:48:02.6507916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6507984Z return func(*args, **kwargs) 2025-08-14T21:48:02.6508198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6508281Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6508573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6508970Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6509382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6509459Z return func(*args, **kwargs) 2025-08-14T21:48:02.6509717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6509789Z return func(*args, **kwargs) 2025-08-14T21:48:02.6510039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6510118Z return func(*args, **kwargs) 2025-08-14T21:48:02.6510202Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6510429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6510518Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6510814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6510905Z layer_outputs = layer_module( 2025-08-14T21:48:02.6511140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6511236Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6511484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6511553Z return func(*args, **kwargs) 2025-08-14T21:48:02.6511788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6511867Z return func(*args, **kwargs) 2025-08-14T21:48:02.6512126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6512277Z return func(*args, **kwargs) 2025-08-14T21:48:02.6512569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6512662Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6512920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6512991Z return func(*args, **kwargs) 2025-08-14T21:48:02.6513256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6513333Z return func(*args, **kwargs) 2025-08-14T21:48:02.6513591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6513672Z return func(*args, **kwargs) 2025-08-14T21:48:02.6513967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6514081Z self_outputs = self.self( 2025-08-14T21:48:02.6514386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6514459Z return func(*args, **kwargs) 2025-08-14T21:48:02.6514725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6514795Z return func(*args, **kwargs) 2025-08-14T21:48:02.6515051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6515130Z return func(*args, **kwargs) 2025-08-14T21:48:02.6515428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6515686Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6515707Z 2025-08-14T21:48:02.6515831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6516046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6516125Z return mod(**inputs) 2025-08-14T21:48:02.6516384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6516454Z return func(*args, **kwargs) 2025-08-14T21:48:02.6516712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6516780Z return func(*args, **kwargs) 2025-08-14T21:48:02.6517004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6517090Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6517386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6517475Z outputs = self.layoutlm( 2025-08-14T21:48:02.6517735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6517806Z return func(*args, **kwargs) 2025-08-14T21:48:02.6518073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6518143Z return func(*args, **kwargs) 2025-08-14T21:48:02.6518375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6518454Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6518749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6518837Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6519121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6519195Z return func(*args, **kwargs) 2025-08-14T21:48:02.6519449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6519518Z return func(*args, **kwargs) 2025-08-14T21:48:02.6519779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6519851Z return func(*args, **kwargs) 2025-08-14T21:48:02.6519932Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6520167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6520244Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6520536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6520620Z layer_outputs = layer_module( 2025-08-14T21:48:02.6520890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6520985Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6521234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6521306Z return func(*args, **kwargs) 2025-08-14T21:48:02.6521567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6521639Z return func(*args, **kwargs) 2025-08-14T21:48:02.6521888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6521966Z return func(*args, **kwargs) 2025-08-14T21:48:02.6522285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6522389Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6522618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6522684Z return func(*args, **kwargs) 2025-08-14T21:48:02.6522920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6522985Z return func(*args, **kwargs) 2025-08-14T21:48:02.6523219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6523286Z return func(*args, **kwargs) 2025-08-14T21:48:02.6523543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6523622Z self_outputs = self.self( 2025-08-14T21:48:02.6523851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6523920Z return func(*args, **kwargs) 2025-08-14T21:48:02.6524156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6524221Z return func(*args, **kwargs) 2025-08-14T21:48:02.6524454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6524518Z return func(*args, **kwargs) 2025-08-14T21:48:02.6524772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6524926Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6524932Z 2025-08-14T21:48:02.6525033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6525248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6525314Z return mod(**inputs) 2025-08-14T21:48:02.6525538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6525612Z return func(*args, **kwargs) 2025-08-14T21:48:02.6525839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6525901Z return func(*args, **kwargs) 2025-08-14T21:48:02.6526112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6526183Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6526440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6526508Z outputs = self.layoutlm( 2025-08-14T21:48:02.6526730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6526829Z return func(*args, **kwargs) 2025-08-14T21:48:02.6527056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6527119Z return func(*args, **kwargs) 2025-08-14T21:48:02.6527326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6527396Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6527654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6527721Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6527961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6528025Z return func(*args, **kwargs) 2025-08-14T21:48:02.6528241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6528314Z return func(*args, **kwargs) 2025-08-14T21:48:02.6528540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6528604Z return func(*args, **kwargs) 2025-08-14T21:48:02.6528675Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6528880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6528956Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6529209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6529278Z layer_outputs = layer_module( 2025-08-14T21:48:02.6529503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6529582Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6529822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6529888Z return func(*args, **kwargs) 2025-08-14T21:48:02.6530114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6530186Z return func(*args, **kwargs) 2025-08-14T21:48:02.6530414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6530484Z return func(*args, **kwargs) 2025-08-14T21:48:02.6530745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6530827Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6531087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6531154Z return func(*args, **kwargs) 2025-08-14T21:48:02.6531372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6531444Z return func(*args, **kwargs) 2025-08-14T21:48:02.6531666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6531732Z return func(*args, **kwargs) 2025-08-14T21:48:02.6531980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6532046Z self_outputs = self.self( 2025-08-14T21:48:02.6532280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6532345Z return func(*args, **kwargs) 2025-08-14T21:48:02.6532585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6532683Z return func(*args, **kwargs) 2025-08-14T21:48:02.6532913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6532984Z return func(*args, **kwargs) 2025-08-14T21:48:02.6533246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6533389Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6533392Z 2025-08-14T21:48:02.6533483Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6533561Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6533682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6533882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6533947Z return mod(**inputs) 2025-08-14T21:48:02.6534188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6534254Z return func(*args, **kwargs) 2025-08-14T21:48:02.6534485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6534558Z return func(*args, **kwargs) 2025-08-14T21:48:02.6534768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6534849Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6535110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6535178Z outputs = self.layoutlm( 2025-08-14T21:48:02.6535417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6535485Z return func(*args, **kwargs) 2025-08-14T21:48:02.6535729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6535802Z return func(*args, **kwargs) 2025-08-14T21:48:02.6536006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6536085Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6536340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6536411Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6536644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6536727Z return func(*args, **kwargs) 2025-08-14T21:48:02.6536954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6537028Z return func(*args, **kwargs) 2025-08-14T21:48:02.6537255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6537328Z return func(*args, **kwargs) 2025-08-14T21:48:02.6537402Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6537609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6537686Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6537946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6538015Z layer_outputs = layer_module( 2025-08-14T21:48:02.6538237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6538331Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6538576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6538643Z return func(*args, **kwargs) 2025-08-14T21:48:02.6538872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6538945Z return func(*args, **kwargs) 2025-08-14T21:48:02.6539177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6539249Z return func(*args, **kwargs) 2025-08-14T21:48:02.6539506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6539601Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6539839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6539907Z return func(*args, **kwargs) 2025-08-14T21:48:02.6540136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6540211Z return func(*args, **kwargs) 2025-08-14T21:48:02.6540441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6540512Z return func(*args, **kwargs) 2025-08-14T21:48:02.6540769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6540897Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6541164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6541249Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6541254Z 2025-08-14T21:48:02.6541366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6541556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6541622Z return mod(**inputs) 2025-08-14T21:48:02.6541867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6541934Z return func(*args, **kwargs) 2025-08-14T21:48:02.6542171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6542246Z return func(*args, **kwargs) 2025-08-14T21:48:02.6542461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6542544Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6542827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6542900Z outputs = self.layoutlm( 2025-08-14T21:48:02.6543140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6543207Z return func(*args, **kwargs) 2025-08-14T21:48:02.6543440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6543515Z return func(*args, **kwargs) 2025-08-14T21:48:02.6543737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6543817Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6544073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6544147Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6544398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6544483Z return func(*args, **kwargs) 2025-08-14T21:48:02.6544708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6544779Z return func(*args, **kwargs) 2025-08-14T21:48:02.6545005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6545076Z return func(*args, **kwargs) 2025-08-14T21:48:02.6545150Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6545355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6545432Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6545702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6545776Z layer_outputs = layer_module( 2025-08-14T21:48:02.6545996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6546071Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6546304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6546368Z return func(*args, **kwargs) 2025-08-14T21:48:02.6546595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6546668Z return func(*args, **kwargs) 2025-08-14T21:48:02.6546895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6546969Z return func(*args, **kwargs) 2025-08-14T21:48:02.6547224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6547311Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6547569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6547644Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6547940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6548071Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6548331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6548422Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6548427Z 2025-08-14T21:48:02.6548732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6548928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6549009Z return mod(**inputs) 2025-08-14T21:48:02.6549247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6549324Z return func(*args, **kwargs) 2025-08-14T21:48:02.6549560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6549627Z return func(*args, **kwargs) 2025-08-14T21:48:02.6549848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6549923Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6550189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6550273Z outputs = self.layoutlm( 2025-08-14T21:48:02.6550533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6550629Z return func(*args, **kwargs) 2025-08-14T21:48:02.6550874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6550939Z return func(*args, **kwargs) 2025-08-14T21:48:02.6551156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6551229Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6551492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6551575Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6551827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6551905Z return func(*args, **kwargs) 2025-08-14T21:48:02.6552144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6552211Z return func(*args, **kwargs) 2025-08-14T21:48:02.6552455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6552522Z return func(*args, **kwargs) 2025-08-14T21:48:02.6552599Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6552823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6552895Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6553175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6553246Z layer_outputs = layer_module( 2025-08-14T21:48:02.6553481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6553577Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6553827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6553896Z return func(*args, **kwargs) 2025-08-14T21:48:02.6554159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6554229Z return func(*args, **kwargs) 2025-08-14T21:48:02.6554492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6554560Z return func(*args, **kwargs) 2025-08-14T21:48:02.6554853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6554974Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6555248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6555336Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6555750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6555886Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6556186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6556307Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6556533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6556622Z return self.act(input) 2025-08-14T21:48:02.6556628Z 2025-08-14T21:48:02.6556739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6557015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6557083Z return mod(**inputs) 2025-08-14T21:48:02.6557317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6557393Z return func(*args, **kwargs) 2025-08-14T21:48:02.6557621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6557695Z return func(*args, **kwargs) 2025-08-14T21:48:02.6557901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6557975Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6558254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6558325Z outputs = self.layoutlm( 2025-08-14T21:48:02.6558558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6558631Z return func(*args, **kwargs) 2025-08-14T21:48:02.6558864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6558933Z return func(*args, **kwargs) 2025-08-14T21:48:02.6559140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6559211Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6559477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6559549Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6559781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6559857Z return func(*args, **kwargs) 2025-08-14T21:48:02.6560090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6560165Z return func(*args, **kwargs) 2025-08-14T21:48:02.6560396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6560461Z return func(*args, **kwargs) 2025-08-14T21:48:02.6560542Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6560754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6560825Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6561092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6561182Z layer_outputs = layer_module( 2025-08-14T21:48:02.6561406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6561484Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6561713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6561785Z return func(*args, **kwargs) 2025-08-14T21:48:02.6562016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6562087Z return func(*args, **kwargs) 2025-08-14T21:48:02.6562316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6562382Z return func(*args, **kwargs) 2025-08-14T21:48:02.6562654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6562738Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6563029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6563114Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6563409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6563550Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6563820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6563898Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6563902Z 2025-08-14T21:48:02.6564011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6564225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6564304Z return mod(**inputs) 2025-08-14T21:48:02.6564540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6564607Z return func(*args, **kwargs) 2025-08-14T21:48:02.6564846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6564913Z return func(*args, **kwargs) 2025-08-14T21:48:02.6565122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6565203Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6565463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6565542Z outputs = self.layoutlm( 2025-08-14T21:48:02.6565777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6565844Z return func(*args, **kwargs) 2025-08-14T21:48:02.6566087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6566153Z return func(*args, **kwargs) 2025-08-14T21:48:02.6566363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6566447Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6566709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6566789Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6567023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6567091Z return func(*args, **kwargs) 2025-08-14T21:48:02.6567357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6567426Z return func(*args, **kwargs) 2025-08-14T21:48:02.6567667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6567742Z return func(*args, **kwargs) 2025-08-14T21:48:02.6567819Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6568041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6568114Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6568381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6568460Z layer_outputs = layer_module( 2025-08-14T21:48:02.6568685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6568765Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6569050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6569120Z return func(*args, **kwargs) 2025-08-14T21:48:02.6569366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6569432Z return func(*args, **kwargs) 2025-08-14T21:48:02.6569666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6569739Z return func(*args, **kwargs) 2025-08-14T21:48:02.6570003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6570106Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6570344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6570414Z return func(*args, **kwargs) 2025-08-14T21:48:02.6570656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6570722Z return func(*args, **kwargs) 2025-08-14T21:48:02.6570955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6571030Z return func(*args, **kwargs) 2025-08-14T21:48:02.6571300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6571377Z self_outputs = self.self( 2025-08-14T21:48:02.6571615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6571681Z return func(*args, **kwargs) 2025-08-14T21:48:02.6571927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6571997Z return func(*args, **kwargs) 2025-08-14T21:48:02.6572234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6572307Z return func(*args, **kwargs) 2025-08-14T21:48:02.6572573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6572728Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6572732Z 2025-08-14T21:48:02.6572836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6573036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6573113Z return mod(**inputs) 2025-08-14T21:48:02.6573366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6573443Z return func(*args, **kwargs) 2025-08-14T21:48:02.6573675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6573742Z return func(*args, **kwargs) 2025-08-14T21:48:02.6573959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6574033Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6574295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6574372Z outputs = self.layoutlm( 2025-08-14T21:48:02.6574607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6574681Z return func(*args, **kwargs) 2025-08-14T21:48:02.6574914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6575013Z return func(*args, **kwargs) 2025-08-14T21:48:02.6575238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6575311Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6575578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6575659Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6575892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6575965Z return func(*args, **kwargs) 2025-08-14T21:48:02.6576217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6576287Z return func(*args, **kwargs) 2025-08-14T21:48:02.6576534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6576598Z return func(*args, **kwargs) 2025-08-14T21:48:02.6576675Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6576896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6576970Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6577245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6577314Z layer_outputs = layer_module( 2025-08-14T21:48:02.6577544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6577628Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6577853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6577924Z return func(*args, **kwargs) 2025-08-14T21:48:02.6578146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6578208Z return func(*args, **kwargs) 2025-08-14T21:48:02.6578439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6578502Z return func(*args, **kwargs) 2025-08-14T21:48:02.6578756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6578843Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6579069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6579168Z return func(*args, **kwargs) 2025-08-14T21:48:02.6579393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6579459Z return func(*args, **kwargs) 2025-08-14T21:48:02.6579690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6579752Z return func(*args, **kwargs) 2025-08-14T21:48:02.6580003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6580078Z self_outputs = self.self( 2025-08-14T21:48:02.6580303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6580374Z return func(*args, **kwargs) 2025-08-14T21:48:02.6580600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6580664Z return func(*args, **kwargs) 2025-08-14T21:48:02.6580929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6580994Z return func(*args, **kwargs) 2025-08-14T21:48:02.6581249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6581391Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6581395Z 2025-08-14T21:48:02.6581495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6581689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6581752Z return mod(**inputs) 2025-08-14T21:48:02.6581995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6582070Z return func(*args, **kwargs) 2025-08-14T21:48:02.6582296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6582367Z return func(*args, **kwargs) 2025-08-14T21:48:02.6582574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6582647Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6582914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6582982Z outputs = self.layoutlm( 2025-08-14T21:48:02.6583215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6583288Z return func(*args, **kwargs) 2025-08-14T21:48:02.6583518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6583593Z return func(*args, **kwargs) 2025-08-14T21:48:02.6583806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6583878Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6584142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6584212Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6584443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6584515Z return func(*args, **kwargs) 2025-08-14T21:48:02.6584742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6584813Z return func(*args, **kwargs) 2025-08-14T21:48:02.6585042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6585125Z return func(*args, **kwargs) 2025-08-14T21:48:02.6585209Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6585416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6585499Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6585755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6585825Z layer_outputs = layer_module( 2025-08-14T21:48:02.6586047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6586125Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6586362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6586437Z return func(*args, **kwargs) 2025-08-14T21:48:02.6586696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6586786Z return func(*args, **kwargs) 2025-08-14T21:48:02.6587014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6587081Z return func(*args, **kwargs) 2025-08-14T21:48:02.6587349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6587434Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6587679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6587746Z return func(*args, **kwargs) 2025-08-14T21:48:02.6588021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6588098Z return func(*args, **kwargs) 2025-08-14T21:48:02.6588329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6588394Z return func(*args, **kwargs) 2025-08-14T21:48:02.6588662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6588730Z self_outputs = self.self( 2025-08-14T21:48:02.6588971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6589034Z return func(*args, **kwargs) 2025-08-14T21:48:02.6589264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6589336Z return func(*args, **kwargs) 2025-08-14T21:48:02.6589566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6589634Z return func(*args, **kwargs) 2025-08-14T21:48:02.6589902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6590046Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6590050Z 2025-08-14T21:48:02.6590135Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6590210Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6590311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6590513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6590580Z return mod(**inputs) 2025-08-14T21:48:02.6590815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6590911Z return func(*args, **kwargs) 2025-08-14T21:48:02.6591145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6591220Z return func(*args, **kwargs) 2025-08-14T21:48:02.6591434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6591506Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6591778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6591845Z outputs = self.layoutlm( 2025-08-14T21:48:02.6592087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6592152Z return func(*args, **kwargs) 2025-08-14T21:48:02.6592393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6592469Z return func(*args, **kwargs) 2025-08-14T21:48:02.6592746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6592824Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6593103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6593179Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6593421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6593486Z return func(*args, **kwargs) 2025-08-14T21:48:02.6593718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6593792Z return func(*args, **kwargs) 2025-08-14T21:48:02.6594043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6594111Z return func(*args, **kwargs) 2025-08-14T21:48:02.6594197Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6594409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6594486Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6594750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6594819Z layer_outputs = layer_module( 2025-08-14T21:48:02.6595044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6595123Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6595360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6595436Z return func(*args, **kwargs) 2025-08-14T21:48:02.6595772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6595861Z return func(*args, **kwargs) 2025-08-14T21:48:02.6596112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6596181Z return func(*args, **kwargs) 2025-08-14T21:48:02.6596469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6596557Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6596811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6596881Z return func(*args, **kwargs) 2025-08-14T21:48:02.6597140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6597243Z return func(*args, **kwargs) 2025-08-14T21:48:02.6597502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6597572Z return func(*args, **kwargs) 2025-08-14T21:48:02.6597878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6598019Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6598324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6598414Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6598419Z 2025-08-14T21:48:02.6598529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6598751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6598834Z return mod(**inputs) 2025-08-14T21:48:02.6599092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6599184Z return func(*args, **kwargs) 2025-08-14T21:48:02.6599433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6599534Z return func(*args, **kwargs) 2025-08-14T21:48:02.6599793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6599870Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6600143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6600213Z outputs = self.layoutlm( 2025-08-14T21:48:02.6600484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6600555Z return func(*args, **kwargs) 2025-08-14T21:48:02.6600798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6600876Z return func(*args, **kwargs) 2025-08-14T21:48:02.6601100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6601177Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6601477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6601554Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6601856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6601925Z return func(*args, **kwargs) 2025-08-14T21:48:02.6602181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6602263Z return func(*args, **kwargs) 2025-08-14T21:48:02.6602512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6602582Z return func(*args, **kwargs) 2025-08-14T21:48:02.6602674Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6602908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6602991Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6603280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6603354Z layer_outputs = layer_module( 2025-08-14T21:48:02.6603596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6603699Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6603947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6604026Z return func(*args, **kwargs) 2025-08-14T21:48:02.6604271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6604348Z return func(*args, **kwargs) 2025-08-14T21:48:02.6604596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6604662Z return func(*args, **kwargs) 2025-08-14T21:48:02.6604932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6605019Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6605284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6605384Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6605694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6605827Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6606090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6606173Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6606185Z 2025-08-14T21:48:02.6606290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6606491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6606565Z return mod(**inputs) 2025-08-14T21:48:02.6606825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6606898Z return func(*args, **kwargs) 2025-08-14T21:48:02.6607143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6607220Z return func(*args, **kwargs) 2025-08-14T21:48:02.6607444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6607519Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6607789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6607866Z outputs = self.layoutlm( 2025-08-14T21:48:02.6608104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6608179Z return func(*args, **kwargs) 2025-08-14T21:48:02.6608431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6608504Z return func(*args, **kwargs) 2025-08-14T21:48:02.6609020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6609139Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6609421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6609507Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6609755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6609834Z return func(*args, **kwargs) 2025-08-14T21:48:02.6610083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6610155Z return func(*args, **kwargs) 2025-08-14T21:48:02.6610471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6610544Z return func(*args, **kwargs) 2025-08-14T21:48:02.6610626Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6610858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6610935Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6611236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6611310Z layer_outputs = layer_module( 2025-08-14T21:48:02.6611539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6611631Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6611879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6611952Z return func(*args, **kwargs) 2025-08-14T21:48:02.6612269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6612344Z return func(*args, **kwargs) 2025-08-14T21:48:02.6612601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6612670Z return func(*args, **kwargs) 2025-08-14T21:48:02.6612970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6613067Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6613345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6613462Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6613774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6613904Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6614186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6614308Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6614531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6614616Z return self.act(input) 2025-08-14T21:48:02.6614620Z 2025-08-14T21:48:02.6614733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6614953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6615030Z return mod(**inputs) 2025-08-14T21:48:02.6615283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6615370Z return func(*args, **kwargs) 2025-08-14T21:48:02.6615621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6615696Z return func(*args, **kwargs) 2025-08-14T21:48:02.6615931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6616013Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6616308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6616380Z outputs = self.layoutlm( 2025-08-14T21:48:02.6616611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6616689Z return func(*args, **kwargs) 2025-08-14T21:48:02.6616931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6617004Z return func(*args, **kwargs) 2025-08-14T21:48:02.6617209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6617282Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6617541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6617613Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6617844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6617917Z return func(*args, **kwargs) 2025-08-14T21:48:02.6618144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6618217Z return func(*args, **kwargs) 2025-08-14T21:48:02.6618459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6618542Z return func(*args, **kwargs) 2025-08-14T21:48:02.6618625Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6618832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6618904Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6619169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6619237Z layer_outputs = layer_module( 2025-08-14T21:48:02.6619454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6619548Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6619779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6619856Z return func(*args, **kwargs) 2025-08-14T21:48:02.6620090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6620155Z return func(*args, **kwargs) 2025-08-14T21:48:02.6620394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6620457Z return func(*args, **kwargs) 2025-08-14T21:48:02.6620730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6620815Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6621077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6621161Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6621465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6621609Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6621878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6621959Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6621963Z 2025-08-14T21:48:02.6622075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6622277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6622343Z return mod(**inputs) 2025-08-14T21:48:02.6622599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6622666Z return func(*args, **kwargs) 2025-08-14T21:48:02.6622916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6622983Z return func(*args, **kwargs) 2025-08-14T21:48:02.6623188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6623267Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6623519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6623594Z outputs = self.layoutlm( 2025-08-14T21:48:02.6623818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6623883Z return func(*args, **kwargs) 2025-08-14T21:48:02.6624119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6624186Z return func(*args, **kwargs) 2025-08-14T21:48:02.6624404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6624509Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6624763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6624840Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6625066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6625132Z return func(*args, **kwargs) 2025-08-14T21:48:02.6625367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6625431Z return func(*args, **kwargs) 2025-08-14T21:48:02.6625674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6625749Z return func(*args, **kwargs) 2025-08-14T21:48:02.6625827Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6626042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6626114Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6626370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6626447Z layer_outputs = layer_module( 2025-08-14T21:48:02.6626658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6626734Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6626972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6627037Z return func(*args, **kwargs) 2025-08-14T21:48:02.6627270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6627339Z return func(*args, **kwargs) 2025-08-14T21:48:02.6627565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6627637Z return func(*args, **kwargs) 2025-08-14T21:48:02.6627894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6627982Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6628215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6628281Z return func(*args, **kwargs) 2025-08-14T21:48:02.6628522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6628621Z return func(*args, **kwargs) 2025-08-14T21:48:02.6628854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6628927Z return func(*args, **kwargs) 2025-08-14T21:48:02.6629187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6629263Z self_outputs = self.self( 2025-08-14T21:48:02.6629514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6629582Z return func(*args, **kwargs) 2025-08-14T21:48:02.6629838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6629908Z return func(*args, **kwargs) 2025-08-14T21:48:02.6630159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6630235Z return func(*args, **kwargs) 2025-08-14T21:48:02.6630554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6630715Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6630720Z 2025-08-14T21:48:02.6630828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6631035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6631112Z return mod(**inputs) 2025-08-14T21:48:02.6631359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6631435Z return func(*args, **kwargs) 2025-08-14T21:48:02.6631694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6631768Z return func(*args, **kwargs) 2025-08-14T21:48:02.6632004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6632084Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6632364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6632444Z outputs = self.layoutlm( 2025-08-14T21:48:02.6632691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6632768Z return func(*args, **kwargs) 2025-08-14T21:48:02.6633015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6633084Z return func(*args, **kwargs) 2025-08-14T21:48:02.6633315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6633394Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6633674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6633759Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6634006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6634084Z return func(*args, **kwargs) 2025-08-14T21:48:02.6634329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6634398Z return func(*args, **kwargs) 2025-08-14T21:48:02.6634650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6634721Z return func(*args, **kwargs) 2025-08-14T21:48:02.6634823Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6635061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6635143Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6635446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6635521Z layer_outputs = layer_module( 2025-08-14T21:48:02.6635833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6635934Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6636209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6636281Z return func(*args, **kwargs) 2025-08-14T21:48:02.6636568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6636641Z return func(*args, **kwargs) 2025-08-14T21:48:02.6636953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6637031Z return func(*args, **kwargs) 2025-08-14T21:48:02.6637329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6637427Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6637736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6637828Z return func(*args, **kwargs) 2025-08-14T21:48:02.6638105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6638176Z return func(*args, **kwargs) 2025-08-14T21:48:02.6638467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6638543Z return func(*args, **kwargs) 2025-08-14T21:48:02.6638854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6638939Z self_outputs = self.self( 2025-08-14T21:48:02.6639226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6639306Z return func(*args, **kwargs) 2025-08-14T21:48:02.6639578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6639648Z return func(*args, **kwargs) 2025-08-14T21:48:02.6639924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6639994Z return func(*args, **kwargs) 2025-08-14T21:48:02.6640308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6640463Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6640468Z 2025-08-14T21:48:02.6640576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6640793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6640863Z return mod(**inputs) 2025-08-14T21:48:02.6641134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6641212Z return func(*args, **kwargs) 2025-08-14T21:48:02.6641485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6641561Z return func(*args, **kwargs) 2025-08-14T21:48:02.6641802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6641906Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6642209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6642281Z outputs = self.layoutlm( 2025-08-14T21:48:02.6642551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6642629Z return func(*args, **kwargs) 2025-08-14T21:48:02.6642898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6642974Z return func(*args, **kwargs) 2025-08-14T21:48:02.6643208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6643286Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6643574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6643694Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6643971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6644052Z return func(*args, **kwargs) 2025-08-14T21:48:02.6644319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6644397Z return func(*args, **kwargs) 2025-08-14T21:48:02.6644646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6644711Z return func(*args, **kwargs) 2025-08-14T21:48:02.6644796Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6645021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6645098Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6645383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6645458Z layer_outputs = layer_module( 2025-08-14T21:48:02.6645696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6645787Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6646019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6646092Z return func(*args, **kwargs) 2025-08-14T21:48:02.6646325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6646399Z return func(*args, **kwargs) 2025-08-14T21:48:02.6646630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6646698Z return func(*args, **kwargs) 2025-08-14T21:48:02.6646966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6647047Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6647279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6647352Z return func(*args, **kwargs) 2025-08-14T21:48:02.6647582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6647654Z return func(*args, **kwargs) 2025-08-14T21:48:02.6647886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6647953Z return func(*args, **kwargs) 2025-08-14T21:48:02.6648241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6648316Z self_outputs = self.self( 2025-08-14T21:48:02.6648549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6648622Z return func(*args, **kwargs) 2025-08-14T21:48:02.6648853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6648926Z return func(*args, **kwargs) 2025-08-14T21:48:02.6649158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6649223Z return func(*args, **kwargs) 2025-08-14T21:48:02.6649503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6649644Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6649672Z 2025-08-14T21:48:02.6649796Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6649876Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6649977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6650176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6650241Z return mod(**inputs) 2025-08-14T21:48:02.6650468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6650545Z return func(*args, **kwargs) 2025-08-14T21:48:02.6650778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6650852Z return func(*args, **kwargs) 2025-08-14T21:48:02.6651083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6651162Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6651437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6651506Z outputs = self.layoutlm( 2025-08-14T21:48:02.6651738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6651812Z return func(*args, **kwargs) 2025-08-14T21:48:02.6652046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6652122Z return func(*args, **kwargs) 2025-08-14T21:48:02.6652332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6652411Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6652685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6652758Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6652993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6653066Z return func(*args, **kwargs) 2025-08-14T21:48:02.6653299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6653371Z return func(*args, **kwargs) 2025-08-14T21:48:02.6653604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6653669Z return func(*args, **kwargs) 2025-08-14T21:48:02.6653754Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6653969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6654062Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6654335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6654406Z layer_outputs = layer_module( 2025-08-14T21:48:02.6654631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6654710Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6654943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6655016Z return func(*args, **kwargs) 2025-08-14T21:48:02.6655250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6655323Z return func(*args, **kwargs) 2025-08-14T21:48:02.6655556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6655636Z return func(*args, **kwargs) 2025-08-14T21:48:02.6655940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6656024Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6656256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6656329Z return func(*args, **kwargs) 2025-08-14T21:48:02.6656556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6656628Z return func(*args, **kwargs) 2025-08-14T21:48:02.6656856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6656937Z return func(*args, **kwargs) 2025-08-14T21:48:02.6657212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6657347Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6657615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6657709Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6657712Z 2025-08-14T21:48:02.6657816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6658020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6658086Z return mod(**inputs) 2025-08-14T21:48:02.6658322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6658396Z return func(*args, **kwargs) 2025-08-14T21:48:02.6658628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6658704Z return func(*args, **kwargs) 2025-08-14T21:48:02.6658917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6658991Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6659262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6659342Z outputs = self.layoutlm( 2025-08-14T21:48:02.6659570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6659643Z return func(*args, **kwargs) 2025-08-14T21:48:02.6659874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6659945Z return func(*args, **kwargs) 2025-08-14T21:48:02.6660168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6660242Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6660506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6660577Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6660803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6660875Z return func(*args, **kwargs) 2025-08-14T21:48:02.6661099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6661169Z return func(*args, **kwargs) 2025-08-14T21:48:02.6661402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6661469Z return func(*args, **kwargs) 2025-08-14T21:48:02.6661552Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6661794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6661867Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6662134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6662204Z layer_outputs = layer_module( 2025-08-14T21:48:02.6662427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6662508Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6662743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6662821Z return func(*args, **kwargs) 2025-08-14T21:48:02.6663067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6663136Z return func(*args, **kwargs) 2025-08-14T21:48:02.6663379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6663444Z return func(*args, **kwargs) 2025-08-14T21:48:02.6663715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6663801Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6664061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6664146Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6664449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6664582Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6664851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6664934Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6664937Z 2025-08-14T21:48:02.6665048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6665246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6665319Z return mod(**inputs) 2025-08-14T21:48:02.6665568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6665635Z return func(*args, **kwargs) 2025-08-14T21:48:02.6665874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6665940Z return func(*args, **kwargs) 2025-08-14T21:48:02.6666167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6666252Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6666509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6666587Z outputs = self.layoutlm( 2025-08-14T21:48:02.6666824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6666896Z return func(*args, **kwargs) 2025-08-14T21:48:02.6667138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6667207Z return func(*args, **kwargs) 2025-08-14T21:48:02.6667423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6667522Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6667810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6667911Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6668145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6668211Z return func(*args, **kwargs) 2025-08-14T21:48:02.6668449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6668515Z return func(*args, **kwargs) 2025-08-14T21:48:02.6668745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6668818Z return func(*args, **kwargs) 2025-08-14T21:48:02.6668908Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6669133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6669211Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6669478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6669556Z layer_outputs = layer_module( 2025-08-14T21:48:02.6669771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6669849Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6670089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6670156Z return func(*args, **kwargs) 2025-08-14T21:48:02.6670396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6670464Z return func(*args, **kwargs) 2025-08-14T21:48:02.6670700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6670777Z return func(*args, **kwargs) 2025-08-14T21:48:02.6671039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6671131Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6671389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6671464Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6671766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6671886Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6672151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6672297Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6672508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6672586Z return self.act(input) 2025-08-14T21:48:02.6672589Z 2025-08-14T21:48:02.6672692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6672891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6672962Z return mod(**inputs) 2025-08-14T21:48:02.6673198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6673273Z return func(*args, **kwargs) 2025-08-14T21:48:02.6673510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6673578Z return func(*args, **kwargs) 2025-08-14T21:48:02.6673813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6673904Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6674166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6674244Z outputs = self.layoutlm( 2025-08-14T21:48:02.6674477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6674549Z return func(*args, **kwargs) 2025-08-14T21:48:02.6674782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6674848Z return func(*args, **kwargs) 2025-08-14T21:48:02.6675079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6675156Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6675420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6675499Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6675829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6675908Z return func(*args, **kwargs) 2025-08-14T21:48:02.6676140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6676207Z return func(*args, **kwargs) 2025-08-14T21:48:02.6676454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6676524Z return func(*args, **kwargs) 2025-08-14T21:48:02.6676610Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6676844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6676926Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6677218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6677290Z layer_outputs = layer_module( 2025-08-14T21:48:02.6677508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6677595Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6677831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6677906Z return func(*args, **kwargs) 2025-08-14T21:48:02.6678139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6678233Z return func(*args, **kwargs) 2025-08-14T21:48:02.6678478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6678547Z return func(*args, **kwargs) 2025-08-14T21:48:02.6678811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6678907Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6679163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6679244Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6679537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6679671Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6679945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6680072Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6680077Z 2025-08-14T21:48:02.6680192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6680400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6680467Z return mod(**inputs) 2025-08-14T21:48:02.6680708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6680779Z return func(*args, **kwargs) 2025-08-14T21:48:02.6681009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6681085Z return func(*args, **kwargs) 2025-08-14T21:48:02.6681309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6681391Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6681650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6681717Z outputs = self.layoutlm( 2025-08-14T21:48:02.6681950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6682014Z return func(*args, **kwargs) 2025-08-14T21:48:02.6682241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6682314Z return func(*args, **kwargs) 2025-08-14T21:48:02.6682520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6682598Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6682858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6682934Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6683182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6683248Z return func(*args, **kwargs) 2025-08-14T21:48:02.6683482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6683554Z return func(*args, **kwargs) 2025-08-14T21:48:02.6683788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6683862Z return func(*args, **kwargs) 2025-08-14T21:48:02.6683938Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6684151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6684256Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6684522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6684596Z layer_outputs = layer_module( 2025-08-14T21:48:02.6684820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6684898Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6685138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6685205Z return func(*args, **kwargs) 2025-08-14T21:48:02.6685445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6685516Z return func(*args, **kwargs) 2025-08-14T21:48:02.6685741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6685814Z return func(*args, **kwargs) 2025-08-14T21:48:02.6686104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6686189Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6686421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6686487Z return func(*args, **kwargs) 2025-08-14T21:48:02.6686716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6686791Z return func(*args, **kwargs) 2025-08-14T21:48:02.6687024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6687115Z return func(*args, **kwargs) 2025-08-14T21:48:02.6687378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6687451Z self_outputs = self.self( 2025-08-14T21:48:02.6687690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6687756Z return func(*args, **kwargs) 2025-08-14T21:48:02.6687984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6688059Z return func(*args, **kwargs) 2025-08-14T21:48:02.6688289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6688360Z return func(*args, **kwargs) 2025-08-14T21:48:02.6688662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6688819Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6688824Z 2025-08-14T21:48:02.6688945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6689155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6689232Z return mod(**inputs) 2025-08-14T21:48:02.6689477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6689546Z return func(*args, **kwargs) 2025-08-14T21:48:02.6689796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6689865Z return func(*args, **kwargs) 2025-08-14T21:48:02.6690087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6690173Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6690468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6690546Z outputs = self.layoutlm( 2025-08-14T21:48:02.6690777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6690842Z return func(*args, **kwargs) 2025-08-14T21:48:02.6691078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6691142Z return func(*args, **kwargs) 2025-08-14T21:48:02.6691353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6691434Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6691709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6691793Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6692040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6692186Z return func(*args, **kwargs) 2025-08-14T21:48:02.6692431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6692498Z return func(*args, **kwargs) 2025-08-14T21:48:02.6692733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6692809Z return func(*args, **kwargs) 2025-08-14T21:48:02.6692890Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6693123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6693203Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6693528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6693616Z layer_outputs = layer_module( 2025-08-14T21:48:02.6693849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6693940Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6694189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6694260Z return func(*args, **kwargs) 2025-08-14T21:48:02.6694516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6694586Z return func(*args, **kwargs) 2025-08-14T21:48:02.6694833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6694914Z return func(*args, **kwargs) 2025-08-14T21:48:02.6695195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6695303Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6695542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6695609Z return func(*args, **kwargs) 2025-08-14T21:48:02.6695853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6695921Z return func(*args, **kwargs) 2025-08-14T21:48:02.6696154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6696227Z return func(*args, **kwargs) 2025-08-14T21:48:02.6696528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6696628Z self_outputs = self.self( 2025-08-14T21:48:02.6696877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6696950Z return func(*args, **kwargs) 2025-08-14T21:48:02.6697205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6697276Z return func(*args, **kwargs) 2025-08-14T21:48:02.6697520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6697598Z return func(*args, **kwargs) 2025-08-14T21:48:02.6697876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6698027Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6698031Z 2025-08-14T21:48:02.6698143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6698353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6698466Z return mod(**inputs) 2025-08-14T21:48:02.6698713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6698790Z return func(*args, **kwargs) 2025-08-14T21:48:02.6699035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6699107Z return func(*args, **kwargs) 2025-08-14T21:48:02.6699337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6699416Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6699713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6699800Z outputs = self.layoutlm( 2025-08-14T21:48:02.6700046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6700126Z return func(*args, **kwargs) 2025-08-14T21:48:02.6700373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6700441Z return func(*args, **kwargs) 2025-08-14T21:48:02.6700661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6700734Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6700996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6701076Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6701311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6701392Z return func(*args, **kwargs) 2025-08-14T21:48:02.6701647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6701718Z return func(*args, **kwargs) 2025-08-14T21:48:02.6701975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6702044Z return func(*args, **kwargs) 2025-08-14T21:48:02.6702126Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6702361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6702438Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6702724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6702799Z layer_outputs = layer_module( 2025-08-14T21:48:02.6703052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6703149Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6703398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6703474Z return func(*args, **kwargs) 2025-08-14T21:48:02.6703721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6703792Z return func(*args, **kwargs) 2025-08-14T21:48:02.6704045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6704112Z return func(*args, **kwargs) 2025-08-14T21:48:02.6704381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6704473Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6704724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6704818Z return func(*args, **kwargs) 2025-08-14T21:48:02.6705054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6705122Z return func(*args, **kwargs) 2025-08-14T21:48:02.6705365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6705430Z return func(*args, **kwargs) 2025-08-14T21:48:02.6705694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6705773Z self_outputs = self.self( 2025-08-14T21:48:02.6706025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6706102Z return func(*args, **kwargs) 2025-08-14T21:48:02.6706339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6706404Z return func(*args, **kwargs) 2025-08-14T21:48:02.6706645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6706709Z return func(*args, **kwargs) 2025-08-14T21:48:02.6706990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6707144Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6707147Z 2025-08-14T21:48:02.6707232Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6707325Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6707436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6707647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6707728Z return mod(**inputs) 2025-08-14T21:48:02.6707978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6708056Z return func(*args, **kwargs) 2025-08-14T21:48:02.6708304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6708374Z return func(*args, **kwargs) 2025-08-14T21:48:02.6708610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6708938Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6709359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6709517Z outputs = self.layoutlm( 2025-08-14T21:48:02.6709765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6709850Z return func(*args, **kwargs) 2025-08-14T21:48:02.6710099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6710169Z return func(*args, **kwargs) 2025-08-14T21:48:02.6710402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6710483Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6710785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6710870Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6711117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6711199Z return func(*args, **kwargs) 2025-08-14T21:48:02.6711507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6711581Z return func(*args, **kwargs) 2025-08-14T21:48:02.6711839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6711909Z return func(*args, **kwargs) 2025-08-14T21:48:02.6711990Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6712224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6712301Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6712601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6712705Z layer_outputs = layer_module( 2025-08-14T21:48:02.6712942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6713037Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6713288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6713365Z return func(*args, **kwargs) 2025-08-14T21:48:02.6713612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6713681Z return func(*args, **kwargs) 2025-08-14T21:48:02.6713935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6714005Z return func(*args, **kwargs) 2025-08-14T21:48:02.6714304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6714403Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6714649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6714730Z return func(*args, **kwargs) 2025-08-14T21:48:02.6714976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6715046Z return func(*args, **kwargs) 2025-08-14T21:48:02.6715301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6715373Z return func(*args, **kwargs) 2025-08-14T21:48:02.6715714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6715866Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6716168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6716296Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6716302Z 2025-08-14T21:48:02.6716419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6716636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6716720Z return mod(**inputs) 2025-08-14T21:48:02.6716984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6717063Z return func(*args, **kwargs) 2025-08-14T21:48:02.6717308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6717378Z return func(*args, **kwargs) 2025-08-14T21:48:02.6717612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6717693Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6717989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6718107Z outputs = self.layoutlm( 2025-08-14T21:48:02.6718353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6718432Z return func(*args, **kwargs) 2025-08-14T21:48:02.6718677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6718747Z return func(*args, **kwargs) 2025-08-14T21:48:02.6718979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6719057Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6719352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6719440Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6719690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6719767Z return func(*args, **kwargs) 2025-08-14T21:48:02.6720014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6720083Z return func(*args, **kwargs) 2025-08-14T21:48:02.6720337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6720407Z return func(*args, **kwargs) 2025-08-14T21:48:02.6720487Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6720717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6720795Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6721080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6721158Z layer_outputs = layer_module( 2025-08-14T21:48:02.6721387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6721476Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6721724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6721799Z return func(*args, **kwargs) 2025-08-14T21:48:02.6722043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6722114Z return func(*args, **kwargs) 2025-08-14T21:48:02.6722366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6722455Z return func(*args, **kwargs) 2025-08-14T21:48:02.6722740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6722836Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6723098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6723182Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6723492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6723624Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6723916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6724006Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6724011Z 2025-08-14T21:48:02.6724129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6724381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6724453Z return mod(**inputs) 2025-08-14T21:48:02.6724717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6724789Z return func(*args, **kwargs) 2025-08-14T21:48:02.6725034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6725114Z return func(*args, **kwargs) 2025-08-14T21:48:02.6725348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6725431Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6725714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6725787Z outputs = self.layoutlm( 2025-08-14T21:48:02.6726039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6726104Z return func(*args, **kwargs) 2025-08-14T21:48:02.6726329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6726400Z return func(*args, **kwargs) 2025-08-14T21:48:02.6726609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6726691Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6726953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6727026Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6727266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6727334Z return func(*args, **kwargs) 2025-08-14T21:48:02.6727566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6727638Z return func(*args, **kwargs) 2025-08-14T21:48:02.6727877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6727952Z return func(*args, **kwargs) 2025-08-14T21:48:02.6728035Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6728246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6728325Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6728587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6728691Z layer_outputs = layer_module( 2025-08-14T21:48:02.6728914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6728994Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6729243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6729307Z return func(*args, **kwargs) 2025-08-14T21:48:02.6729538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6729610Z return func(*args, **kwargs) 2025-08-14T21:48:02.6729839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6729909Z return func(*args, **kwargs) 2025-08-14T21:48:02.6730169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6730254Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6730542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6730618Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6730908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6731037Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6731293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6731413Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6731633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6731706Z return self.act(input) 2025-08-14T21:48:02.6731709Z 2025-08-14T21:48:02.6731819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6732022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6732095Z return mod(**inputs) 2025-08-14T21:48:02.6732327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6732394Z return func(*args, **kwargs) 2025-08-14T21:48:02.6732631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6732695Z return func(*args, **kwargs) 2025-08-14T21:48:02.6732902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6732982Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6733243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6733323Z outputs = self.layoutlm( 2025-08-14T21:48:02.6733565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6733633Z return func(*args, **kwargs) 2025-08-14T21:48:02.6733887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6733953Z return func(*args, **kwargs) 2025-08-14T21:48:02.6734159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6734238Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6734495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6734572Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6734820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6734888Z return func(*args, **kwargs) 2025-08-14T21:48:02.6735119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6735184Z return func(*args, **kwargs) 2025-08-14T21:48:02.6735417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6735481Z return func(*args, **kwargs) 2025-08-14T21:48:02.6735555Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6735768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6735837Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6736094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6736170Z layer_outputs = layer_module( 2025-08-14T21:48:02.6736415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6736499Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6736727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6736791Z return func(*args, **kwargs) 2025-08-14T21:48:02.6737026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6737094Z return func(*args, **kwargs) 2025-08-14T21:48:02.6737324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6737397Z return func(*args, **kwargs) 2025-08-14T21:48:02.6737673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6737771Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6738028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6738103Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6738407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6738540Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6738809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6738903Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6738906Z 2025-08-14T21:48:02.6739294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6739499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6739566Z return mod(**inputs) 2025-08-14T21:48:02.6739795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6739871Z return func(*args, **kwargs) 2025-08-14T21:48:02.6740099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6740175Z return func(*args, **kwargs) 2025-08-14T21:48:02.6740381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6740454Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6740720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6740791Z outputs = self.layoutlm( 2025-08-14T21:48:02.6741051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6741122Z return func(*args, **kwargs) 2025-08-14T21:48:02.6741349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6741419Z return func(*args, **kwargs) 2025-08-14T21:48:02.6741627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6741699Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6741962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6742033Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6742269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6742335Z return func(*args, **kwargs) 2025-08-14T21:48:02.6742589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6742682Z return func(*args, **kwargs) 2025-08-14T21:48:02.6742908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6742972Z return func(*args, **kwargs) 2025-08-14T21:48:02.6743055Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6743263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6743342Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6743598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6743683Z layer_outputs = layer_module( 2025-08-14T21:48:02.6743902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6743982Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6744213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6744289Z return func(*args, **kwargs) 2025-08-14T21:48:02.6744518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6744595Z return func(*args, **kwargs) 2025-08-14T21:48:02.6744826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6744892Z return func(*args, **kwargs) 2025-08-14T21:48:02.6745158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6745244Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6745486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6745563Z return func(*args, **kwargs) 2025-08-14T21:48:02.6745786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6745860Z return func(*args, **kwargs) 2025-08-14T21:48:02.6746084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6746151Z return func(*args, **kwargs) 2025-08-14T21:48:02.6746412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6746483Z self_outputs = self.self( 2025-08-14T21:48:02.6746716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6746799Z return func(*args, **kwargs) 2025-08-14T21:48:02.6747023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6747094Z return func(*args, **kwargs) 2025-08-14T21:48:02.6747313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6747375Z return func(*args, **kwargs) 2025-08-14T21:48:02.6747633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6747772Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6747776Z 2025-08-14T21:48:02.6747881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6748071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6748137Z return mod(**inputs) 2025-08-14T21:48:02.6748365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6748467Z return func(*args, **kwargs) 2025-08-14T21:48:02.6748692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6748762Z return func(*args, **kwargs) 2025-08-14T21:48:02.6748963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6749039Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6749289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6749355Z outputs = self.layoutlm( 2025-08-14T21:48:02.6749601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6749666Z return func(*args, **kwargs) 2025-08-14T21:48:02.6749902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6749964Z return func(*args, **kwargs) 2025-08-14T21:48:02.6750165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6750242Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6750493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6750562Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6750795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6750858Z return func(*args, **kwargs) 2025-08-14T21:48:02.6751094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6751159Z return func(*args, **kwargs) 2025-08-14T21:48:02.6751390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6751461Z return func(*args, **kwargs) 2025-08-14T21:48:02.6751536Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6751744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6751822Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6752080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6752157Z layer_outputs = layer_module( 2025-08-14T21:48:02.6752372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6752449Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6752709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6752778Z return func(*args, **kwargs) 2025-08-14T21:48:02.6753009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6753082Z return func(*args, **kwargs) 2025-08-14T21:48:02.6753325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6753400Z return func(*args, **kwargs) 2025-08-14T21:48:02.6753679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6753766Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6754023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6754097Z return func(*args, **kwargs) 2025-08-14T21:48:02.6754392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6754464Z return func(*args, **kwargs) 2025-08-14T21:48:02.6754715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6754792Z return func(*args, **kwargs) 2025-08-14T21:48:02.6755071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6755145Z self_outputs = self.self( 2025-08-14T21:48:02.6755402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6755471Z return func(*args, **kwargs) 2025-08-14T21:48:02.6755833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6755914Z return func(*args, **kwargs) 2025-08-14T21:48:02.6756173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6756256Z return func(*args, **kwargs) 2025-08-14T21:48:02.6756546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6756697Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6756709Z 2025-08-14T21:48:02.6756826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6757039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6757113Z return mod(**inputs) 2025-08-14T21:48:02.6757344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6757411Z return func(*args, **kwargs) 2025-08-14T21:48:02.6757663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6757728Z return func(*args, **kwargs) 2025-08-14T21:48:02.6757938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6758010Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6758259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6758336Z outputs = self.layoutlm( 2025-08-14T21:48:02.6758560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6758623Z return func(*args, **kwargs) 2025-08-14T21:48:02.6758854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6758941Z return func(*args, **kwargs) 2025-08-14T21:48:02.6759154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6759225Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6759479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6759558Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6759784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6759848Z return func(*args, **kwargs) 2025-08-14T21:48:02.6760081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6760145Z return func(*args, **kwargs) 2025-08-14T21:48:02.6760380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6760463Z return func(*args, **kwargs) 2025-08-14T21:48:02.6760552Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6760762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6760832Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6761082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6761157Z layer_outputs = layer_module( 2025-08-14T21:48:02.6761367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6761449Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6761694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6761761Z return func(*args, **kwargs) 2025-08-14T21:48:02.6762001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6762065Z return func(*args, **kwargs) 2025-08-14T21:48:02.6762299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6762363Z return func(*args, **kwargs) 2025-08-14T21:48:02.6762621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6762713Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6762941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6763006Z return func(*args, **kwargs) 2025-08-14T21:48:02.6763244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6763310Z return func(*args, **kwargs) 2025-08-14T21:48:02.6763550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6763614Z return func(*args, **kwargs) 2025-08-14T21:48:02.6763873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6763949Z self_outputs = self.self( 2025-08-14T21:48:02.6764176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6764239Z return func(*args, **kwargs) 2025-08-14T21:48:02.6764472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6764538Z return func(*args, **kwargs) 2025-08-14T21:48:02.6764799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6764867Z return func(*args, **kwargs) 2025-08-14T21:48:02.6765120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6765268Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6765272Z 2025-08-14T21:48:02.6765350Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6765430Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6765531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6765720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6765791Z return mod(**inputs) 2025-08-14T21:48:02.6766020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6766088Z return func(*args, **kwargs) 2025-08-14T21:48:02.6766372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6766439Z return func(*args, **kwargs) 2025-08-14T21:48:02.6766657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6766731Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6766989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6767063Z outputs = self.layoutlm( 2025-08-14T21:48:02.6767291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6767355Z return func(*args, **kwargs) 2025-08-14T21:48:02.6767612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6767680Z return func(*args, **kwargs) 2025-08-14T21:48:02.6767899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6767971Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6768229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6768307Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6768535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6768600Z return func(*args, **kwargs) 2025-08-14T21:48:02.6768838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6768905Z return func(*args, **kwargs) 2025-08-14T21:48:02.6769142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6769211Z return func(*args, **kwargs) 2025-08-14T21:48:02.6769286Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6769503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6769574Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6769836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6769912Z layer_outputs = layer_module( 2025-08-14T21:48:02.6770125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6770210Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6770445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6770530Z return func(*args, **kwargs) 2025-08-14T21:48:02.6770770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6770835Z return func(*args, **kwargs) 2025-08-14T21:48:02.6771062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6771132Z return func(*args, **kwargs) 2025-08-14T21:48:02.6771388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6771475Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6771704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6771769Z return func(*args, **kwargs) 2025-08-14T21:48:02.6772005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6772105Z return func(*args, **kwargs) 2025-08-14T21:48:02.6772360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6772426Z return func(*args, **kwargs) 2025-08-14T21:48:02.6772678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6772813Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6773076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6773161Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6773171Z 2025-08-14T21:48:02.6773276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6773493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6773568Z return mod(**inputs) 2025-08-14T21:48:02.6773805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6773870Z return func(*args, **kwargs) 2025-08-14T21:48:02.6774111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6774178Z return func(*args, **kwargs) 2025-08-14T21:48:02.6774401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6774477Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6774748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6774827Z outputs = self.layoutlm( 2025-08-14T21:48:02.6775070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6775140Z return func(*args, **kwargs) 2025-08-14T21:48:02.6775388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6775454Z return func(*args, **kwargs) 2025-08-14T21:48:02.6775679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6775755Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6776021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6776100Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6776337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6776405Z return func(*args, **kwargs) 2025-08-14T21:48:02.6776666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6776734Z return func(*args, **kwargs) 2025-08-14T21:48:02.6776975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6777041Z return func(*args, **kwargs) 2025-08-14T21:48:02.6777118Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6777335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6777408Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6777667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6777747Z layer_outputs = layer_module( 2025-08-14T21:48:02.6777970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6778055Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6778320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6778387Z return func(*args, **kwargs) 2025-08-14T21:48:02.6778625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6778688Z return func(*args, **kwargs) 2025-08-14T21:48:02.6778916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6778987Z return func(*args, **kwargs) 2025-08-14T21:48:02.6779250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6779356Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6779605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6779681Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6779976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6780095Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6780353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6780437Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6780440Z 2025-08-14T21:48:02.6780540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6780742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6780808Z return mod(**inputs) 2025-08-14T21:48:02.6781038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6781116Z return func(*args, **kwargs) 2025-08-14T21:48:02.6781344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6781417Z return func(*args, **kwargs) 2025-08-14T21:48:02.6781623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6781697Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6781959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6782029Z outputs = self.layoutlm( 2025-08-14T21:48:02.6782264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6782331Z return func(*args, **kwargs) 2025-08-14T21:48:02.6782575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6782651Z return func(*args, **kwargs) 2025-08-14T21:48:02.6782858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6782930Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6783194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6783266Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6783504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6783569Z return func(*args, **kwargs) 2025-08-14T21:48:02.6783795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6783871Z return func(*args, **kwargs) 2025-08-14T21:48:02.6784115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6784199Z return func(*args, **kwargs) 2025-08-14T21:48:02.6784285Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6784496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6784577Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6784838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6784910Z layer_outputs = layer_module( 2025-08-14T21:48:02.6785134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6785249Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6785491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6785568Z return func(*args, **kwargs) 2025-08-14T21:48:02.6785800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6785873Z return func(*args, **kwargs) 2025-08-14T21:48:02.6786105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6786171Z return func(*args, **kwargs) 2025-08-14T21:48:02.6786441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6786528Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6786794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6786870Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6787167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6787297Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6787558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6787674Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6787886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6787955Z return self.act(input) 2025-08-14T21:48:02.6787959Z 2025-08-14T21:48:02.6788068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6788266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6788335Z return mod(**inputs) 2025-08-14T21:48:02.6788624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6788693Z return func(*args, **kwargs) 2025-08-14T21:48:02.6788933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6789000Z return func(*args, **kwargs) 2025-08-14T21:48:02.6789211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6789291Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6789552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6789622Z outputs = self.layoutlm( 2025-08-14T21:48:02.6789863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6789930Z return func(*args, **kwargs) 2025-08-14T21:48:02.6790184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6790268Z return func(*args, **kwargs) 2025-08-14T21:48:02.6790482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6790563Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6790829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6790901Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6791141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6791207Z return func(*args, **kwargs) 2025-08-14T21:48:02.6791478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6791552Z return func(*args, **kwargs) 2025-08-14T21:48:02.6791800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6791876Z return func(*args, **kwargs) 2025-08-14T21:48:02.6791956Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6792179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6792264Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6792564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6792646Z layer_outputs = layer_module( 2025-08-14T21:48:02.6792877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6792961Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6793225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6793301Z return func(*args, **kwargs) 2025-08-14T21:48:02.6793551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6793633Z return func(*args, **kwargs) 2025-08-14T21:48:02.6793887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6793967Z return func(*args, **kwargs) 2025-08-14T21:48:02.6794268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6794360Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6794648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6794750Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6795082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6795227Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6795510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6795682Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6795690Z 2025-08-14T21:48:02.6795808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6796036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6796107Z return mod(**inputs) 2025-08-14T21:48:02.6796367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6796452Z return func(*args, **kwargs) 2025-08-14T21:48:02.6796732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6796825Z return func(*args, **kwargs) 2025-08-14T21:48:02.6797067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6797158Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6797429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6797498Z outputs = self.layoutlm( 2025-08-14T21:48:02.6797734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6797810Z return func(*args, **kwargs) 2025-08-14T21:48:02.6798061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6798131Z return func(*args, **kwargs) 2025-08-14T21:48:02.6798366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6798446Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6798733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6798812Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6799061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6799140Z return func(*args, **kwargs) 2025-08-14T21:48:02.6799392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6799463Z return func(*args, **kwargs) 2025-08-14T21:48:02.6799726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6799799Z return func(*args, **kwargs) 2025-08-14T21:48:02.6799889Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6800115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6800192Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6800496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6800570Z layer_outputs = layer_module( 2025-08-14T21:48:02.6800801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6800895Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6801146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6801245Z return func(*args, **kwargs) 2025-08-14T21:48:02.6801505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6801576Z return func(*args, **kwargs) 2025-08-14T21:48:02.6801830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6801900Z return func(*args, **kwargs) 2025-08-14T21:48:02.6802197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6802284Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6802540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6802617Z return func(*args, **kwargs) 2025-08-14T21:48:02.6802898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6802969Z return func(*args, **kwargs) 2025-08-14T21:48:02.6803275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6803346Z return func(*args, **kwargs) 2025-08-14T21:48:02.6803649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6803724Z self_outputs = self.self( 2025-08-14T21:48:02.6803982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6804059Z return func(*args, **kwargs) 2025-08-14T21:48:02.6804319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6804406Z return func(*args, **kwargs) 2025-08-14T21:48:02.6804676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6804749Z return func(*args, **kwargs) 2025-08-14T21:48:02.6805047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6805204Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6805208Z 2025-08-14T21:48:02.6805317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6805531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6805601Z return mod(**inputs) 2025-08-14T21:48:02.6805866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6805936Z return func(*args, **kwargs) 2025-08-14T21:48:02.6806193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6806271Z return func(*args, **kwargs) 2025-08-14T21:48:02.6806498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6806576Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6806869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6806941Z outputs = self.layoutlm( 2025-08-14T21:48:02.6807202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6807271Z return func(*args, **kwargs) 2025-08-14T21:48:02.6807524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6807603Z return func(*args, **kwargs) 2025-08-14T21:48:02.6807827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6807934Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6808231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6808307Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6808566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6808797Z return func(*args, **kwargs) 2025-08-14T21:48:02.6809255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6809345Z return func(*args, **kwargs) 2025-08-14T21:48:02.6809627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6809703Z return func(*args, **kwargs) 2025-08-14T21:48:02.6809797Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6810133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6810219Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6810507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6810582Z layer_outputs = layer_module( 2025-08-14T21:48:02.6810821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6810902Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6811169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6811241Z return func(*args, **kwargs) 2025-08-14T21:48:02.6811526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6811611Z return func(*args, **kwargs) 2025-08-14T21:48:02.6811868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6811939Z return func(*args, **kwargs) 2025-08-14T21:48:02.6812233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6812321Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6812583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6812662Z return func(*args, **kwargs) 2025-08-14T21:48:02.6812900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6812980Z return func(*args, **kwargs) 2025-08-14T21:48:02.6813224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6813292Z return func(*args, **kwargs) 2025-08-14T21:48:02.6813572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6813642Z self_outputs = self.self( 2025-08-14T21:48:02.6813895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6813961Z return func(*args, **kwargs) 2025-08-14T21:48:02.6814192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6814265Z return func(*args, **kwargs) 2025-08-14T21:48:02.6814499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6814565Z return func(*args, **kwargs) 2025-08-14T21:48:02.6814858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6814998Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6815002Z 2025-08-14T21:48:02.6815111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6815302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6815366Z return mod(**inputs) 2025-08-14T21:48:02.6815603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6815669Z return func(*args, **kwargs) 2025-08-14T21:48:02.6815907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6815973Z return func(*args, **kwargs) 2025-08-14T21:48:02.6816180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6816282Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6816553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6816622Z outputs = self.layoutlm( 2025-08-14T21:48:02.6816858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6816922Z return func(*args, **kwargs) 2025-08-14T21:48:02.6817156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6817220Z return func(*args, **kwargs) 2025-08-14T21:48:02.6817425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6817524Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6817784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6817858Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6818100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6818164Z return func(*args, **kwargs) 2025-08-14T21:48:02.6818399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6818465Z return func(*args, **kwargs) 2025-08-14T21:48:02.6818687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6818760Z return func(*args, **kwargs) 2025-08-14T21:48:02.6818835Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6819041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6819120Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6819378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6819454Z layer_outputs = layer_module( 2025-08-14T21:48:02.6819666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6819742Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6819978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6820043Z return func(*args, **kwargs) 2025-08-14T21:48:02.6820276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6820343Z return func(*args, **kwargs) 2025-08-14T21:48:02.6820588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6820662Z return func(*args, **kwargs) 2025-08-14T21:48:02.6820918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6820999Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6821233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6821297Z return func(*args, **kwargs) 2025-08-14T21:48:02.6821533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6821597Z return func(*args, **kwargs) 2025-08-14T21:48:02.6821828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6821902Z return func(*args, **kwargs) 2025-08-14T21:48:02.6822177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6822265Z self_outputs = self.self( 2025-08-14T21:48:02.6822509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6822573Z return func(*args, **kwargs) 2025-08-14T21:48:02.6822813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6822878Z return func(*args, **kwargs) 2025-08-14T21:48:02.6823109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6823182Z return func(*args, **kwargs) 2025-08-14T21:48:02.6823472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6823623Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6823629Z 2025-08-14T21:48:02.6823707Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6823784Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6823889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6824077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6824139Z return mod(**inputs) 2025-08-14T21:48:02.6824375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6824441Z return func(*args, **kwargs) 2025-08-14T21:48:02.6824674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6824740Z return func(*args, **kwargs) 2025-08-14T21:48:02.6824947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6825029Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6825283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6825349Z outputs = self.layoutlm( 2025-08-14T21:48:02.6825582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6825646Z return func(*args, **kwargs) 2025-08-14T21:48:02.6825880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6825944Z return func(*args, **kwargs) 2025-08-14T21:48:02.6826153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6826234Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6826510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6826587Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6826831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6826899Z return func(*args, **kwargs) 2025-08-14T21:48:02.6827141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6827205Z return func(*args, **kwargs) 2025-08-14T21:48:02.6827440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6827514Z return func(*args, **kwargs) 2025-08-14T21:48:02.6827589Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6827805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6827889Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6828189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6828269Z layer_outputs = layer_module( 2025-08-14T21:48:02.6828488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6828566Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6828812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6828878Z return func(*args, **kwargs) 2025-08-14T21:48:02.6829131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6829218Z return func(*args, **kwargs) 2025-08-14T21:48:02.6829467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6829545Z return func(*args, **kwargs) 2025-08-14T21:48:02.6829824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6829910Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6830167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6830235Z return func(*args, **kwargs) 2025-08-14T21:48:02.6830490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6830560Z return func(*args, **kwargs) 2025-08-14T21:48:02.6830808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6830888Z return func(*args, **kwargs) 2025-08-14T21:48:02.6831165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6831304Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6831590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6831679Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6831682Z 2025-08-14T21:48:02.6831798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6832003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6832075Z return mod(**inputs) 2025-08-14T21:48:02.6832331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6832402Z return func(*args, **kwargs) 2025-08-14T21:48:02.6832690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6832765Z return func(*args, **kwargs) 2025-08-14T21:48:02.6832989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6833076Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6833374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6833447Z outputs = self.layoutlm( 2025-08-14T21:48:02.6833699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6833770Z return func(*args, **kwargs) 2025-08-14T21:48:02.6834028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6834100Z return func(*args, **kwargs) 2025-08-14T21:48:02.6834324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6834457Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6834735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6834814Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6835067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6835137Z return func(*args, **kwargs) 2025-08-14T21:48:02.6835389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6835459Z return func(*args, **kwargs) 2025-08-14T21:48:02.6835852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6835942Z return func(*args, **kwargs) 2025-08-14T21:48:02.6836027Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6836262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6836351Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6836640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6836726Z layer_outputs = layer_module( 2025-08-14T21:48:02.6836962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6837047Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6837307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6837393Z return func(*args, **kwargs) 2025-08-14T21:48:02.6837636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6837710Z return func(*args, **kwargs) 2025-08-14T21:48:02.6837941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6838017Z return func(*args, **kwargs) 2025-08-14T21:48:02.6838281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6838367Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6838637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6838715Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6839018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6839165Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6839444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6839539Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6839542Z 2025-08-14T21:48:02.6839651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6839871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6839936Z return mod(**inputs) 2025-08-14T21:48:02.6840169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6840243Z return func(*args, **kwargs) 2025-08-14T21:48:02.6840477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6840545Z return func(*args, **kwargs) 2025-08-14T21:48:02.6840784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6840881Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6841151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6841221Z outputs = self.layoutlm( 2025-08-14T21:48:02.6841457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6841530Z return func(*args, **kwargs) 2025-08-14T21:48:02.6841762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6841828Z return func(*args, **kwargs) 2025-08-14T21:48:02.6842068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6842147Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6842439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6842515Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6842765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6842843Z return func(*args, **kwargs) 2025-08-14T21:48:02.6843093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6843159Z return func(*args, **kwargs) 2025-08-14T21:48:02.6843401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6843466Z return func(*args, **kwargs) 2025-08-14T21:48:02.6843551Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6843766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6843843Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6844114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6844185Z layer_outputs = layer_module( 2025-08-14T21:48:02.6844405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6844492Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6844730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6844803Z return func(*args, **kwargs) 2025-08-14T21:48:02.6845041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6845130Z return func(*args, **kwargs) 2025-08-14T21:48:02.6845373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6845443Z return func(*args, **kwargs) 2025-08-14T21:48:02.6845713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6845798Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6846057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6846156Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6846467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6846595Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6846879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6847040Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6847272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6847346Z return self.act(input) 2025-08-14T21:48:02.6847350Z 2025-08-14T21:48:02.6847458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6847674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6847743Z return mod(**inputs) 2025-08-14T21:48:02.6848001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6848073Z return func(*args, **kwargs) 2025-08-14T21:48:02.6848336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6848419Z return func(*args, **kwargs) 2025-08-14T21:48:02.6848648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6848727Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6849014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6849088Z outputs = self.layoutlm( 2025-08-14T21:48:02.6849344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6849414Z return func(*args, **kwargs) 2025-08-14T21:48:02.6849665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6849745Z return func(*args, **kwargs) 2025-08-14T21:48:02.6849970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6850053Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6850340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6850418Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6850674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6850744Z return func(*args, **kwargs) 2025-08-14T21:48:02.6850989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6851067Z return func(*args, **kwargs) 2025-08-14T21:48:02.6851316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6851387Z return func(*args, **kwargs) 2025-08-14T21:48:02.6851494Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6851717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6851805Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6852082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6852159Z layer_outputs = layer_module( 2025-08-14T21:48:02.6852393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6852475Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6852729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6852798Z return func(*args, **kwargs) 2025-08-14T21:48:02.6853044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6853122Z return func(*args, **kwargs) 2025-08-14T21:48:02.6853409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6853481Z return func(*args, **kwargs) 2025-08-14T21:48:02.6853768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6853857Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6854141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6854220Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6854533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6854705Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6854984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6855074Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6855085Z 2025-08-14T21:48:02.6855195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6855403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6855478Z return mod(**inputs) 2025-08-14T21:48:02.6855725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6855796Z return func(*args, **kwargs) 2025-08-14T21:48:02.6856050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6856118Z return func(*args, **kwargs) 2025-08-14T21:48:02.6856350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6856432Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6856709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6856791Z outputs = self.layoutlm( 2025-08-14T21:48:02.6857039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6857110Z return func(*args, **kwargs) 2025-08-14T21:48:02.6857366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6857435Z return func(*args, **kwargs) 2025-08-14T21:48:02.6857665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6857745Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6858053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6858142Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6858393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6858462Z return func(*args, **kwargs) 2025-08-14T21:48:02.6858714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6858784Z return func(*args, **kwargs) 2025-08-14T21:48:02.6859039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6859108Z return func(*args, **kwargs) 2025-08-14T21:48:02.6859188Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6859420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6859499Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6859813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6859898Z layer_outputs = layer_module( 2025-08-14T21:48:02.6860127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6860218Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6860466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6860536Z return func(*args, **kwargs) 2025-08-14T21:48:02.6860788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6860858Z return func(*args, **kwargs) 2025-08-14T21:48:02.6861133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6861209Z return func(*args, **kwargs) 2025-08-14T21:48:02.6861495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6861584Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6861816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6861882Z return func(*args, **kwargs) 2025-08-14T21:48:02.6862124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6862191Z return func(*args, **kwargs) 2025-08-14T21:48:02.6862447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6862518Z return func(*args, **kwargs) 2025-08-14T21:48:02.6862800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6862884Z self_outputs = self.self( 2025-08-14T21:48:02.6863134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6863204Z return func(*args, **kwargs) 2025-08-14T21:48:02.6863457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6863525Z return func(*args, **kwargs) 2025-08-14T21:48:02.6863782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6863850Z return func(*args, **kwargs) 2025-08-14T21:48:02.6864129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6864318Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6864324Z 2025-08-14T21:48:02.6864430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6864633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6864699Z return mod(**inputs) 2025-08-14T21:48:02.6864934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6865007Z return func(*args, **kwargs) 2025-08-14T21:48:02.6865245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6865311Z return func(*args, **kwargs) 2025-08-14T21:48:02.6865530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6865607Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6865880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6865987Z outputs = self.layoutlm( 2025-08-14T21:48:02.6866221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6866297Z return func(*args, **kwargs) 2025-08-14T21:48:02.6866529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6866595Z return func(*args, **kwargs) 2025-08-14T21:48:02.6866811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6866884Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6867172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6867247Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6867492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6867567Z return func(*args, **kwargs) 2025-08-14T21:48:02.6867800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6867870Z return func(*args, **kwargs) 2025-08-14T21:48:02.6868103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6868166Z return func(*args, **kwargs) 2025-08-14T21:48:02.6868247Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6868456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6868529Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6868799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6868868Z layer_outputs = layer_module( 2025-08-14T21:48:02.6869094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6869171Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6869407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6869479Z return func(*args, **kwargs) 2025-08-14T21:48:02.6869709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6869775Z return func(*args, **kwargs) 2025-08-14T21:48:02.6870015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6870082Z return func(*args, **kwargs) 2025-08-14T21:48:02.6870397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6870487Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6870734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6870820Z return func(*args, **kwargs) 2025-08-14T21:48:02.6871054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6871121Z return func(*args, **kwargs) 2025-08-14T21:48:02.6871363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6871428Z return func(*args, **kwargs) 2025-08-14T21:48:02.6871721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6871796Z self_outputs = self.self( 2025-08-14T21:48:02.6872096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6872177Z return func(*args, **kwargs) 2025-08-14T21:48:02.6872436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6872506Z return func(*args, **kwargs) 2025-08-14T21:48:02.6872769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6872839Z return func(*args, **kwargs) 2025-08-14T21:48:02.6873131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6873296Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6873301Z 2025-08-14T21:48:02.6873415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6873638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6873707Z return mod(**inputs) 2025-08-14T21:48:02.6873962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6874032Z return func(*args, **kwargs) 2025-08-14T21:48:02.6874279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6874357Z return func(*args, **kwargs) 2025-08-14T21:48:02.6874583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6874662Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6874972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6875046Z outputs = self.layoutlm( 2025-08-14T21:48:02.6875302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6875373Z return func(*args, **kwargs) 2025-08-14T21:48:02.6875702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6875793Z return func(*args, **kwargs) 2025-08-14T21:48:02.6876018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6876097Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6876405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6876483Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6876744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6876838Z return func(*args, **kwargs) 2025-08-14T21:48:02.6877093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6877175Z return func(*args, **kwargs) 2025-08-14T21:48:02.6877424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6877503Z return func(*args, **kwargs) 2025-08-14T21:48:02.6877585Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6877814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6877904Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6878204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6878281Z layer_outputs = layer_module( 2025-08-14T21:48:02.6878526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6878649Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6878904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6878976Z return func(*args, **kwargs) 2025-08-14T21:48:02.6879221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6879300Z return func(*args, **kwargs) 2025-08-14T21:48:02.6879545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6879615Z return func(*args, **kwargs) 2025-08-14T21:48:02.6879920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6880012Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6880267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6880339Z return func(*args, **kwargs) 2025-08-14T21:48:02.6880585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6880664Z return func(*args, **kwargs) 2025-08-14T21:48:02.6880910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6880979Z return func(*args, **kwargs) 2025-08-14T21:48:02.6881283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6881356Z self_outputs = self.self( 2025-08-14T21:48:02.6881612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6881685Z return func(*args, **kwargs) 2025-08-14T21:48:02.6881930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6882010Z return func(*args, **kwargs) 2025-08-14T21:48:02.6882257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6882335Z return func(*args, **kwargs) 2025-08-14T21:48:02.6882611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6882765Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6882769Z 2025-08-14T21:48:02.6882862Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6882946Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6883074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6883292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6883367Z return mod(**inputs) 2025-08-14T21:48:02.6883624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6883695Z return func(*args, **kwargs) 2025-08-14T21:48:02.6883940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6884016Z return func(*args, **kwargs) 2025-08-14T21:48:02.6884241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6884318Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6884606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6884680Z outputs = self.layoutlm( 2025-08-14T21:48:02.6884949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6885039Z return func(*args, **kwargs) 2025-08-14T21:48:02.6885291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6885370Z return func(*args, **kwargs) 2025-08-14T21:48:02.6885599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6885677Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6885973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6886049Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6886354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6886428Z return func(*args, **kwargs) 2025-08-14T21:48:02.6886682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6886760Z return func(*args, **kwargs) 2025-08-14T21:48:02.6887015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6887092Z return func(*args, **kwargs) 2025-08-14T21:48:02.6887172Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6887400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6887484Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6887769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6887844Z layer_outputs = layer_module( 2025-08-14T21:48:02.6888088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6888174Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6888434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6888504Z return func(*args, **kwargs) 2025-08-14T21:48:02.6888756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6888834Z return func(*args, **kwargs) 2025-08-14T21:48:02.6889084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6889155Z return func(*args, **kwargs) 2025-08-14T21:48:02.6889448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6889557Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6889825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6889900Z return func(*args, **kwargs) 2025-08-14T21:48:02.6890154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6890236Z return func(*args, **kwargs) 2025-08-14T21:48:02.6890491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6890565Z return func(*args, **kwargs) 2025-08-14T21:48:02.6890861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6891003Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6891292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6891422Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6891426Z 2025-08-14T21:48:02.6891540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6891759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6891828Z return mod(**inputs) 2025-08-14T21:48:02.6892085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6892155Z return func(*args, **kwargs) 2025-08-14T21:48:02.6892401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6892478Z return func(*args, **kwargs) 2025-08-14T21:48:02.6892724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6892804Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6893089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6893162Z outputs = self.layoutlm( 2025-08-14T21:48:02.6893413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6893482Z return func(*args, **kwargs) 2025-08-14T21:48:02.6893728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6893805Z return func(*args, **kwargs) 2025-08-14T21:48:02.6894030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6894108Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6894395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6894476Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6894733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6894802Z return func(*args, **kwargs) 2025-08-14T21:48:02.6895047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6895123Z return func(*args, **kwargs) 2025-08-14T21:48:02.6895373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6895451Z return func(*args, **kwargs) 2025-08-14T21:48:02.6895530Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6895755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6895858Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6896135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6896211Z layer_outputs = layer_module( 2025-08-14T21:48:02.6896448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6896530Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6896784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6896856Z return func(*args, **kwargs) 2025-08-14T21:48:02.6897100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6897177Z return func(*args, **kwargs) 2025-08-14T21:48:02.6897423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6897493Z return func(*args, **kwargs) 2025-08-14T21:48:02.6897808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6897901Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6898183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6898262Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6898572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6898709Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6898999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6899098Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6899103Z 2025-08-14T21:48:02.6899213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6899426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6899503Z return mod(**inputs) 2025-08-14T21:48:02.6899751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6899820Z return func(*args, **kwargs) 2025-08-14T21:48:02.6900076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6900148Z return func(*args, **kwargs) 2025-08-14T21:48:02.6900381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6900458Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6900739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6900823Z outputs = self.layoutlm( 2025-08-14T21:48:02.6901075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6901146Z return func(*args, **kwargs) 2025-08-14T21:48:02.6901401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6901470Z return func(*args, **kwargs) 2025-08-14T21:48:02.6901702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6901782Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6902062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6902150Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6902421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6902497Z return func(*args, **kwargs) 2025-08-14T21:48:02.6902756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6902829Z return func(*args, **kwargs) 2025-08-14T21:48:02.6903090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6903162Z return func(*args, **kwargs) 2025-08-14T21:48:02.6903245Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6903484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6903562Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6903853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6903932Z layer_outputs = layer_module( 2025-08-14T21:48:02.6904216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6904310Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6904564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6904636Z return func(*args, **kwargs) 2025-08-14T21:48:02.6904900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6904972Z return func(*args, **kwargs) 2025-08-14T21:48:02.6905233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6905305Z return func(*args, **kwargs) 2025-08-14T21:48:02.6905606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6905713Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6905995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6906080Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6906410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6906540Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6906831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6906954Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6907184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6907273Z return self.act(input) 2025-08-14T21:48:02.6907279Z 2025-08-14T21:48:02.6907392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6907619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6907690Z return mod(**inputs) 2025-08-14T21:48:02.6907956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6908038Z return func(*args, **kwargs) 2025-08-14T21:48:02.6908296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6908367Z return func(*args, **kwargs) 2025-08-14T21:48:02.6908607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6908903Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6909377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6909461Z outputs = self.layoutlm( 2025-08-14T21:48:02.6909719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6909802Z return func(*args, **kwargs) 2025-08-14T21:48:02.6910055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6910127Z return func(*args, **kwargs) 2025-08-14T21:48:02.6910366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6910451Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6910746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6910827Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6911118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6911231Z return func(*args, **kwargs) 2025-08-14T21:48:02.6911492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6911573Z return func(*args, **kwargs) 2025-08-14T21:48:02.6911830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6911901Z return func(*args, **kwargs) 2025-08-14T21:48:02.6911993Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6912228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6912309Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6912639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6912721Z layer_outputs = layer_module( 2025-08-14T21:48:02.6912967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6913053Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6913308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6913390Z return func(*args, **kwargs) 2025-08-14T21:48:02.6913646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6913720Z return func(*args, **kwargs) 2025-08-14T21:48:02.6913983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6914057Z return func(*args, **kwargs) 2025-08-14T21:48:02.6914352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6914448Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6914726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6914816Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6915136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6915281Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6915670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6915780Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6915790Z 2025-08-14T21:48:02.6915938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6916155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6916231Z return mod(**inputs) 2025-08-14T21:48:02.6916497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6916572Z return func(*args, **kwargs) 2025-08-14T21:48:02.6916834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6916905Z return func(*args, **kwargs) 2025-08-14T21:48:02.6917134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6917225Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6917532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6917611Z outputs = self.layoutlm( 2025-08-14T21:48:02.6917895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6917988Z return func(*args, **kwargs) 2025-08-14T21:48:02.6918255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6918327Z return func(*args, **kwargs) 2025-08-14T21:48:02.6918558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6918649Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6918930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6919001Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6919256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6919325Z return func(*args, **kwargs) 2025-08-14T21:48:02.6919566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6919631Z return func(*args, **kwargs) 2025-08-14T21:48:02.6919860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6919935Z return func(*args, **kwargs) 2025-08-14T21:48:02.6920010Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6920224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6920296Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6920553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6920631Z layer_outputs = layer_module( 2025-08-14T21:48:02.6920845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6920924Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6921163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6921228Z return func(*args, **kwargs) 2025-08-14T21:48:02.6921463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6921529Z return func(*args, **kwargs) 2025-08-14T21:48:02.6921761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6921835Z return func(*args, **kwargs) 2025-08-14T21:48:02.6922096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6922197Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6922433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6922500Z return func(*args, **kwargs) 2025-08-14T21:48:02.6922738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6922804Z return func(*args, **kwargs) 2025-08-14T21:48:02.6923036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6923110Z return func(*args, **kwargs) 2025-08-14T21:48:02.6923374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6923445Z self_outputs = self.self( 2025-08-14T21:48:02.6923700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6923770Z return func(*args, **kwargs) 2025-08-14T21:48:02.6924061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6924134Z return func(*args, **kwargs) 2025-08-14T21:48:02.6924382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6924459Z return func(*args, **kwargs) 2025-08-14T21:48:02.6924759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:02.6924921Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6924925Z 2025-08-14T21:48:02.6925044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6925257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6925335Z return mod(**inputs) 2025-08-14T21:48:02.6925572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6925638Z return func(*args, **kwargs) 2025-08-14T21:48:02.6925880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6925947Z return func(*args, **kwargs) 2025-08-14T21:48:02.6926168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6926243Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6926512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6926594Z outputs = self.layoutlm( 2025-08-14T21:48:02.6926848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6926920Z return func(*args, **kwargs) 2025-08-14T21:48:02.6927179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6927250Z return func(*args, **kwargs) 2025-08-14T21:48:02.6927486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6927560Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6927825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6927907Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6928144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6928221Z return func(*args, **kwargs) 2025-08-14T21:48:02.6928454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6928545Z return func(*args, **kwargs) 2025-08-14T21:48:02.6928787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6928854Z return func(*args, **kwargs) 2025-08-14T21:48:02.6928930Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6929152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6929225Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6929498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6929569Z layer_outputs = layer_module( 2025-08-14T21:48:02.6929788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6929873Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6930158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6930227Z return func(*args, **kwargs) 2025-08-14T21:48:02.6930473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6930538Z return func(*args, **kwargs) 2025-08-14T21:48:02.6930778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6930844Z return func(*args, **kwargs) 2025-08-14T21:48:02.6931112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6931202Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6931458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6931528Z return func(*args, **kwargs) 2025-08-14T21:48:02.6931775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6931841Z return func(*args, **kwargs) 2025-08-14T21:48:02.6932086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6932152Z return func(*args, **kwargs) 2025-08-14T21:48:02.6932435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6932516Z self_outputs = self.self( 2025-08-14T21:48:02.6932772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6932850Z return func(*args, **kwargs) 2025-08-14T21:48:02.6933098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6933169Z return func(*args, **kwargs) 2025-08-14T21:48:02.6933432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6933502Z return func(*args, **kwargs) 2025-08-14T21:48:02.6933793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:02.6933946Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6933949Z 2025-08-14T21:48:02.6934059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6934275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6934346Z return mod(**inputs) 2025-08-14T21:48:02.6934609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6934706Z return func(*args, **kwargs) 2025-08-14T21:48:02.6934964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6935033Z return func(*args, **kwargs) 2025-08-14T21:48:02.6935270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6935349Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6935641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6935714Z outputs = self.layoutlm( 2025-08-14T21:48:02.6935966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6936046Z return func(*args, **kwargs) 2025-08-14T21:48:02.6936300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6936400Z return func(*args, **kwargs) 2025-08-14T21:48:02.6936652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6936736Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6937020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6937098Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6937344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6937421Z return func(*args, **kwargs) 2025-08-14T21:48:02.6937684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6937765Z return func(*args, **kwargs) 2025-08-14T21:48:02.6938016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6938089Z return func(*args, **kwargs) 2025-08-14T21:48:02.6938177Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6938402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6938480Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6938766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6938842Z layer_outputs = layer_module( 2025-08-14T21:48:02.6939081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6939164Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6939415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6939495Z return func(*args, **kwargs) 2025-08-14T21:48:02.6939742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6939813Z return func(*args, **kwargs) 2025-08-14T21:48:02.6940067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6940136Z return func(*args, **kwargs) 2025-08-14T21:48:02.6940421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6940508Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6940753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6940831Z return func(*args, **kwargs) 2025-08-14T21:48:02.6941094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6941173Z return func(*args, **kwargs) 2025-08-14T21:48:02.6941418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6941489Z return func(*args, **kwargs) 2025-08-14T21:48:02.6941774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:02.6941847Z self_outputs = self.self( 2025-08-14T21:48:02.6942093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6942172Z return func(*args, **kwargs) 2025-08-14T21:48:02.6942419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6942498Z return func(*args, **kwargs) 2025-08-14T21:48:02.6942744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6942855Z return func(*args, **kwargs) 2025-08-14T21:48:02.6943139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:02.6943291Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:02.6943295Z 2025-08-14T21:48:02.6943378Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6943469Z cudagraph partition due to non gpu ops 2025-08-14T21:48:02.6943579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6943790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6943860Z return mod(**inputs) 2025-08-14T21:48:02.6944123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6944203Z return func(*args, **kwargs) 2025-08-14T21:48:02.6944453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6944524Z return func(*args, **kwargs) 2025-08-14T21:48:02.6944757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6944835Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6945135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6945209Z outputs = self.layoutlm( 2025-08-14T21:48:02.6945456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6945528Z return func(*args, **kwargs) 2025-08-14T21:48:02.6945755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6945820Z return func(*args, **kwargs) 2025-08-14T21:48:02.6946037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6946108Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6946380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6946454Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6946690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6946762Z return func(*args, **kwargs) 2025-08-14T21:48:02.6946995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6947070Z return func(*args, **kwargs) 2025-08-14T21:48:02.6947323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6947393Z return func(*args, **kwargs) 2025-08-14T21:48:02.6947474Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6947683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6947756Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6948026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6948095Z layer_outputs = layer_module( 2025-08-14T21:48:02.6948320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6948397Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6948630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6948705Z return func(*args, **kwargs) 2025-08-14T21:48:02.6948975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6949044Z return func(*args, **kwargs) 2025-08-14T21:48:02.6949289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6949355Z return func(*args, **kwargs) 2025-08-14T21:48:02.6949625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:02.6949707Z self_attention_outputs = self.attention( 2025-08-14T21:48:02.6949955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6950052Z return func(*args, **kwargs) 2025-08-14T21:48:02.6950304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6950376Z return func(*args, **kwargs) 2025-08-14T21:48:02.6950632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6950703Z return func(*args, **kwargs) 2025-08-14T21:48:02.6950991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:02.6951130Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:02.6951410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:02.6951508Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6951512Z 2025-08-14T21:48:02.6951624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6951842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6951914Z return mod(**inputs) 2025-08-14T21:48:02.6952166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6952244Z return func(*args, **kwargs) 2025-08-14T21:48:02.6952494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6952565Z return func(*args, **kwargs) 2025-08-14T21:48:02.6952798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6952877Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6953162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6953238Z outputs = self.layoutlm( 2025-08-14T21:48:02.6953512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6953593Z return func(*args, **kwargs) 2025-08-14T21:48:02.6953848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6953919Z return func(*args, **kwargs) 2025-08-14T21:48:02.6954161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6954241Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6954536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6954618Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6954877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6954959Z return func(*args, **kwargs) 2025-08-14T21:48:02.6955214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6955338Z return func(*args, **kwargs) 2025-08-14T21:48:02.6955680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6955763Z return func(*args, **kwargs) 2025-08-14T21:48:02.6955856Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6956088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6956170Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6956485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6956562Z layer_outputs = layer_module( 2025-08-14T21:48:02.6956837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6956919Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6957148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6957225Z return func(*args, **kwargs) 2025-08-14T21:48:02.6957451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6957518Z return func(*args, **kwargs) 2025-08-14T21:48:02.6957750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6957816Z return func(*args, **kwargs) 2025-08-14T21:48:02.6958082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6958169Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6958428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6958516Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6958807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6958935Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6959195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:02.6959278Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6959282Z 2025-08-14T21:48:02.6959391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6959588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6959655Z return mod(**inputs) 2025-08-14T21:48:02.6959923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6959995Z return func(*args, **kwargs) 2025-08-14T21:48:02.6960239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6960306Z return func(*args, **kwargs) 2025-08-14T21:48:02.6960520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6960600Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6960863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6960932Z outputs = self.layoutlm( 2025-08-14T21:48:02.6961177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6961249Z return func(*args, **kwargs) 2025-08-14T21:48:02.6961524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6961613Z return func(*args, **kwargs) 2025-08-14T21:48:02.6961836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6961923Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6962221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6962304Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6962552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6962621Z return func(*args, **kwargs) 2025-08-14T21:48:02.6962889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6962964Z return func(*args, **kwargs) 2025-08-14T21:48:02.6963213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6963289Z return func(*args, **kwargs) 2025-08-14T21:48:02.6963370Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6963602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6963679Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6963978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6964060Z layer_outputs = layer_module( 2025-08-14T21:48:02.6964291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6964375Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6964632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6964706Z return func(*args, **kwargs) 2025-08-14T21:48:02.6964959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6965029Z return func(*args, **kwargs) 2025-08-14T21:48:02.6965273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6965352Z return func(*args, **kwargs) 2025-08-14T21:48:02.6965651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6965741Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6966021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6966130Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6966453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:02.6966582Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:02.6966857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:02.6966985Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:02.6967211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:02.6967292Z return self.act(input) 2025-08-14T21:48:02.6967296Z 2025-08-14T21:48:02.6967405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6967617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6967698Z return mod(**inputs) 2025-08-14T21:48:02.6967962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6968057Z return func(*args, **kwargs) 2025-08-14T21:48:02.6968313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6968385Z return func(*args, **kwargs) 2025-08-14T21:48:02.6968619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6968697Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6968980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:48:02.6969060Z outputs = self.layoutlm( 2025-08-14T21:48:02.6969328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6969401Z return func(*args, **kwargs) 2025-08-14T21:48:02.6969669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6969740Z return func(*args, **kwargs) 2025-08-14T21:48:02.6969972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6970050Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6970330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:02.6970415Z encoder_outputs = self.encoder( 2025-08-14T21:48:02.6970665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6970744Z return func(*args, **kwargs) 2025-08-14T21:48:02.6970995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6971068Z return func(*args, **kwargs) 2025-08-14T21:48:02.6971329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6971399Z return func(*args, **kwargs) 2025-08-14T21:48:02.6971481Z [Previous line repeated 1 more time] 2025-08-14T21:48:02.6971717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6971793Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6972081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:02.6972154Z layer_outputs = layer_module( 2025-08-14T21:48:02.6972387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:02.6972500Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:02.6972751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6972823Z return func(*args, **kwargs) 2025-08-14T21:48:02.6973079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6973150Z return func(*args, **kwargs) 2025-08-14T21:48:02.6973406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6973476Z return func(*args, **kwargs) 2025-08-14T21:48:02.6973761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:02.6973858Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:02.6974131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:02.6974218Z return forward_fn(*input_tensors) 2025-08-14T21:48:02.6974563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:02.6974707Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:02.6974994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:02.6975084Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6975088Z 2025-08-14T21:48:02.6975200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6975418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6975491Z return mod(**inputs) 2025-08-14T21:48:02.6975775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6975849Z return func(*args, **kwargs) 2025-08-14T21:48:02.6976098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6976179Z return func(*args, **kwargs) 2025-08-14T21:48:02.6976404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6976483Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6976789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 771, in forward 2025-08-14T21:48:02.6976890Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:48:02.6977191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 484, in forward 2025-08-14T21:48:02.6977310Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:48:02.6977593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 472, in forward 2025-08-14T21:48:02.6977702Z hidden_states = self.transform(hidden_states) 2025-08-14T21:48:02.6977977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 447, in forward 2025-08-14T21:48:02.6978070Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:02.6978074Z 2025-08-14T21:48:02.6978181Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6978389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6978469Z return mod(**inputs) 2025-08-14T21:48:02.6978720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6978794Z return func(*args, **kwargs) 2025-08-14T21:48:02.6979049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6979141Z return func(*args, **kwargs) 2025-08-14T21:48:02.6979384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6979464Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6979752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 771, in forward 2025-08-14T21:48:02.6979860Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:48:02.6980160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 484, in forward 2025-08-14T21:48:02.6980285Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:48:02.6980588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 473, in forward 2025-08-14T21:48:02.6980688Z hidden_states = self.decoder(hidden_states) 2025-08-14T21:48:02.6980711Z 2025-08-14T21:48:02.6980847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:02.6981052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:02.6981128Z return mod(**inputs) 2025-08-14T21:48:02.6981368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6981434Z return func(*args, **kwargs) 2025-08-14T21:48:02.6981673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:02.6981740Z return func(*args, **kwargs) 2025-08-14T21:48:02.6981950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:02.6982050Z output = func(self, *args, **kwargs) 2025-08-14T21:48:02.6982315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 776, in forward 2025-08-14T21:48:02.6982397Z masked_lm_loss = loss_fct( 2025-08-14T21:48:02.6982400Z 2025-08-14T21:48:11.0558244Z Compilation time (from dynamo_timed): 15.557269063 2025-08-14T21:48:11.0595299Z pass 2025-08-14T21:48:11.0596645Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:11.0597548Z TIMING: _recursive_pre_grad_passes:0.00824 _recursive_joint_graph_passes:0.45992 _recursive_post_grad_passes:0.08533 async_compile.wait:0.60659 code_gen:7.604 inductor_compile:8.81695 backend_compile:12.47361 gc:0.00025 entire_frame_compile:15.55727 total_wall_time:15.55727 2025-08-14T21:48:11.0598541Z STATS: call_* op count: 432 | FakeTensorMode.__torch_dispatch__:15442 | FakeTensor.__torch_dispatch__:4798 | ProxyTorchDispatchMode.__torch_dispatch__:5848 2025-08-14T21:48:11.0599122Z Dynamo produced 1 graphs covering 432 ops with 0 graph breaks (0 unique) 2025-08-14T21:48:16.3140940Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:48:16.3142112Z from pkg_resources import resource_filename 2025-08-14T21:48:16.9127861Z 2025-08-14T21:48:18.1329643Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:48:18.1336656Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:48:18.1343284Z cpu eval LayoutLMForSequenceClassification 2025-08-14T21:48:18.6509641Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:18.8503241Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:19.0470170Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:27.5267833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5268506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5268971Z return mod(**inputs) 2025-08-14T21:48:27.5269369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5269740Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5270314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5270799Z outputs = self.layoutlm( 2025-08-14T21:48:27.5271174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5271582Z return func(*args, **kwargs) 2025-08-14T21:48:27.5271991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5272782Z return func(*args, **kwargs) 2025-08-14T21:48:27.5273203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5273603Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5274044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5274503Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5274909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5275307Z return func(*args, **kwargs) 2025-08-14T21:48:27.5275922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5276412Z return func(*args, **kwargs) 2025-08-14T21:48:27.5276807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5277247Z return func(*args, **kwargs) 2025-08-14T21:48:27.5277467Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5277850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5278231Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5278670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5279089Z layer_outputs = layer_module( 2025-08-14T21:48:27.5279465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5279901Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5280342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5280736Z return func(*args, **kwargs) 2025-08-14T21:48:27.5281125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5281518Z return func(*args, **kwargs) 2025-08-14T21:48:27.5281967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5282365Z return func(*args, **kwargs) 2025-08-14T21:48:27.5282794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5283300Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5283712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5284121Z return func(*args, **kwargs) 2025-08-14T21:48:27.5284618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5285016Z return func(*args, **kwargs) 2025-08-14T21:48:27.5285398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5285836Z return func(*args, **kwargs) 2025-08-14T21:48:27.5286248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5286664Z self_outputs = self.self( 2025-08-14T21:48:27.5287055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5287449Z return func(*args, **kwargs) 2025-08-14T21:48:27.5287832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5288223Z return func(*args, **kwargs) 2025-08-14T21:48:27.5288610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5289063Z return func(*args, **kwargs) 2025-08-14T21:48:27.5289479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.5289995Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5290237Z 2025-08-14T21:48:27.5290353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5290745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5291097Z return mod(**inputs) 2025-08-14T21:48:27.5291445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5291843Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5292278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5292702Z outputs = self.layoutlm( 2025-08-14T21:48:27.5293157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5293609Z return func(*args, **kwargs) 2025-08-14T21:48:27.5293990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5294371Z return func(*args, **kwargs) 2025-08-14T21:48:27.5294716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5295070Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5295470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5295915Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5296323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5296723Z return func(*args, **kwargs) 2025-08-14T21:48:27.5297093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5297484Z return func(*args, **kwargs) 2025-08-14T21:48:27.5297865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5298259Z return func(*args, **kwargs) 2025-08-14T21:48:27.5298471Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5298823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5299180Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5299582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5300013Z layer_outputs = layer_module( 2025-08-14T21:48:27.5300391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5300769Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5301174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5301580Z return func(*args, **kwargs) 2025-08-14T21:48:27.5301964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5302364Z return func(*args, **kwargs) 2025-08-14T21:48:27.5302746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5303192Z return func(*args, **kwargs) 2025-08-14T21:48:27.5303605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5304075Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5304485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5304880Z return func(*args, **kwargs) 2025-08-14T21:48:27.5305248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5305646Z return func(*args, **kwargs) 2025-08-14T21:48:27.5306027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5306430Z return func(*args, **kwargs) 2025-08-14T21:48:27.5306857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5307285Z self_outputs = self.self( 2025-08-14T21:48:27.5307675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5308072Z return func(*args, **kwargs) 2025-08-14T21:48:27.5308455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5309058Z return func(*args, **kwargs) 2025-08-14T21:48:27.5309438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5309821Z return func(*args, **kwargs) 2025-08-14T21:48:27.5310229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.5310722Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5310925Z 2025-08-14T21:48:27.5311043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5311435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5311793Z return mod(**inputs) 2025-08-14T21:48:27.5312149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5312538Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5312972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5313410Z outputs = self.layoutlm( 2025-08-14T21:48:27.5313798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5314196Z return func(*args, **kwargs) 2025-08-14T21:48:27.5314587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5315038Z return func(*args, **kwargs) 2025-08-14T21:48:27.5315395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5315950Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5316393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5316840Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5317245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5317651Z return func(*args, **kwargs) 2025-08-14T21:48:27.5318031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5318419Z return func(*args, **kwargs) 2025-08-14T21:48:27.5318804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5319203Z return func(*args, **kwargs) 2025-08-14T21:48:27.5319449Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5319848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5320227Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5320649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5321069Z layer_outputs = layer_module( 2025-08-14T21:48:27.5321437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5321827Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5322230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5322652Z return func(*args, **kwargs) 2025-08-14T21:48:27.5323033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5323428Z return func(*args, **kwargs) 2025-08-14T21:48:27.5323799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5324205Z return func(*args, **kwargs) 2025-08-14T21:48:27.5324617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5325068Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5325468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5325861Z return func(*args, **kwargs) 2025-08-14T21:48:27.5326240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5326693Z return func(*args, **kwargs) 2025-08-14T21:48:27.5327062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5327454Z return func(*args, **kwargs) 2025-08-14T21:48:27.5327855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5328270Z self_outputs = self.self( 2025-08-14T21:48:27.5328650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5329033Z return func(*args, **kwargs) 2025-08-14T21:48:27.5329405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5329791Z return func(*args, **kwargs) 2025-08-14T21:48:27.5330169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5330575Z return func(*args, **kwargs) 2025-08-14T21:48:27.5330977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.5331489Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5331698Z 2025-08-14T21:48:27.5331783Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5332034Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5332270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5332635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5332962Z return mod(**inputs) 2025-08-14T21:48:27.5333287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5333673Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5334081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5334594Z outputs = self.layoutlm( 2025-08-14T21:48:27.5334971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5335359Z return func(*args, **kwargs) 2025-08-14T21:48:27.5335737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5336123Z return func(*args, **kwargs) 2025-08-14T21:48:27.5336473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5336850Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5337882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5338280Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5338653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5339019Z return func(*args, **kwargs) 2025-08-14T21:48:27.5339371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5339727Z return func(*args, **kwargs) 2025-08-14T21:48:27.5340080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5340443Z return func(*args, **kwargs) 2025-08-14T21:48:27.5340631Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5340980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5341328Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5341725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5342118Z layer_outputs = layer_module( 2025-08-14T21:48:27.5342460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5342821Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5343185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5343583Z return func(*args, **kwargs) 2025-08-14T21:48:27.5343958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5344350Z return func(*args, **kwargs) 2025-08-14T21:48:27.5344717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5345112Z return func(*args, **kwargs) 2025-08-14T21:48:27.5345541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5345974Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5346368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5346754Z return func(*args, **kwargs) 2025-08-14T21:48:27.5347133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5347519Z return func(*args, **kwargs) 2025-08-14T21:48:27.5347899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5348273Z return func(*args, **kwargs) 2025-08-14T21:48:27.5348681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.5349161Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.5349688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.5350133Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5350283Z 2025-08-14T21:48:27.5350401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5350774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5351128Z return mod(**inputs) 2025-08-14T21:48:27.5351473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5351836Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5352285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5352715Z outputs = self.layoutlm( 2025-08-14T21:48:27.5353099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5353493Z return func(*args, **kwargs) 2025-08-14T21:48:27.5353879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5354276Z return func(*args, **kwargs) 2025-08-14T21:48:27.5354621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5354994Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5355416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5355956Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5356373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5356786Z return func(*args, **kwargs) 2025-08-14T21:48:27.5357189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5357588Z return func(*args, **kwargs) 2025-08-14T21:48:27.5357980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5358380Z return func(*args, **kwargs) 2025-08-14T21:48:27.5358589Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5358960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5359338Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5359764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5360194Z layer_outputs = layer_module( 2025-08-14T21:48:27.5360629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5361014Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5361412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5361797Z return func(*args, **kwargs) 2025-08-14T21:48:27.5362150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5362538Z return func(*args, **kwargs) 2025-08-14T21:48:27.5362903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5363293Z return func(*args, **kwargs) 2025-08-14T21:48:27.5363701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5364143Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5364586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5364999Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5365432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5365921Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5366373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.5366814Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5366961Z 2025-08-14T21:48:27.5367079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5367453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5367780Z return mod(**inputs) 2025-08-14T21:48:27.5368109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5368469Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5368860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5369260Z outputs = self.layoutlm( 2025-08-14T21:48:27.5369637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5370007Z return func(*args, **kwargs) 2025-08-14T21:48:27.5370355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5370720Z return func(*args, **kwargs) 2025-08-14T21:48:27.5371059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5371405Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5371811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5372210Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5372580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5372939Z return func(*args, **kwargs) 2025-08-14T21:48:27.5373292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5373665Z return func(*args, **kwargs) 2025-08-14T21:48:27.5374032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5374416Z return func(*args, **kwargs) 2025-08-14T21:48:27.5374613Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5374988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5375337Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5375738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5376143Z layer_outputs = layer_module( 2025-08-14T21:48:27.5376482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5376860Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5377258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5377647Z return func(*args, **kwargs) 2025-08-14T21:48:27.5378029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5378401Z return func(*args, **kwargs) 2025-08-14T21:48:27.5378755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5379145Z return func(*args, **kwargs) 2025-08-14T21:48:27.5379529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5379939Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5380342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5380727Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5381153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5381629Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5382094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.5382554Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.5382957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.5383317Z return self.act(input) 2025-08-14T21:48:27.5383436Z 2025-08-14T21:48:27.5383548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5383933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5384278Z return mod(**inputs) 2025-08-14T21:48:27.5384625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5384992Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5385416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5385840Z outputs = self.layoutlm( 2025-08-14T21:48:27.5386215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5386603Z return func(*args, **kwargs) 2025-08-14T21:48:27.5386978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5387377Z return func(*args, **kwargs) 2025-08-14T21:48:27.5387720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5388092Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5388540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5388973Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5389358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5389766Z return func(*args, **kwargs) 2025-08-14T21:48:27.5390139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5390516Z return func(*args, **kwargs) 2025-08-14T21:48:27.5390889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5391283Z return func(*args, **kwargs) 2025-08-14T21:48:27.5391489Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5391850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5392219Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5392639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5393053Z layer_outputs = layer_module( 2025-08-14T21:48:27.5393421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5393862Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5394260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5394637Z return func(*args, **kwargs) 2025-08-14T21:48:27.5395009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5395404Z return func(*args, **kwargs) 2025-08-14T21:48:27.5395875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5396278Z return func(*args, **kwargs) 2025-08-14T21:48:27.5396721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5397221Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5397662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5398079Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5398527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.5399001Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.5399448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.5399859Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5399997Z 2025-08-14T21:48:27.5400113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5400467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5400793Z return mod(**inputs) 2025-08-14T21:48:27.5401129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5401470Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5401853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5402245Z outputs = self.layoutlm( 2025-08-14T21:48:27.5402608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5402971Z return func(*args, **kwargs) 2025-08-14T21:48:27.5403329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5403697Z return func(*args, **kwargs) 2025-08-14T21:48:27.5404033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5404399Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5404799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5405202Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5405563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5405933Z return func(*args, **kwargs) 2025-08-14T21:48:27.5406285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5406656Z return func(*args, **kwargs) 2025-08-14T21:48:27.5406996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5407355Z return func(*args, **kwargs) 2025-08-14T21:48:27.5407548Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5407901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5408260Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5408825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5409229Z layer_outputs = layer_module( 2025-08-14T21:48:27.5409568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5409934Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5410311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5410685Z return func(*args, **kwargs) 2025-08-14T21:48:27.5411090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5411467Z return func(*args, **kwargs) 2025-08-14T21:48:27.5411815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5412168Z return func(*args, **kwargs) 2025-08-14T21:48:27.5412546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5412952Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5413337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5413701Z return func(*args, **kwargs) 2025-08-14T21:48:27.5414061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5414439Z return func(*args, **kwargs) 2025-08-14T21:48:27.5414783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5415147Z return func(*args, **kwargs) 2025-08-14T21:48:27.5415527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5415923Z self_outputs = self.self( 2025-08-14T21:48:27.5416281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5416662Z return func(*args, **kwargs) 2025-08-14T21:48:27.5417010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5417368Z return func(*args, **kwargs) 2025-08-14T21:48:27.5417712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5418073Z return func(*args, **kwargs) 2025-08-14T21:48:27.5418483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.5418937Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5419140Z 2025-08-14T21:48:27.5419244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5419595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5419907Z return mod(**inputs) 2025-08-14T21:48:27.5420218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5420561Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5420954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5421335Z outputs = self.layoutlm( 2025-08-14T21:48:27.5421687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5422081Z return func(*args, **kwargs) 2025-08-14T21:48:27.5422503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5422869Z return func(*args, **kwargs) 2025-08-14T21:48:27.5423205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5423577Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5423991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5424416Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5424808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5425213Z return func(*args, **kwargs) 2025-08-14T21:48:27.5425583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5425973Z return func(*args, **kwargs) 2025-08-14T21:48:27.5426347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5426740Z return func(*args, **kwargs) 2025-08-14T21:48:27.5426938Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5427304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5427674Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5428084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5428503Z layer_outputs = layer_module( 2025-08-14T21:48:27.5428873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5429252Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5429648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5430030Z return func(*args, **kwargs) 2025-08-14T21:48:27.5430402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5430779Z return func(*args, **kwargs) 2025-08-14T21:48:27.5431149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5431540Z return func(*args, **kwargs) 2025-08-14T21:48:27.5431957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5432393Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5432817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5433207Z return func(*args, **kwargs) 2025-08-14T21:48:27.5433573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5433959Z return func(*args, **kwargs) 2025-08-14T21:48:27.5434335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5434718Z return func(*args, **kwargs) 2025-08-14T21:48:27.5435117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5435538Z self_outputs = self.self( 2025-08-14T21:48:27.5435992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5436390Z return func(*args, **kwargs) 2025-08-14T21:48:27.5436782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5437223Z return func(*args, **kwargs) 2025-08-14T21:48:27.5437585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5437945Z return func(*args, **kwargs) 2025-08-14T21:48:27.5438330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.5438796Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5438989Z 2025-08-14T21:48:27.5439104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5439459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5439816Z return mod(**inputs) 2025-08-14T21:48:27.5440146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5440493Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5440903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5441296Z outputs = self.layoutlm( 2025-08-14T21:48:27.5441653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5442013Z return func(*args, **kwargs) 2025-08-14T21:48:27.5442366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5442731Z return func(*args, **kwargs) 2025-08-14T21:48:27.5443065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5443465Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5443888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5444314Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5444696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5445087Z return func(*args, **kwargs) 2025-08-14T21:48:27.5445439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5445809Z return func(*args, **kwargs) 2025-08-14T21:48:27.5446152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5446513Z return func(*args, **kwargs) 2025-08-14T21:48:27.5446709Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5447051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5447424Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5447837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5448225Z layer_outputs = layer_module( 2025-08-14T21:48:27.5448557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5448907Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5449271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5449622Z return func(*args, **kwargs) 2025-08-14T21:48:27.5449976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5450354Z return func(*args, **kwargs) 2025-08-14T21:48:27.5450700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5451103Z return func(*args, **kwargs) 2025-08-14T21:48:27.5451477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5451873Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5452236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5452602Z return func(*args, **kwargs) 2025-08-14T21:48:27.5452961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5453336Z return func(*args, **kwargs) 2025-08-14T21:48:27.5453708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5454081Z return func(*args, **kwargs) 2025-08-14T21:48:27.5454470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5454883Z self_outputs = self.self( 2025-08-14T21:48:27.5455278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5455675Z return func(*args, **kwargs) 2025-08-14T21:48:27.5456058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5456452Z return func(*args, **kwargs) 2025-08-14T21:48:27.5456827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5457197Z return func(*args, **kwargs) 2025-08-14T21:48:27.5457581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.5458063Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5458269Z 2025-08-14T21:48:27.5458351Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5458567Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5458799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5459165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5459497Z return mod(**inputs) 2025-08-14T21:48:27.5459833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5460184Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5460590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5461012Z outputs = self.layoutlm( 2025-08-14T21:48:27.5461411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5461806Z return func(*args, **kwargs) 2025-08-14T21:48:27.5462186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5462548Z return func(*args, **kwargs) 2025-08-14T21:48:27.5462877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5463220Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5463603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5463990Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5464359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5464721Z return func(*args, **kwargs) 2025-08-14T21:48:27.5465076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5465468Z return func(*args, **kwargs) 2025-08-14T21:48:27.5465823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5466188Z return func(*args, **kwargs) 2025-08-14T21:48:27.5466375Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5466718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5467063Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5467456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5467844Z layer_outputs = layer_module( 2025-08-14T21:48:27.5468207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5468571Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5468940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5469306Z return func(*args, **kwargs) 2025-08-14T21:48:27.5469661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5470044Z return func(*args, **kwargs) 2025-08-14T21:48:27.5470409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5470795Z return func(*args, **kwargs) 2025-08-14T21:48:27.5471202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5471646Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5472043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5472434Z return func(*args, **kwargs) 2025-08-14T21:48:27.5472807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5473192Z return func(*args, **kwargs) 2025-08-14T21:48:27.5473567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5473961Z return func(*args, **kwargs) 2025-08-14T21:48:27.5474367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.5474836Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.5475326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.5475887Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5476052Z 2025-08-14T21:48:27.5476171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5476574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5476940Z return mod(**inputs) 2025-08-14T21:48:27.5477303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5477672Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5478079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5478483Z outputs = self.layoutlm( 2025-08-14T21:48:27.5478837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5479215Z return func(*args, **kwargs) 2025-08-14T21:48:27.5479578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5480003Z return func(*args, **kwargs) 2025-08-14T21:48:27.5480337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5480697Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5481103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5481507Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5481876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5482251Z return func(*args, **kwargs) 2025-08-14T21:48:27.5482633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5482997Z return func(*args, **kwargs) 2025-08-14T21:48:27.5483353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5483721Z return func(*args, **kwargs) 2025-08-14T21:48:27.5483913Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5484258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5484609Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5485006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5485398Z layer_outputs = layer_module( 2025-08-14T21:48:27.5485747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5486113Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5486495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5486862Z return func(*args, **kwargs) 2025-08-14T21:48:27.5487225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5487597Z return func(*args, **kwargs) 2025-08-14T21:48:27.5487944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5488339Z return func(*args, **kwargs) 2025-08-14T21:48:27.5488751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5489207Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5489625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5490048Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5490482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5490994Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5491461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.5491902Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5492049Z 2025-08-14T21:48:27.5492171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5492544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5492890Z return mod(**inputs) 2025-08-14T21:48:27.5493237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5493612Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5494027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5494488Z outputs = self.layoutlm( 2025-08-14T21:48:27.5494871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5495258Z return func(*args, **kwargs) 2025-08-14T21:48:27.5495634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5496021Z return func(*args, **kwargs) 2025-08-14T21:48:27.5496375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5496742Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5498081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5498537Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5498943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5499330Z return func(*args, **kwargs) 2025-08-14T21:48:27.5499710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5500101Z return func(*args, **kwargs) 2025-08-14T21:48:27.5500476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5500868Z return func(*args, **kwargs) 2025-08-14T21:48:27.5501087Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5501444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5501794Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5502200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5502604Z layer_outputs = layer_module( 2025-08-14T21:48:27.5502949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5503314Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5503694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5504064Z return func(*args, **kwargs) 2025-08-14T21:48:27.5504416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5504789Z return func(*args, **kwargs) 2025-08-14T21:48:27.5505149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5505517Z return func(*args, **kwargs) 2025-08-14T21:48:27.5505928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5506333Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5506728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5507111Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5507533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5508015Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5508459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.5509038Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.5509429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.5509847Z return self.act(input) 2025-08-14T21:48:27.5509995Z 2025-08-14T21:48:27.5510108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5510491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5510834Z return mod(**inputs) 2025-08-14T21:48:27.5511183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5511548Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5511973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5512401Z outputs = self.layoutlm( 2025-08-14T21:48:27.5512799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5513193Z return func(*args, **kwargs) 2025-08-14T21:48:27.5513579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5513983Z return func(*args, **kwargs) 2025-08-14T21:48:27.5514337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5514720Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5515154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5515625Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5516045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5516449Z return func(*args, **kwargs) 2025-08-14T21:48:27.5516841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5517234Z return func(*args, **kwargs) 2025-08-14T21:48:27.5517628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5518029Z return func(*args, **kwargs) 2025-08-14T21:48:27.5518233Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5518613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5518994Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5519430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5519860Z layer_outputs = layer_module( 2025-08-14T21:48:27.5520237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5520627Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5521073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5521468Z return func(*args, **kwargs) 2025-08-14T21:48:27.5521855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5522259Z return func(*args, **kwargs) 2025-08-14T21:48:27.5522634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5523028Z return func(*args, **kwargs) 2025-08-14T21:48:27.5523454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5523887Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5524312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5524707Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5525173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.5525656Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.5526115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.5526525Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5526665Z 2025-08-14T21:48:27.5526780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5527135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5527461Z return mod(**inputs) 2025-08-14T21:48:27.5527810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5528171Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5528565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5528961Z outputs = self.layoutlm( 2025-08-14T21:48:27.5529318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5529743Z return func(*args, **kwargs) 2025-08-14T21:48:27.5530102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5530469Z return func(*args, **kwargs) 2025-08-14T21:48:27.5530801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5531143Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5531539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5531944Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5532307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5532675Z return func(*args, **kwargs) 2025-08-14T21:48:27.5533031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5533394Z return func(*args, **kwargs) 2025-08-14T21:48:27.5533745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5534108Z return func(*args, **kwargs) 2025-08-14T21:48:27.5534304Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5534648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5535026Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5535427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5535828Z layer_outputs = layer_module( 2025-08-14T21:48:27.5536168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5536531Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5536908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5537274Z return func(*args, **kwargs) 2025-08-14T21:48:27.5537624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5537983Z return func(*args, **kwargs) 2025-08-14T21:48:27.5538337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5538692Z return func(*args, **kwargs) 2025-08-14T21:48:27.5539114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5539526Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5539892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5540242Z return func(*args, **kwargs) 2025-08-14T21:48:27.5540585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5540941Z return func(*args, **kwargs) 2025-08-14T21:48:27.5541276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5541634Z return func(*args, **kwargs) 2025-08-14T21:48:27.5542039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5542451Z self_outputs = self.self( 2025-08-14T21:48:27.5542800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5543174Z return func(*args, **kwargs) 2025-08-14T21:48:27.5543550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5543905Z return func(*args, **kwargs) 2025-08-14T21:48:27.5544253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5544615Z return func(*args, **kwargs) 2025-08-14T21:48:27.5544997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.5545481Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5545705Z 2025-08-14T21:48:27.5545824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5546213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5546561Z return mod(**inputs) 2025-08-14T21:48:27.5546915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5547306Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5547754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5548184Z outputs = self.layoutlm( 2025-08-14T21:48:27.5548578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5548993Z return func(*args, **kwargs) 2025-08-14T21:48:27.5549373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5549779Z return func(*args, **kwargs) 2025-08-14T21:48:27.5550144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5550526Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5550951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5551385Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5551784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5552190Z return func(*args, **kwargs) 2025-08-14T21:48:27.5552566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5552965Z return func(*args, **kwargs) 2025-08-14T21:48:27.5553353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5553780Z return func(*args, **kwargs) 2025-08-14T21:48:27.5553996Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5554376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5554756Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5555182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5555694Z layer_outputs = layer_module( 2025-08-14T21:48:27.5556085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5556465Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5556914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5557322Z return func(*args, **kwargs) 2025-08-14T21:48:27.5557725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5558126Z return func(*args, **kwargs) 2025-08-14T21:48:27.5558520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5558928Z return func(*args, **kwargs) 2025-08-14T21:48:27.5559352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5559801Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5560223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5560639Z return func(*args, **kwargs) 2025-08-14T21:48:27.5561032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5561441Z return func(*args, **kwargs) 2025-08-14T21:48:27.5561836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5562246Z return func(*args, **kwargs) 2025-08-14T21:48:27.5562664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5563106Z self_outputs = self.self( 2025-08-14T21:48:27.5563514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5563913Z return func(*args, **kwargs) 2025-08-14T21:48:27.5564312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5564723Z return func(*args, **kwargs) 2025-08-14T21:48:27.5565117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5565498Z return func(*args, **kwargs) 2025-08-14T21:48:27.5565902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.5566388Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5566588Z 2025-08-14T21:48:27.5566707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5567080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5567417Z return mod(**inputs) 2025-08-14T21:48:27.5567767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5568136Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5568561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5569017Z outputs = self.layoutlm( 2025-08-14T21:48:27.5569397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5569788Z return func(*args, **kwargs) 2025-08-14T21:48:27.5570164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5570567Z return func(*args, **kwargs) 2025-08-14T21:48:27.5570922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5571308Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5571761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5572200Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5572599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5573016Z return func(*args, **kwargs) 2025-08-14T21:48:27.5573391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5573783Z return func(*args, **kwargs) 2025-08-14T21:48:27.5574158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5574546Z return func(*args, **kwargs) 2025-08-14T21:48:27.5574752Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5575114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5575490Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5575909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5576325Z layer_outputs = layer_module( 2025-08-14T21:48:27.5576692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5577073Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5577475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5577867Z return func(*args, **kwargs) 2025-08-14T21:48:27.5578251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5578648Z return func(*args, **kwargs) 2025-08-14T21:48:27.5579016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5579413Z return func(*args, **kwargs) 2025-08-14T21:48:27.5579840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5580274Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5580665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5581049Z return func(*args, **kwargs) 2025-08-14T21:48:27.5581431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5581909Z return func(*args, **kwargs) 2025-08-14T21:48:27.5582298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5582753Z return func(*args, **kwargs) 2025-08-14T21:48:27.5583185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5583629Z self_outputs = self.self( 2025-08-14T21:48:27.5584060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5584491Z return func(*args, **kwargs) 2025-08-14T21:48:27.5584891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5585304Z return func(*args, **kwargs) 2025-08-14T21:48:27.5585698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5586097Z return func(*args, **kwargs) 2025-08-14T21:48:27.5586507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.5587019Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5587240Z 2025-08-14T21:48:27.5587381Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5587621Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5587881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5588283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5588646Z return mod(**inputs) 2025-08-14T21:48:27.5589002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5589394Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5589832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5590270Z outputs = self.layoutlm( 2025-08-14T21:48:27.5590661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5591076Z return func(*args, **kwargs) 2025-08-14T21:48:27.5591470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5591877Z return func(*args, **kwargs) 2025-08-14T21:48:27.5592236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5592622Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5593060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5593490Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5593896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5594308Z return func(*args, **kwargs) 2025-08-14T21:48:27.5594695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5618185Z return func(*args, **kwargs) 2025-08-14T21:48:27.5618857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5619286Z return func(*args, **kwargs) 2025-08-14T21:48:27.5619512Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5619906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5620286Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5620725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5621160Z layer_outputs = layer_module( 2025-08-14T21:48:27.5621536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5621920Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5622331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5622786Z return func(*args, **kwargs) 2025-08-14T21:48:27.5623198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5623601Z return func(*args, **kwargs) 2025-08-14T21:48:27.5623985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5624377Z return func(*args, **kwargs) 2025-08-14T21:48:27.5624785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5625235Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5625634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5626041Z return func(*args, **kwargs) 2025-08-14T21:48:27.5626397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5626792Z return func(*args, **kwargs) 2025-08-14T21:48:27.5627168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5627550Z return func(*args, **kwargs) 2025-08-14T21:48:27.5627960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.5628441Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.5628916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.5629346Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5629505Z 2025-08-14T21:48:27.5629625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5630023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5630376Z return mod(**inputs) 2025-08-14T21:48:27.5630743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5631133Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5631570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5631998Z outputs = self.layoutlm( 2025-08-14T21:48:27.5632395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5632807Z return func(*args, **kwargs) 2025-08-14T21:48:27.5633197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5633617Z return func(*args, **kwargs) 2025-08-14T21:48:27.5634009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5634404Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5634833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5635279Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5635785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5636210Z return func(*args, **kwargs) 2025-08-14T21:48:27.5636597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5637010Z return func(*args, **kwargs) 2025-08-14T21:48:27.5637409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5637809Z return func(*args, **kwargs) 2025-08-14T21:48:27.5638032Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5638482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5638860Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5639273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5639701Z layer_outputs = layer_module( 2025-08-14T21:48:27.5640070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5640458Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5640856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5641264Z return func(*args, **kwargs) 2025-08-14T21:48:27.5641644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5642036Z return func(*args, **kwargs) 2025-08-14T21:48:27.5642425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5642823Z return func(*args, **kwargs) 2025-08-14T21:48:27.5643232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5643680Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5644118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5644547Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5644999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5645518Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5645999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.5646439Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5646590Z 2025-08-14T21:48:27.5646707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5647098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5647454Z return mod(**inputs) 2025-08-14T21:48:27.5647804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5648187Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5648618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5649051Z outputs = self.layoutlm( 2025-08-14T21:48:27.5649450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5649846Z return func(*args, **kwargs) 2025-08-14T21:48:27.5650218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5650588Z return func(*args, **kwargs) 2025-08-14T21:48:27.5650932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5651305Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5651730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5652148Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5652546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5652938Z return func(*args, **kwargs) 2025-08-14T21:48:27.5653337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5653737Z return func(*args, **kwargs) 2025-08-14T21:48:27.5654113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5654511Z return func(*args, **kwargs) 2025-08-14T21:48:27.5654711Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5655088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5655464Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5655904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5656345Z layer_outputs = layer_module( 2025-08-14T21:48:27.5656716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5657086Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5657458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5657832Z return func(*args, **kwargs) 2025-08-14T21:48:27.5658214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5658603Z return func(*args, **kwargs) 2025-08-14T21:48:27.5658973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5659362Z return func(*args, **kwargs) 2025-08-14T21:48:27.5659774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5660208Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5660643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5661043Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5661475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5661951Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5662411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.5662881Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.5663289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.5663647Z return self.act(input) 2025-08-14T21:48:27.5663777Z 2025-08-14T21:48:27.5663912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5664298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5664639Z return mod(**inputs) 2025-08-14T21:48:27.5664989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5665362Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5665791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5666206Z outputs = self.layoutlm( 2025-08-14T21:48:27.5666561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5666952Z return func(*args, **kwargs) 2025-08-14T21:48:27.5667330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5667718Z return func(*args, **kwargs) 2025-08-14T21:48:27.5668083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5668478Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5668902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5669319Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5669709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5670097Z return func(*args, **kwargs) 2025-08-14T21:48:27.5670479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5670870Z return func(*args, **kwargs) 2025-08-14T21:48:27.5671275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5671682Z return func(*args, **kwargs) 2025-08-14T21:48:27.5671891Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5672274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5672659Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5673094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5673522Z layer_outputs = layer_module( 2025-08-14T21:48:27.5673902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5674306Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5674717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5675113Z return func(*args, **kwargs) 2025-08-14T21:48:27.5675500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5675999Z return func(*args, **kwargs) 2025-08-14T21:48:27.5676384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5676795Z return func(*args, **kwargs) 2025-08-14T21:48:27.5677233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5677706Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5678127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5678551Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5679007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.5679546Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.5680036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.5680470Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5680617Z 2025-08-14T21:48:27.5680737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5681117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5681464Z return mod(**inputs) 2025-08-14T21:48:27.5681815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5682193Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5682635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5683060Z outputs = self.layoutlm( 2025-08-14T21:48:27.5683489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5683881Z return func(*args, **kwargs) 2025-08-14T21:48:27.5684265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5684655Z return func(*args, **kwargs) 2025-08-14T21:48:27.5685009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5685375Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5685802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5686228Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5686629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5687021Z return func(*args, **kwargs) 2025-08-14T21:48:27.5687397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5687785Z return func(*args, **kwargs) 2025-08-14T21:48:27.5688151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5688541Z return func(*args, **kwargs) 2025-08-14T21:48:27.5688751Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5689113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5689485Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5689910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5690334Z layer_outputs = layer_module( 2025-08-14T21:48:27.5690694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5691088Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5691486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5691560Z return func(*args, **kwargs) 2025-08-14T21:48:27.5691817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5691890Z return func(*args, **kwargs) 2025-08-14T21:48:27.5692137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5692216Z return func(*args, **kwargs) 2025-08-14T21:48:27.5692498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5692609Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5692869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5692941Z return func(*args, **kwargs) 2025-08-14T21:48:27.5693197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5693268Z return func(*args, **kwargs) 2025-08-14T21:48:27.5693513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5693593Z return func(*args, **kwargs) 2025-08-14T21:48:27.5693892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5693969Z self_outputs = self.self( 2025-08-14T21:48:27.5694225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5694328Z return func(*args, **kwargs) 2025-08-14T21:48:27.5694598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5694671Z return func(*args, **kwargs) 2025-08-14T21:48:27.5694917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5694996Z return func(*args, **kwargs) 2025-08-14T21:48:27.5695296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.5695457Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5695461Z 2025-08-14T21:48:27.5695576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5695804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5695888Z return mod(**inputs) 2025-08-14T21:48:27.5696119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5696202Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5696490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5696565Z outputs = self.layoutlm( 2025-08-14T21:48:27.5696827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5696899Z return func(*args, **kwargs) 2025-08-14T21:48:27.5697146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5697227Z return func(*args, **kwargs) 2025-08-14T21:48:27.5697453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5697536Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5697826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5697906Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5698162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5698233Z return func(*args, **kwargs) 2025-08-14T21:48:27.5698482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5698561Z return func(*args, **kwargs) 2025-08-14T21:48:27.5698809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5698888Z return func(*args, **kwargs) 2025-08-14T21:48:27.5698995Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5699220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5699306Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5699584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5699660Z layer_outputs = layer_module( 2025-08-14T21:48:27.5699897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5699982Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5700239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5700310Z return func(*args, **kwargs) 2025-08-14T21:48:27.5700564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5700646Z return func(*args, **kwargs) 2025-08-14T21:48:27.5700941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5701016Z return func(*args, **kwargs) 2025-08-14T21:48:27.5701318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5701408Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5701661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5701732Z return func(*args, **kwargs) 2025-08-14T21:48:27.5701979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5702074Z return func(*args, **kwargs) 2025-08-14T21:48:27.5702331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5702404Z return func(*args, **kwargs) 2025-08-14T21:48:27.5702710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5702787Z self_outputs = self.self( 2025-08-14T21:48:27.5703053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5703124Z return func(*args, **kwargs) 2025-08-14T21:48:27.5703384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5703464Z return func(*args, **kwargs) 2025-08-14T21:48:27.5703733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5703808Z return func(*args, **kwargs) 2025-08-14T21:48:27.5704124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.5704280Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5704284Z 2025-08-14T21:48:27.5704408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5704628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5704701Z return mod(**inputs) 2025-08-14T21:48:27.5704946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5705027Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5705337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5705415Z outputs = self.layoutlm( 2025-08-14T21:48:27.5705710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5705793Z return func(*args, **kwargs) 2025-08-14T21:48:27.5706053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5706123Z return func(*args, **kwargs) 2025-08-14T21:48:27.5706362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5706440Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5706762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5706840Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5707100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5707182Z return func(*args, **kwargs) 2025-08-14T21:48:27.5707433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5707546Z return func(*args, **kwargs) 2025-08-14T21:48:27.5707814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5707885Z return func(*args, **kwargs) 2025-08-14T21:48:27.5707975Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5708203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5708281Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5708569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5708801Z layer_outputs = layer_module( 2025-08-14T21:48:27.5709108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5709200Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5709464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5709547Z return func(*args, **kwargs) 2025-08-14T21:48:27.5709814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5709888Z return func(*args, **kwargs) 2025-08-14T21:48:27.5710159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5710231Z return func(*args, **kwargs) 2025-08-14T21:48:27.5710535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5710627Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5710881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5710965Z return func(*args, **kwargs) 2025-08-14T21:48:27.5711220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5711293Z return func(*args, **kwargs) 2025-08-14T21:48:27.5711563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5711637Z return func(*args, **kwargs) 2025-08-14T21:48:27.5711946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5712024Z self_outputs = self.self( 2025-08-14T21:48:27.5712290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5712417Z return func(*args, **kwargs) 2025-08-14T21:48:27.5712674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5712750Z return func(*args, **kwargs) 2025-08-14T21:48:27.5713013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5713086Z return func(*args, **kwargs) 2025-08-14T21:48:27.5713381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.5713541Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5713546Z 2025-08-14T21:48:27.5713636Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5713732Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5713846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5714071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5714189Z return mod(**inputs) 2025-08-14T21:48:27.5714452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5714543Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5714828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5714903Z outputs = self.layoutlm( 2025-08-14T21:48:27.5715165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5715239Z return func(*args, **kwargs) 2025-08-14T21:48:27.5715500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5715646Z return func(*args, **kwargs) 2025-08-14T21:48:27.5715922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5716016Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5716303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5716384Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5716649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5716722Z return func(*args, **kwargs) 2025-08-14T21:48:27.5716985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5717057Z return func(*args, **kwargs) 2025-08-14T21:48:27.5717310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5717393Z return func(*args, **kwargs) 2025-08-14T21:48:27.5717478Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5717717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5717805Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5718092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5718178Z layer_outputs = layer_module( 2025-08-14T21:48:27.5718415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5718500Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5718767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5718842Z return func(*args, **kwargs) 2025-08-14T21:48:27.5719106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5719213Z return func(*args, **kwargs) 2025-08-14T21:48:27.5719478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5719558Z return func(*args, **kwargs) 2025-08-14T21:48:27.5719851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5719943Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5720208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5720281Z return func(*args, **kwargs) 2025-08-14T21:48:27.5720540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5720627Z return func(*args, **kwargs) 2025-08-14T21:48:27.5720888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5720991Z return func(*args, **kwargs) 2025-08-14T21:48:27.5721302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.5721446Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.5721743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.5721835Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5721839Z 2025-08-14T21:48:27.5721959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5722176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5722249Z return mod(**inputs) 2025-08-14T21:48:27.5722507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5722593Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5722879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5722962Z outputs = self.layoutlm( 2025-08-14T21:48:27.5723215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5723296Z return func(*args, **kwargs) 2025-08-14T21:48:27.5723550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5723624Z return func(*args, **kwargs) 2025-08-14T21:48:27.5723859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5723934Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5724199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5724284Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5724526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5724599Z return func(*args, **kwargs) 2025-08-14T21:48:27.5724825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5724889Z return func(*args, **kwargs) 2025-08-14T21:48:27.5725124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5725189Z return func(*args, **kwargs) 2025-08-14T21:48:27.5725270Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5725477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5725568Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5725834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5725905Z layer_outputs = layer_module( 2025-08-14T21:48:27.5726116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5726202Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5726436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5726509Z return func(*args, **kwargs) 2025-08-14T21:48:27.5726743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5726811Z return func(*args, **kwargs) 2025-08-14T21:48:27.5727054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5727144Z return func(*args, **kwargs) 2025-08-14T21:48:27.5727425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5727522Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5727780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5727863Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5728158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5728283Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5728572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.5728658Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5728663Z 2025-08-14T21:48:27.5728775Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5728975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5729040Z return mod(**inputs) 2025-08-14T21:48:27.5729262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5729338Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5729602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5729678Z outputs = self.layoutlm( 2025-08-14T21:48:27.5729914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5729990Z return func(*args, **kwargs) 2025-08-14T21:48:27.5730227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5730298Z return func(*args, **kwargs) 2025-08-14T21:48:27.5730524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5730600Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5730857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5730934Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5731166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5731238Z return func(*args, **kwargs) 2025-08-14T21:48:27.5731469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5731553Z return func(*args, **kwargs) 2025-08-14T21:48:27.5731791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5731859Z return func(*args, **kwargs) 2025-08-14T21:48:27.5731933Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5732149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5732222Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5732484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5732554Z layer_outputs = layer_module( 2025-08-14T21:48:27.5732765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5732852Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5733083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5733173Z return func(*args, **kwargs) 2025-08-14T21:48:27.5733416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5733484Z return func(*args, **kwargs) 2025-08-14T21:48:27.5733723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5733788Z return func(*args, **kwargs) 2025-08-14T21:48:27.5734046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5734137Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5734411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5734499Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5734794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5734917Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5735195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.5735310Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.5735540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.5735615Z return self.act(input) 2025-08-14T21:48:27.5735619Z 2025-08-14T21:48:27.5735732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5735952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5736035Z return mod(**inputs) 2025-08-14T21:48:27.5736251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5736337Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5736604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5736681Z outputs = self.layoutlm( 2025-08-14T21:48:27.5736916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5736984Z return func(*args, **kwargs) 2025-08-14T21:48:27.5737227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5737297Z return func(*args, **kwargs) 2025-08-14T21:48:27.5737518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5737619Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5737878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5737959Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5738188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5738254Z return func(*args, **kwargs) 2025-08-14T21:48:27.5738492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5738556Z return func(*args, **kwargs) 2025-08-14T21:48:27.5738793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5738858Z return func(*args, **kwargs) 2025-08-14T21:48:27.5738935Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5739152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5739243Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5739515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5739595Z layer_outputs = layer_module( 2025-08-14T21:48:27.5739807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5739891Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5740121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5740187Z return func(*args, **kwargs) 2025-08-14T21:48:27.5740423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5740514Z return func(*args, **kwargs) 2025-08-14T21:48:27.5740742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5740819Z return func(*args, **kwargs) 2025-08-14T21:48:27.5741075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5741166Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5741413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5741491Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5741787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.5741916Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.5742200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.5742286Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5742291Z 2025-08-14T21:48:27.5742395Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5742599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5742664Z return mod(**inputs) 2025-08-14T21:48:27.5742889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5742960Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5743214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5743290Z outputs = self.layoutlm( 2025-08-14T21:48:27.5743523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5743613Z return func(*args, **kwargs) 2025-08-14T21:48:27.5743859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5743928Z return func(*args, **kwargs) 2025-08-14T21:48:27.5744155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5744227Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5744482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5744560Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5744790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5744857Z return func(*args, **kwargs) 2025-08-14T21:48:27.5745101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5745170Z return func(*args, **kwargs) 2025-08-14T21:48:27.5745451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5745519Z return func(*args, **kwargs) 2025-08-14T21:48:27.5745596Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5745817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5745889Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5746155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5746234Z layer_outputs = layer_module( 2025-08-14T21:48:27.5746470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5746560Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5746792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5746861Z return func(*args, **kwargs) 2025-08-14T21:48:27.5747102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5747168Z return func(*args, **kwargs) 2025-08-14T21:48:27.5747410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5747477Z return func(*args, **kwargs) 2025-08-14T21:48:27.5747743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5747833Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5748067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5748136Z return func(*args, **kwargs) 2025-08-14T21:48:27.5748381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5748447Z return func(*args, **kwargs) 2025-08-14T21:48:27.5748691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5748756Z return func(*args, **kwargs) 2025-08-14T21:48:27.5749019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5749096Z self_outputs = self.self( 2025-08-14T21:48:27.5749332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5749398Z return func(*args, **kwargs) 2025-08-14T21:48:27.5749642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5749730Z return func(*args, **kwargs) 2025-08-14T21:48:27.5749976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5750043Z return func(*args, **kwargs) 2025-08-14T21:48:27.5750307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.5750471Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5750475Z 2025-08-14T21:48:27.5750585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5750800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5750870Z return mod(**inputs) 2025-08-14T21:48:27.5751106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5751193Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5751519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5751595Z outputs = self.layoutlm( 2025-08-14T21:48:27.5751863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5751934Z return func(*args, **kwargs) 2025-08-14T21:48:27.5752190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5752262Z return func(*args, **kwargs) 2025-08-14T21:48:27.5752490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5752577Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5752893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5752979Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5753254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5753328Z return func(*args, **kwargs) 2025-08-14T21:48:27.5753593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5753666Z return func(*args, **kwargs) 2025-08-14T21:48:27.5753932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5754013Z return func(*args, **kwargs) 2025-08-14T21:48:27.5754097Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5754331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5754421Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5754708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5754795Z layer_outputs = layer_module( 2025-08-14T21:48:27.5755030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5755116Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5755376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5755449Z return func(*args, **kwargs) 2025-08-14T21:48:27.5755801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5755882Z return func(*args, **kwargs) 2025-08-14T21:48:27.5756142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5756249Z return func(*args, **kwargs) 2025-08-14T21:48:27.5756542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5756635Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5756894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5756961Z return func(*args, **kwargs) 2025-08-14T21:48:27.5757209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5757278Z return func(*args, **kwargs) 2025-08-14T21:48:27.5757512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5757588Z return func(*args, **kwargs) 2025-08-14T21:48:27.5757854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5757943Z self_outputs = self.self( 2025-08-14T21:48:27.5758239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5758309Z return func(*args, **kwargs) 2025-08-14T21:48:27.5758553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5758620Z return func(*args, **kwargs) 2025-08-14T21:48:27.5758858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5758934Z return func(*args, **kwargs) 2025-08-14T21:48:27.5759200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.5759352Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5759366Z 2025-08-14T21:48:27.5759473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5759670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5759742Z return mod(**inputs) 2025-08-14T21:48:27.5759952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5760027Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5760303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5760372Z outputs = self.layoutlm( 2025-08-14T21:48:27.5760615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5760684Z return func(*args, **kwargs) 2025-08-14T21:48:27.5760917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5760995Z return func(*args, **kwargs) 2025-08-14T21:48:27.5761208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5761282Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5761552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5761627Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5761867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5761933Z return func(*args, **kwargs) 2025-08-14T21:48:27.5762165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5762243Z return func(*args, **kwargs) 2025-08-14T21:48:27.5762496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5762567Z return func(*args, **kwargs) 2025-08-14T21:48:27.5762651Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5762885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5762963Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5763228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5763300Z layer_outputs = layer_module( 2025-08-14T21:48:27.5763526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5763604Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5763851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5763919Z return func(*args, **kwargs) 2025-08-14T21:48:27.5764198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5764273Z return func(*args, **kwargs) 2025-08-14T21:48:27.5764506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5764572Z return func(*args, **kwargs) 2025-08-14T21:48:27.5764843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5764926Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5765170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5765252Z return func(*args, **kwargs) 2025-08-14T21:48:27.5765492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5765567Z return func(*args, **kwargs) 2025-08-14T21:48:27.5765806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5765872Z return func(*args, **kwargs) 2025-08-14T21:48:27.5766148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5766217Z self_outputs = self.self( 2025-08-14T21:48:27.5766463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5766529Z return func(*args, **kwargs) 2025-08-14T21:48:27.5766767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5766840Z return func(*args, **kwargs) 2025-08-14T21:48:27.5767077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5767148Z return func(*args, **kwargs) 2025-08-14T21:48:27.5767423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.5767567Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5767571Z 2025-08-14T21:48:27.5767661Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5767739Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5767844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5768051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5768116Z return mod(**inputs) 2025-08-14T21:48:27.5768336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5768430Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5768703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5768781Z outputs = self.layoutlm( 2025-08-14T21:48:27.5769018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5769087Z return func(*args, **kwargs) 2025-08-14T21:48:27.5769333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5769400Z return func(*args, **kwargs) 2025-08-14T21:48:27.5769623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5769698Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5769968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5770069Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5770321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5770390Z return func(*args, **kwargs) 2025-08-14T21:48:27.5770636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5770702Z return func(*args, **kwargs) 2025-08-14T21:48:27.5770947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5771013Z return func(*args, **kwargs) 2025-08-14T21:48:27.5771087Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5771323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5771401Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5771666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5771745Z layer_outputs = layer_module( 2025-08-14T21:48:27.5771960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5772044Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5772278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5772345Z return func(*args, **kwargs) 2025-08-14T21:48:27.5772588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5772654Z return func(*args, **kwargs) 2025-08-14T21:48:27.5772890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5772964Z return func(*args, **kwargs) 2025-08-14T21:48:27.5773229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5773323Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5773570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5773640Z return func(*args, **kwargs) 2025-08-14T21:48:27.5773904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5773971Z return func(*args, **kwargs) 2025-08-14T21:48:27.5774212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5774281Z return func(*args, **kwargs) 2025-08-14T21:48:27.5774580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.5774728Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.5775008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.5775097Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5775109Z 2025-08-14T21:48:27.5775223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5775421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5775494Z return mod(**inputs) 2025-08-14T21:48:27.5775706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5775780Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5776054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5776144Z outputs = self.layoutlm( 2025-08-14T21:48:27.5776408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5776478Z return func(*args, **kwargs) 2025-08-14T21:48:27.5776713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5776788Z return func(*args, **kwargs) 2025-08-14T21:48:27.5777001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5777076Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5777344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5777436Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5777683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5777754Z return func(*args, **kwargs) 2025-08-14T21:48:27.5777986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5778060Z return func(*args, **kwargs) 2025-08-14T21:48:27.5778293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5778359Z return func(*args, **kwargs) 2025-08-14T21:48:27.5778442Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5778655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5778734Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5778999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5779072Z layer_outputs = layer_module( 2025-08-14T21:48:27.5779299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5779376Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5779619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5779696Z return func(*args, **kwargs) 2025-08-14T21:48:27.5779941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5780017Z return func(*args, **kwargs) 2025-08-14T21:48:27.5780263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5780330Z return func(*args, **kwargs) 2025-08-14T21:48:27.5780601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5780709Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5780986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5781068Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5781378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5781511Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5781788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.5781876Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5781887Z 2025-08-14T21:48:27.5781996Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5782206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5782300Z return mod(**inputs) 2025-08-14T21:48:27.5782544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5782632Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5782902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5782971Z outputs = self.layoutlm( 2025-08-14T21:48:27.5783213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5783293Z return func(*args, **kwargs) 2025-08-14T21:48:27.5783544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5783642Z return func(*args, **kwargs) 2025-08-14T21:48:27.5783870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5783954Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5784242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5784319Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5784573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5784645Z return func(*args, **kwargs) 2025-08-14T21:48:27.5784894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5784971Z return func(*args, **kwargs) 2025-08-14T21:48:27.5785220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5785294Z return func(*args, **kwargs) 2025-08-14T21:48:27.5785384Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5785612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5785695Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5785976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5786053Z layer_outputs = layer_module( 2025-08-14T21:48:27.5786290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5786374Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5786620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5786702Z return func(*args, **kwargs) 2025-08-14T21:48:27.5786949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5787050Z return func(*args, **kwargs) 2025-08-14T21:48:27.5787297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5787367Z return func(*args, **kwargs) 2025-08-14T21:48:27.5787652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5787741Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5788012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5788101Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5788414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5788547Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5788882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.5789004Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.5789234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.5789307Z return self.act(input) 2025-08-14T21:48:27.5789314Z 2025-08-14T21:48:27.5789429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5789639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5789707Z return mod(**inputs) 2025-08-14T21:48:27.5789941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5790039Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5790327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5790414Z outputs = self.layoutlm( 2025-08-14T21:48:27.5790671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5790753Z return func(*args, **kwargs) 2025-08-14T21:48:27.5791007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5791080Z return func(*args, **kwargs) 2025-08-14T21:48:27.5791321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5791401Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5791694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5791775Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5792031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5792112Z return func(*args, **kwargs) 2025-08-14T21:48:27.5792364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5792435Z return func(*args, **kwargs) 2025-08-14T21:48:27.5792699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5792771Z return func(*args, **kwargs) 2025-08-14T21:48:27.5792861Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5793092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5793171Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5793464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5793562Z layer_outputs = layer_module( 2025-08-14T21:48:27.5793799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5793893Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5794148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5794228Z return func(*args, **kwargs) 2025-08-14T21:48:27.5794483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5794555Z return func(*args, **kwargs) 2025-08-14T21:48:27.5794817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5794891Z return func(*args, **kwargs) 2025-08-14T21:48:27.5795180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5795320Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5795673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5795775Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5796098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.5796244Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.5796558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.5796647Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5796674Z 2025-08-14T21:48:27.5796800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5797018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5797103Z return mod(**inputs) 2025-08-14T21:48:27.5797338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5797420Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5797712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5797793Z outputs = self.layoutlm( 2025-08-14T21:48:27.5798043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5798126Z return func(*args, **kwargs) 2025-08-14T21:48:27.5798380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5798454Z return func(*args, **kwargs) 2025-08-14T21:48:27.5798693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5798771Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5799059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5799137Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5799385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5799465Z return func(*args, **kwargs) 2025-08-14T21:48:27.5799714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5799787Z return func(*args, **kwargs) 2025-08-14T21:48:27.5800046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5800140Z return func(*args, **kwargs) 2025-08-14T21:48:27.5800232Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5800458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5800536Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5800821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5800895Z layer_outputs = layer_module( 2025-08-14T21:48:27.5801123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5801214Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5801463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5801542Z return func(*args, **kwargs) 2025-08-14T21:48:27.5801789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5801898Z return func(*args, **kwargs) 2025-08-14T21:48:27.5802159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5802230Z return func(*args, **kwargs) 2025-08-14T21:48:27.5802531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5802635Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5802887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5802964Z return func(*args, **kwargs) 2025-08-14T21:48:27.5803233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5803308Z return func(*args, **kwargs) 2025-08-14T21:48:27.5803575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5803649Z return func(*args, **kwargs) 2025-08-14T21:48:27.5803952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5804036Z self_outputs = self.self( 2025-08-14T21:48:27.5804294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5804374Z return func(*args, **kwargs) 2025-08-14T21:48:27.5804633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5804706Z return func(*args, **kwargs) 2025-08-14T21:48:27.5804979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5805051Z return func(*args, **kwargs) 2025-08-14T21:48:27.5805346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.5805507Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5805513Z 2025-08-14T21:48:27.5805626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5805849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5805920Z return mod(**inputs) 2025-08-14T21:48:27.5806155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5806244Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5806538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5806637Z outputs = self.layoutlm( 2025-08-14T21:48:27.5806902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5806975Z return func(*args, **kwargs) 2025-08-14T21:48:27.5807245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5807318Z return func(*args, **kwargs) 2025-08-14T21:48:27.5807557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5807645Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5807940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5808025Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5808291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5808385Z return func(*args, **kwargs) 2025-08-14T21:48:27.5808860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5808949Z return func(*args, **kwargs) 2025-08-14T21:48:27.5809219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5809297Z return func(*args, **kwargs) 2025-08-14T21:48:27.5809384Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5809630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5809716Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5810040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5810134Z layer_outputs = layer_module( 2025-08-14T21:48:27.5810379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5810478Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5810741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5810815Z return func(*args, **kwargs) 2025-08-14T21:48:27.5811079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5811153Z return func(*args, **kwargs) 2025-08-14T21:48:27.5811418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5811500Z return func(*args, **kwargs) 2025-08-14T21:48:27.5811796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5811895Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5812159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5812231Z return func(*args, **kwargs) 2025-08-14T21:48:27.5812498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5812572Z return func(*args, **kwargs) 2025-08-14T21:48:27.5812829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5812910Z return func(*args, **kwargs) 2025-08-14T21:48:27.5813203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5813288Z self_outputs = self.self( 2025-08-14T21:48:27.5813550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5813665Z return func(*args, **kwargs) 2025-08-14T21:48:27.5813932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5814004Z return func(*args, **kwargs) 2025-08-14T21:48:27.5814269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5814342Z return func(*args, **kwargs) 2025-08-14T21:48:27.5814629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.5814789Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5814793Z 2025-08-14T21:48:27.5814905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5815124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5815204Z return mod(**inputs) 2025-08-14T21:48:27.5815482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5815584Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5815861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5815935Z outputs = self.layoutlm( 2025-08-14T21:48:27.5816193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5816264Z return func(*args, **kwargs) 2025-08-14T21:48:27.5816512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5816594Z return func(*args, **kwargs) 2025-08-14T21:48:27.5816840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5816932Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5817215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5817291Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5817534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5817600Z return func(*args, **kwargs) 2025-08-14T21:48:27.5817843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5817912Z return func(*args, **kwargs) 2025-08-14T21:48:27.5818152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5818228Z return func(*args, **kwargs) 2025-08-14T21:48:27.5818307Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5818536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5818627Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5818906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5818988Z layer_outputs = layer_module( 2025-08-14T21:48:27.5819218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5819300Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5819560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5819631Z return func(*args, **kwargs) 2025-08-14T21:48:27.5819880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5819981Z return func(*args, **kwargs) 2025-08-14T21:48:27.5820234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5820313Z return func(*args, **kwargs) 2025-08-14T21:48:27.5820592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5820680Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5820938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5821007Z return func(*args, **kwargs) 2025-08-14T21:48:27.5821255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5821336Z return func(*args, **kwargs) 2025-08-14T21:48:27.5821585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5821686Z return func(*args, **kwargs) 2025-08-14T21:48:27.5821982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5822061Z self_outputs = self.self( 2025-08-14T21:48:27.5822317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5822388Z return func(*args, **kwargs) 2025-08-14T21:48:27.5822645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5822716Z return func(*args, **kwargs) 2025-08-14T21:48:27.5822964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5823063Z return func(*args, **kwargs) 2025-08-14T21:48:27.5823345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.5823502Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5823506Z 2025-08-14T21:48:27.5823599Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5823682Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5823797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5824006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5824075Z return mod(**inputs) 2025-08-14T21:48:27.5824309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5824387Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5824666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5824748Z outputs = self.layoutlm( 2025-08-14T21:48:27.5825001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5825080Z return func(*args, **kwargs) 2025-08-14T21:48:27.5825329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5825400Z return func(*args, **kwargs) 2025-08-14T21:48:27.5825632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5825710Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5826006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5826089Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5826345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5826446Z return func(*args, **kwargs) 2025-08-14T21:48:27.5826708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5826775Z return func(*args, **kwargs) 2025-08-14T21:48:27.5827019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5827085Z return func(*args, **kwargs) 2025-08-14T21:48:27.5827161Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5827385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5827458Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5827734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5827806Z layer_outputs = layer_module( 2025-08-14T21:48:27.5828043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5828147Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5828381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5828453Z return func(*args, **kwargs) 2025-08-14T21:48:27.5828688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5828755Z return func(*args, **kwargs) 2025-08-14T21:48:27.5828995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5829062Z return func(*args, **kwargs) 2025-08-14T21:48:27.5829345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5829439Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5829674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5829751Z return func(*args, **kwargs) 2025-08-14T21:48:27.5829984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5830051Z return func(*args, **kwargs) 2025-08-14T21:48:27.5830293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5830360Z return func(*args, **kwargs) 2025-08-14T21:48:27.5830624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.5830760Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.5831031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.5831130Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5831134Z 2025-08-14T21:48:27.5831246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5831452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5831532Z return mod(**inputs) 2025-08-14T21:48:27.5831757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5831843Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5832132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5832206Z outputs = self.layoutlm( 2025-08-14T21:48:27.5832467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5832560Z return func(*args, **kwargs) 2025-08-14T21:48:27.5832814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5832895Z return func(*args, **kwargs) 2025-08-14T21:48:27.5833125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5833214Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5833511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5833592Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5833856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5833931Z return func(*args, **kwargs) 2025-08-14T21:48:27.5834186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5834305Z return func(*args, **kwargs) 2025-08-14T21:48:27.5834578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5834659Z return func(*args, **kwargs) 2025-08-14T21:48:27.5834742Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5834975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5835063Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5835381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5835465Z layer_outputs = layer_module( 2025-08-14T21:48:27.5835809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5835906Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5836180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5836253Z return func(*args, **kwargs) 2025-08-14T21:48:27.5836510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5836593Z return func(*args, **kwargs) 2025-08-14T21:48:27.5836850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5836930Z return func(*args, **kwargs) 2025-08-14T21:48:27.5837235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5837327Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5837620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5837708Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5838040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5838183Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5838474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.5838574Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5838578Z 2025-08-14T21:48:27.5838692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5838910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5838990Z return mod(**inputs) 2025-08-14T21:48:27.5839226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5839337Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5839623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5839699Z outputs = self.layoutlm( 2025-08-14T21:48:27.5839961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5840034Z return func(*args, **kwargs) 2025-08-14T21:48:27.5840287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5840369Z return func(*args, **kwargs) 2025-08-14T21:48:27.5840603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5840690Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5840979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5841098Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5841368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5841442Z return func(*args, **kwargs) 2025-08-14T21:48:27.5841702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5841782Z return func(*args, **kwargs) 2025-08-14T21:48:27.5842041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5842121Z return func(*args, **kwargs) 2025-08-14T21:48:27.5842203Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5842457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5842548Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5842843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5842920Z layer_outputs = layer_module( 2025-08-14T21:48:27.5843167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5843253Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5843519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5843593Z return func(*args, **kwargs) 2025-08-14T21:48:27.5843850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5843931Z return func(*args, **kwargs) 2025-08-14T21:48:27.5844191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5844277Z return func(*args, **kwargs) 2025-08-14T21:48:27.5844567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5844660Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5844950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5845035Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5845356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5845495Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5845788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.5845926Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.5846142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.5846212Z return self.act(input) 2025-08-14T21:48:27.5846215Z 2025-08-14T21:48:27.5846325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5846522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5846596Z return mod(**inputs) 2025-08-14T21:48:27.5846811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5846886Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5847154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5847227Z outputs = self.layoutlm( 2025-08-14T21:48:27.5847463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5847577Z return func(*args, **kwargs) 2025-08-14T21:48:27.5847834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5847913Z return func(*args, **kwargs) 2025-08-14T21:48:27.5848140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5848230Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5848503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5848577Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5848835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5848913Z return func(*args, **kwargs) 2025-08-14T21:48:27.5849146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5849223Z return func(*args, **kwargs) 2025-08-14T21:48:27.5849460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5849527Z return func(*args, **kwargs) 2025-08-14T21:48:27.5849613Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5849825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5849897Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5850174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5850251Z layer_outputs = layer_module( 2025-08-14T21:48:27.5850486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5850572Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5850821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5850901Z return func(*args, **kwargs) 2025-08-14T21:48:27.5851149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5851228Z return func(*args, **kwargs) 2025-08-14T21:48:27.5851474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5851545Z return func(*args, **kwargs) 2025-08-14T21:48:27.5851829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5851932Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5852220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5852307Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5852609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.5852748Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.5853018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.5853102Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5853105Z 2025-08-14T21:48:27.5853220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5853434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5853512Z return mod(**inputs) 2025-08-14T21:48:27.5853742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5853863Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5854149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5854224Z outputs = self.layoutlm( 2025-08-14T21:48:27.5854474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5854553Z return func(*args, **kwargs) 2025-08-14T21:48:27.5854800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5854877Z return func(*args, **kwargs) 2025-08-14T21:48:27.5855119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5855209Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5855482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5855558Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5855792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5855868Z return func(*args, **kwargs) 2025-08-14T21:48:27.5856101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5856175Z return func(*args, **kwargs) 2025-08-14T21:48:27.5856407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5856476Z return func(*args, **kwargs) 2025-08-14T21:48:27.5856560Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5856776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5856852Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5857122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5857194Z layer_outputs = layer_module( 2025-08-14T21:48:27.5857415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5857494Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5857730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5857806Z return func(*args, **kwargs) 2025-08-14T21:48:27.5858041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5858108Z return func(*args, **kwargs) 2025-08-14T21:48:27.5858368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5858438Z return func(*args, **kwargs) 2025-08-14T21:48:27.5858710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5858793Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5859028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5859103Z return func(*args, **kwargs) 2025-08-14T21:48:27.5859339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5859413Z return func(*args, **kwargs) 2025-08-14T21:48:27.5859649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5859718Z return func(*args, **kwargs) 2025-08-14T21:48:27.5860056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5860129Z self_outputs = self.self( 2025-08-14T21:48:27.5860368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5860444Z return func(*args, **kwargs) 2025-08-14T21:48:27.5860681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5860760Z return func(*args, **kwargs) 2025-08-14T21:48:27.5861008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5861080Z return func(*args, **kwargs) 2025-08-14T21:48:27.5861385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.5861543Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5861550Z 2025-08-14T21:48:27.5861673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5861888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5861964Z return mod(**inputs) 2025-08-14T21:48:27.5862203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5862286Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5862577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5862661Z outputs = self.layoutlm( 2025-08-14T21:48:27.5862904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5862986Z return func(*args, **kwargs) 2025-08-14T21:48:27.5863232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5863301Z return func(*args, **kwargs) 2025-08-14T21:48:27.5863528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5863607Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5863877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5863968Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5864222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5864305Z return func(*args, **kwargs) 2025-08-14T21:48:27.5864560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5864654Z return func(*args, **kwargs) 2025-08-14T21:48:27.5864910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5864982Z return func(*args, **kwargs) 2025-08-14T21:48:27.5865070Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5865301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5865378Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5865679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5865753Z layer_outputs = layer_module( 2025-08-14T21:48:27.5865983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5866076Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5866340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5867353Z return func(*args, **kwargs) 2025-08-14T21:48:27.5867619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5867690Z return func(*args, **kwargs) 2025-08-14T21:48:27.5867942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5868013Z return func(*args, **kwargs) 2025-08-14T21:48:27.5868292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5868388Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5868655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5868737Z return func(*args, **kwargs) 2025-08-14T21:48:27.5868990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5869063Z return func(*args, **kwargs) 2025-08-14T21:48:27.5869322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5869393Z return func(*args, **kwargs) 2025-08-14T21:48:27.5869686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5869770Z self_outputs = self.self( 2025-08-14T21:48:27.5870031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5870109Z return func(*args, **kwargs) 2025-08-14T21:48:27.5870368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5870441Z return func(*args, **kwargs) 2025-08-14T21:48:27.5870703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5870775Z return func(*args, **kwargs) 2025-08-14T21:48:27.5871064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.5871218Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5871223Z 2025-08-14T21:48:27.5871333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5871547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5871616Z return mod(**inputs) 2025-08-14T21:48:27.5871843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5871953Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5872245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5872325Z outputs = self.layoutlm( 2025-08-14T21:48:27.5872580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5872651Z return func(*args, **kwargs) 2025-08-14T21:48:27.5872907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5872977Z return func(*args, **kwargs) 2025-08-14T21:48:27.5873204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5873291Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5873590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5873700Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5873988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5874064Z return func(*args, **kwargs) 2025-08-14T21:48:27.5874335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5874408Z return func(*args, **kwargs) 2025-08-14T21:48:27.5874672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5874752Z return func(*args, **kwargs) 2025-08-14T21:48:27.5874835Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5875091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5875177Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5875480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5875567Z layer_outputs = layer_module( 2025-08-14T21:48:27.5876128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5876220Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5876492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5876566Z return func(*args, **kwargs) 2025-08-14T21:48:27.5876832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5876906Z return func(*args, **kwargs) 2025-08-14T21:48:27.5877203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5877286Z return func(*args, **kwargs) 2025-08-14T21:48:27.5877580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5877676Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5877922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5877993Z return func(*args, **kwargs) 2025-08-14T21:48:27.5878246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5878319Z return func(*args, **kwargs) 2025-08-14T21:48:27.5878566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5878649Z return func(*args, **kwargs) 2025-08-14T21:48:27.5878927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5879037Z self_outputs = self.self( 2025-08-14T21:48:27.5879290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5879362Z return func(*args, **kwargs) 2025-08-14T21:48:27.5879617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5879690Z return func(*args, **kwargs) 2025-08-14T21:48:27.5879939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5880019Z return func(*args, **kwargs) 2025-08-14T21:48:27.5880297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.5880458Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5880464Z 2025-08-14T21:48:27.5880572Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5880676Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5880796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5881005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5881083Z return mod(**inputs) 2025-08-14T21:48:27.5881311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5881391Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5881676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5881750Z outputs = self.layoutlm( 2025-08-14T21:48:27.5882015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5882099Z return func(*args, **kwargs) 2025-08-14T21:48:27.5882351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5882429Z return func(*args, **kwargs) 2025-08-14T21:48:27.5882656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5882733Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5883028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5883104Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5883353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5883431Z return func(*args, **kwargs) 2025-08-14T21:48:27.5883685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5883766Z return func(*args, **kwargs) 2025-08-14T21:48:27.5884017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5884088Z return func(*args, **kwargs) 2025-08-14T21:48:27.5884173Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5884401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5884477Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5884763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5884839Z layer_outputs = layer_module( 2025-08-14T21:48:27.5885081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5885185Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5885434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5885514Z return func(*args, **kwargs) 2025-08-14T21:48:27.5885761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5885832Z return func(*args, **kwargs) 2025-08-14T21:48:27.5886087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5886157Z return func(*args, **kwargs) 2025-08-14T21:48:27.5886441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5886528Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5886776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5886858Z return func(*args, **kwargs) 2025-08-14T21:48:27.5887158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5887239Z return func(*args, **kwargs) 2025-08-14T21:48:27.5887492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5887562Z return func(*args, **kwargs) 2025-08-14T21:48:27.5887869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.5888006Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.5888300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.5888415Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5888421Z 2025-08-14T21:48:27.5888531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5888749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5888818Z return mod(**inputs) 2025-08-14T21:48:27.5889045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5889131Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5889409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5889489Z outputs = self.layoutlm( 2025-08-14T21:48:27.5889738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5889809Z return func(*args, **kwargs) 2025-08-14T21:48:27.5890068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5890143Z return func(*args, **kwargs) 2025-08-14T21:48:27.5890372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5890461Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5890740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5890824Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5891076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5891147Z return func(*args, **kwargs) 2025-08-14T21:48:27.5891404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5891477Z return func(*args, **kwargs) 2025-08-14T21:48:27.5891746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5891829Z return func(*args, **kwargs) 2025-08-14T21:48:27.5891911Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5892146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5892224Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5892502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5892587Z layer_outputs = layer_module( 2025-08-14T21:48:27.5892816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5892901Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5893164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5893238Z return func(*args, **kwargs) 2025-08-14T21:48:27.5893539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5893614Z return func(*args, **kwargs) 2025-08-14T21:48:27.5893863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5893941Z return func(*args, **kwargs) 2025-08-14T21:48:27.5894231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5894322Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5894609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5894713Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5895037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5895170Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5895449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.5895545Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5895549Z 2025-08-14T21:48:27.5895659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5895876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5895947Z return mod(**inputs) 2025-08-14T21:48:27.5896173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5896259Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5896543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5896620Z outputs = self.layoutlm( 2025-08-14T21:48:27.5896881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5896955Z return func(*args, **kwargs) 2025-08-14T21:48:27.5897213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5897285Z return func(*args, **kwargs) 2025-08-14T21:48:27.5897512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5897601Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5897879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5897965Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5898234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5898308Z return func(*args, **kwargs) 2025-08-14T21:48:27.5898565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5898637Z return func(*args, **kwargs) 2025-08-14T21:48:27.5898883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5898961Z return func(*args, **kwargs) 2025-08-14T21:48:27.5899041Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5899273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5899352Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5899649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5899732Z layer_outputs = layer_module( 2025-08-14T21:48:27.5900000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5900085Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5900349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5900422Z return func(*args, **kwargs) 2025-08-14T21:48:27.5900677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5900748Z return func(*args, **kwargs) 2025-08-14T21:48:27.5900993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5901074Z return func(*args, **kwargs) 2025-08-14T21:48:27.5901369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5901464Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5901745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5901827Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5902146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5902272Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5902551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.5902680Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.5902903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.5902987Z return self.act(input) 2025-08-14T21:48:27.5902992Z 2025-08-14T21:48:27.5903104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5903311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5903388Z return mod(**inputs) 2025-08-14T21:48:27.5903614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5903690Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5903976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5904050Z outputs = self.layoutlm( 2025-08-14T21:48:27.5904309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5904382Z return func(*args, **kwargs) 2025-08-14T21:48:27.5904651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5904735Z return func(*args, **kwargs) 2025-08-14T21:48:27.5904961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5905040Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5905328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5905407Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5905665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5905736Z return func(*args, **kwargs) 2025-08-14T21:48:27.5905985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5906065Z return func(*args, **kwargs) 2025-08-14T21:48:27.5906336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5906433Z return func(*args, **kwargs) 2025-08-14T21:48:27.5906515Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5906739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5906823Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5907101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5907177Z layer_outputs = layer_module( 2025-08-14T21:48:27.5907414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5907499Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5907776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5907853Z return func(*args, **kwargs) 2025-08-14T21:48:27.5908104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5908183Z return func(*args, **kwargs) 2025-08-14T21:48:27.5908432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5908504Z return func(*args, **kwargs) 2025-08-14T21:48:27.5908982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5909080Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5909364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5909445Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5909758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.5909909Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.5910188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.5910286Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5910290Z 2025-08-14T21:48:27.5910404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5910612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5910690Z return mod(**inputs) 2025-08-14T21:48:27.5910919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5911001Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5911369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5911447Z outputs = self.layoutlm( 2025-08-14T21:48:27.5911707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5911780Z return func(*args, **kwargs) 2025-08-14T21:48:27.5912042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5912123Z return func(*args, **kwargs) 2025-08-14T21:48:27.5912399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5912477Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5912784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5912866Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5913160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5913257Z return func(*args, **kwargs) 2025-08-14T21:48:27.5913514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5913594Z return func(*args, **kwargs) 2025-08-14T21:48:27.5913868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5913950Z return func(*args, **kwargs) 2025-08-14T21:48:27.5914034Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5914269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5914378Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5914677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5914757Z layer_outputs = layer_module( 2025-08-14T21:48:27.5915014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5915102Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5915374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5915448Z return func(*args, **kwargs) 2025-08-14T21:48:27.5915767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5915854Z return func(*args, **kwargs) 2025-08-14T21:48:27.5916124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5916200Z return func(*args, **kwargs) 2025-08-14T21:48:27.5916511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5916604Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5916868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5916940Z return func(*args, **kwargs) 2025-08-14T21:48:27.5917206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5917286Z return func(*args, **kwargs) 2025-08-14T21:48:27.5917550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5917623Z return func(*args, **kwargs) 2025-08-14T21:48:27.5917919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5918020Z self_outputs = self.self( 2025-08-14T21:48:27.5918289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5918363Z return func(*args, **kwargs) 2025-08-14T21:48:27.5918621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5918703Z return func(*args, **kwargs) 2025-08-14T21:48:27.5918972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5919055Z return func(*args, **kwargs) 2025-08-14T21:48:27.5919342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.5919503Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5919508Z 2025-08-14T21:48:27.5919629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5919867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5919961Z return mod(**inputs) 2025-08-14T21:48:27.5920200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5920281Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5920571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5920647Z outputs = self.layoutlm( 2025-08-14T21:48:27.5920904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5920985Z return func(*args, **kwargs) 2025-08-14T21:48:27.5921257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5921333Z return func(*args, **kwargs) 2025-08-14T21:48:27.5921576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5921657Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5921953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5922031Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5922287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5922368Z return func(*args, **kwargs) 2025-08-14T21:48:27.5922626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5922708Z return func(*args, **kwargs) 2025-08-14T21:48:27.5922970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5923043Z return func(*args, **kwargs) 2025-08-14T21:48:27.5923139Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5923375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5923453Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5923751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5923828Z layer_outputs = layer_module( 2025-08-14T21:48:27.5924077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5924162Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5924427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5924536Z return func(*args, **kwargs) 2025-08-14T21:48:27.5924794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5924870Z return func(*args, **kwargs) 2025-08-14T21:48:27.5925147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5925217Z return func(*args, **kwargs) 2025-08-14T21:48:27.5925519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5925607Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5925856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5925935Z return func(*args, **kwargs) 2025-08-14T21:48:27.5926183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5926257Z return func(*args, **kwargs) 2025-08-14T21:48:27.5926555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5926627Z return func(*args, **kwargs) 2025-08-14T21:48:27.5926913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5926987Z self_outputs = self.self( 2025-08-14T21:48:27.5927234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5927316Z return func(*args, **kwargs) 2025-08-14T21:48:27.5927567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5927646Z return func(*args, **kwargs) 2025-08-14T21:48:27.5927912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5927988Z return func(*args, **kwargs) 2025-08-14T21:48:27.5928278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.5928424Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5928428Z 2025-08-14T21:48:27.5928538Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5928755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5928825Z return mod(**inputs) 2025-08-14T21:48:27.5929059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5929139Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5929437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5929527Z outputs = self.layoutlm( 2025-08-14T21:48:27.5929779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5929850Z return func(*args, **kwargs) 2025-08-14T21:48:27.5930105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5930175Z return func(*args, **kwargs) 2025-08-14T21:48:27.5930406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5930484Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5930762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5930846Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5931096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5931195Z return func(*args, **kwargs) 2025-08-14T21:48:27.5931445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5931516Z return func(*args, **kwargs) 2025-08-14T21:48:27.5931772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5931843Z return func(*args, **kwargs) 2025-08-14T21:48:27.5931924Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5932157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5932235Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5932523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5932599Z layer_outputs = layer_module( 2025-08-14T21:48:27.5932849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5932959Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5933205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5933276Z return func(*args, **kwargs) 2025-08-14T21:48:27.5933530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5933602Z return func(*args, **kwargs) 2025-08-14T21:48:27.5933855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5933926Z return func(*args, **kwargs) 2025-08-14T21:48:27.5934252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5934354Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5934603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5934675Z return func(*args, **kwargs) 2025-08-14T21:48:27.5934927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5934997Z return func(*args, **kwargs) 2025-08-14T21:48:27.5935251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5935322Z return func(*args, **kwargs) 2025-08-14T21:48:27.5935597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5935681Z self_outputs = self.self( 2025-08-14T21:48:27.5935927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5936006Z return func(*args, **kwargs) 2025-08-14T21:48:27.5936253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5936323Z return func(*args, **kwargs) 2025-08-14T21:48:27.5936580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5936646Z return func(*args, **kwargs) 2025-08-14T21:48:27.5936905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.5937055Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5937058Z 2025-08-14T21:48:27.5937138Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5937226Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5937348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5937551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5937623Z return mod(**inputs) 2025-08-14T21:48:27.5937837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5937911Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5938180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5938248Z outputs = self.layoutlm( 2025-08-14T21:48:27.5938491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5938558Z return func(*args, **kwargs) 2025-08-14T21:48:27.5938794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5938869Z return func(*args, **kwargs) 2025-08-14T21:48:27.5939116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5939190Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5939464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5939536Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5939782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5939848Z return func(*args, **kwargs) 2025-08-14T21:48:27.5940082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5940156Z return func(*args, **kwargs) 2025-08-14T21:48:27.5940408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5940487Z return func(*args, **kwargs) 2025-08-14T21:48:27.5940565Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5940781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5940860Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5941125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5941197Z layer_outputs = layer_module( 2025-08-14T21:48:27.5941423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5941502Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5941919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5941992Z return func(*args, **kwargs) 2025-08-14T21:48:27.5942232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5942313Z return func(*args, **kwargs) 2025-08-14T21:48:27.5942550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5942620Z return func(*args, **kwargs) 2025-08-14T21:48:27.5942894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5942976Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5943230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5943303Z return func(*args, **kwargs) 2025-08-14T21:48:27.5943549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5943659Z return func(*args, **kwargs) 2025-08-14T21:48:27.5943901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5943969Z return func(*args, **kwargs) 2025-08-14T21:48:27.5944243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.5944375Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.5944664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.5944757Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5944764Z 2025-08-14T21:48:27.5944874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5945102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5945171Z return mod(**inputs) 2025-08-14T21:48:27.5945430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5945507Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5945770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5945846Z outputs = self.layoutlm( 2025-08-14T21:48:27.5946081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5946148Z return func(*args, **kwargs) 2025-08-14T21:48:27.5946390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5946459Z return func(*args, **kwargs) 2025-08-14T21:48:27.5946701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5946779Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5947042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5947122Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5947357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5947424Z return func(*args, **kwargs) 2025-08-14T21:48:27.5947676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5947748Z return func(*args, **kwargs) 2025-08-14T21:48:27.5948000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5948073Z return func(*args, **kwargs) 2025-08-14T21:48:27.5948157Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5948386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5948466Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5948744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5948826Z layer_outputs = layer_module( 2025-08-14T21:48:27.5949055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5949146Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5949395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5949466Z return func(*args, **kwargs) 2025-08-14T21:48:27.5949721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5949811Z return func(*args, **kwargs) 2025-08-14T21:48:27.5950072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5950144Z return func(*args, **kwargs) 2025-08-14T21:48:27.5950425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5950524Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5950803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5950884Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5951210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5951342Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5951629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.5951756Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5951761Z 2025-08-14T21:48:27.5951870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5952087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5952156Z return mod(**inputs) 2025-08-14T21:48:27.5952388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5952465Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5952744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5952825Z outputs = self.layoutlm( 2025-08-14T21:48:27.5953097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5953177Z return func(*args, **kwargs) 2025-08-14T21:48:27.5953439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5953512Z return func(*args, **kwargs) 2025-08-14T21:48:27.5953751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5953830Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5954120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5954208Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5954468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5954544Z return func(*args, **kwargs) 2025-08-14T21:48:27.5954814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5954892Z return func(*args, **kwargs) 2025-08-14T21:48:27.5955160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5955236Z return func(*args, **kwargs) 2025-08-14T21:48:27.5955319Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5955565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5955697Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5955992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5956077Z layer_outputs = layer_module( 2025-08-14T21:48:27.5956317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5956435Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5956693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5956767Z return func(*args, **kwargs) 2025-08-14T21:48:27.5957036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5957107Z return func(*args, **kwargs) 2025-08-14T21:48:27.5957361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5957432Z return func(*args, **kwargs) 2025-08-14T21:48:27.5957708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5957808Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5958079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5958181Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5958532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.5958665Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.5958950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.5959070Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.5959290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.5959374Z return self.act(input) 2025-08-14T21:48:27.5959378Z 2025-08-14T21:48:27.5959504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5959722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5959793Z return mod(**inputs) 2025-08-14T21:48:27.5960021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5960106Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5960386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5960462Z outputs = self.layoutlm( 2025-08-14T21:48:27.5960719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5960790Z return func(*args, **kwargs) 2025-08-14T21:48:27.5961045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5961117Z return func(*args, **kwargs) 2025-08-14T21:48:27.5961342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5961429Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5961707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5961784Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5962040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5962112Z return func(*args, **kwargs) 2025-08-14T21:48:27.5962364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5962434Z return func(*args, **kwargs) 2025-08-14T21:48:27.5962680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5962812Z return func(*args, **kwargs) 2025-08-14T21:48:27.5962893Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5963120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5963208Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5963485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5963569Z layer_outputs = layer_module( 2025-08-14T21:48:27.5963796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5963879Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5964136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5964207Z return func(*args, **kwargs) 2025-08-14T21:48:27.5964454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5964556Z return func(*args, **kwargs) 2025-08-14T21:48:27.5964818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5964898Z return func(*args, **kwargs) 2025-08-14T21:48:27.5965181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.5965273Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.5965553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.5965635Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.5965968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.5966113Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.5966393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.5966490Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.5966494Z 2025-08-14T21:48:27.5966602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5966816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5966885Z return mod(**inputs) 2025-08-14T21:48:27.5967109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5967193Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5967467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5967542Z outputs = self.layoutlm( 2025-08-14T21:48:27.5967807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5967878Z return func(*args, **kwargs) 2025-08-14T21:48:27.5968117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5968185Z return func(*args, **kwargs) 2025-08-14T21:48:27.5968395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5968475Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5968735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5968808Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5969052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5969137Z return func(*args, **kwargs) 2025-08-14T21:48:27.5969385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5969453Z return func(*args, **kwargs) 2025-08-14T21:48:27.5969693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5969768Z return func(*args, **kwargs) 2025-08-14T21:48:27.5969847Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5970065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5970145Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5970426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5970509Z layer_outputs = layer_module( 2025-08-14T21:48:27.5970746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5970850Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5971125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5971199Z return func(*args, **kwargs) 2025-08-14T21:48:27.5971448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5971528Z return func(*args, **kwargs) 2025-08-14T21:48:27.5971780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5971861Z return func(*args, **kwargs) 2025-08-14T21:48:27.5972144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5972249Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5972506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5972579Z return func(*args, **kwargs) 2025-08-14T21:48:27.5972833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5972904Z return func(*args, **kwargs) 2025-08-14T21:48:27.5973148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5973228Z return func(*args, **kwargs) 2025-08-14T21:48:27.5973504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5973579Z self_outputs = self.self( 2025-08-14T21:48:27.5973834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5973906Z return func(*args, **kwargs) 2025-08-14T21:48:27.5974159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5974233Z return func(*args, **kwargs) 2025-08-14T21:48:27.5974477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5974554Z return func(*args, **kwargs) 2025-08-14T21:48:27.5974837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.5974997Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5975008Z 2025-08-14T21:48:27.5975124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5975337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5975438Z return mod(**inputs) 2025-08-14T21:48:27.5975672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5975755Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5976056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5976129Z outputs = self.layoutlm( 2025-08-14T21:48:27.5976384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5976455Z return func(*args, **kwargs) 2025-08-14T21:48:27.5976703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5976781Z return func(*args, **kwargs) 2025-08-14T21:48:27.5977010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5977089Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5977397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5977493Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5977752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5977824Z return func(*args, **kwargs) 2025-08-14T21:48:27.5978075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5978154Z return func(*args, **kwargs) 2025-08-14T21:48:27.5978408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5978477Z return func(*args, **kwargs) 2025-08-14T21:48:27.5978583Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5978813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5978900Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5979187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5979267Z layer_outputs = layer_module( 2025-08-14T21:48:27.5979513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5979600Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5979854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5979936Z return func(*args, **kwargs) 2025-08-14T21:48:27.5980192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5980275Z return func(*args, **kwargs) 2025-08-14T21:48:27.5980536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5980610Z return func(*args, **kwargs) 2025-08-14T21:48:27.5980898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5980986Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5981245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5981316Z return func(*args, **kwargs) 2025-08-14T21:48:27.5981564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5981642Z return func(*args, **kwargs) 2025-08-14T21:48:27.5981890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5981980Z return func(*args, **kwargs) 2025-08-14T21:48:27.5982279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5982353Z self_outputs = self.self( 2025-08-14T21:48:27.5982614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5982686Z return func(*args, **kwargs) 2025-08-14T21:48:27.5982938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5983016Z return func(*args, **kwargs) 2025-08-14T21:48:27.5983269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5983340Z return func(*args, **kwargs) 2025-08-14T21:48:27.5983635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.5983812Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5983835Z 2025-08-14T21:48:27.5983952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5984158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5984227Z return mod(**inputs) 2025-08-14T21:48:27.5984461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5984541Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5984830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5984903Z outputs = self.layoutlm( 2025-08-14T21:48:27.5985166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5985248Z return func(*args, **kwargs) 2025-08-14T21:48:27.5985505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5985577Z return func(*args, **kwargs) 2025-08-14T21:48:27.5985809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5985888Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5986176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5986253Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5986502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5986581Z return func(*args, **kwargs) 2025-08-14T21:48:27.5986832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5986904Z return func(*args, **kwargs) 2025-08-14T21:48:27.5987165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5987236Z return func(*args, **kwargs) 2025-08-14T21:48:27.5987323Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5987553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5987631Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5987916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5987992Z layer_outputs = layer_module( 2025-08-14T21:48:27.5988224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5988336Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5988589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5988669Z return func(*args, **kwargs) 2025-08-14T21:48:27.5988917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5988989Z return func(*args, **kwargs) 2025-08-14T21:48:27.5989245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5989317Z return func(*args, **kwargs) 2025-08-14T21:48:27.5989602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5989689Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5989938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5990019Z return func(*args, **kwargs) 2025-08-14T21:48:27.5990299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5990373Z return func(*args, **kwargs) 2025-08-14T21:48:27.5990630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5990699Z return func(*args, **kwargs) 2025-08-14T21:48:27.5990985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.5991058Z self_outputs = self.self( 2025-08-14T21:48:27.5991313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5991392Z return func(*args, **kwargs) 2025-08-14T21:48:27.5991665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5991740Z return func(*args, **kwargs) 2025-08-14T21:48:27.5992020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5992092Z return func(*args, **kwargs) 2025-08-14T21:48:27.5992389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.5992543Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.5992546Z 2025-08-14T21:48:27.5992632Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5992724Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.5992833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.5993052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.5993133Z return mod(**inputs) 2025-08-14T21:48:27.5993373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5993465Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5993764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.5993838Z outputs = self.layoutlm( 2025-08-14T21:48:27.5994106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5994180Z return func(*args, **kwargs) 2025-08-14T21:48:27.5994455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5994527Z return func(*args, **kwargs) 2025-08-14T21:48:27.5994767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5994876Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5995184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.5995264Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.5995542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5995688Z return func(*args, **kwargs) 2025-08-14T21:48:27.5995977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5996051Z return func(*args, **kwargs) 2025-08-14T21:48:27.5996317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5996399Z return func(*args, **kwargs) 2025-08-14T21:48:27.5996486Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.5996733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.5996880Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.5997174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.5997258Z layer_outputs = layer_module( 2025-08-14T21:48:27.5997487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.5997571Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.5997832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5997903Z return func(*args, **kwargs) 2025-08-14T21:48:27.5998177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5998259Z return func(*args, **kwargs) 2025-08-14T21:48:27.5998518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5998597Z return func(*args, **kwargs) 2025-08-14T21:48:27.5998895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.5998983Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.5999238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5999311Z return func(*args, **kwargs) 2025-08-14T21:48:27.5999574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5999644Z return func(*args, **kwargs) 2025-08-14T21:48:27.5999900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.5999988Z return func(*args, **kwargs) 2025-08-14T21:48:27.6000257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.6000385Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.6000657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.6000740Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6000744Z 2025-08-14T21:48:27.6000854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6001052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6001118Z return mod(**inputs) 2025-08-14T21:48:27.6001340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6001435Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6001703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6001779Z outputs = self.layoutlm( 2025-08-14T21:48:27.6002016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6002090Z return func(*args, **kwargs) 2025-08-14T21:48:27.6002326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6002393Z return func(*args, **kwargs) 2025-08-14T21:48:27.6002613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6002687Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6002955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6003030Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6003303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6003380Z return func(*args, **kwargs) 2025-08-14T21:48:27.6003616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6003685Z return func(*args, **kwargs) 2025-08-14T21:48:27.6003945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6004015Z return func(*args, **kwargs) 2025-08-14T21:48:27.6004102Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6004327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6004422Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6004710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6004788Z layer_outputs = layer_module( 2025-08-14T21:48:27.6005022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6005116Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6005374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6005455Z return func(*args, **kwargs) 2025-08-14T21:48:27.6005712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6005785Z return func(*args, **kwargs) 2025-08-14T21:48:27.6006048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6006122Z return func(*args, **kwargs) 2025-08-14T21:48:27.6006412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6006522Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6006780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6006866Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6007158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.6007278Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.6007546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.6007631Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6007662Z 2025-08-14T21:48:27.6007773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6007972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6008038Z return mod(**inputs) 2025-08-14T21:48:27.6008264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6008343Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6008620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6008836Z outputs = self.layoutlm( 2025-08-14T21:48:27.6009099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6009181Z return func(*args, **kwargs) 2025-08-14T21:48:27.6009441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6009563Z return func(*args, **kwargs) 2025-08-14T21:48:27.6009826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6009907Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6010203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6010277Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6010514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6010590Z return func(*args, **kwargs) 2025-08-14T21:48:27.6010825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6010913Z return func(*args, **kwargs) 2025-08-14T21:48:27.6011168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6011238Z return func(*args, **kwargs) 2025-08-14T21:48:27.6011324Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6011544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6011618Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6011893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6011964Z layer_outputs = layer_module( 2025-08-14T21:48:27.6012183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6012270Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6012511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6012587Z return func(*args, **kwargs) 2025-08-14T21:48:27.6012830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6012898Z return func(*args, **kwargs) 2025-08-14T21:48:27.6013145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6013213Z return func(*args, **kwargs) 2025-08-14T21:48:27.6013483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6013574Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6013832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6013917Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6014218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.6014368Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.6014669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.6014787Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.6015013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.6015088Z return self.act(input) 2025-08-14T21:48:27.6015092Z 2025-08-14T21:48:27.6015204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6015429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6015500Z return mod(**inputs) 2025-08-14T21:48:27.6015744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6015836Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6016176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6016261Z outputs = self.layoutlm( 2025-08-14T21:48:27.6016526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6016600Z return func(*args, **kwargs) 2025-08-14T21:48:27.6016869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6016939Z return func(*args, **kwargs) 2025-08-14T21:48:27.6017166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6017269Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6017545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6017626Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6017854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6017922Z return func(*args, **kwargs) 2025-08-14T21:48:27.6018181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6018253Z return func(*args, **kwargs) 2025-08-14T21:48:27.6018572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6018640Z return func(*args, **kwargs) 2025-08-14T21:48:27.6018717Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6018941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6019019Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6019316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6019402Z layer_outputs = layer_module( 2025-08-14T21:48:27.6019650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6019746Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6020017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6020090Z return func(*args, **kwargs) 2025-08-14T21:48:27.6020358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6020430Z return func(*args, **kwargs) 2025-08-14T21:48:27.6020697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6020802Z return func(*args, **kwargs) 2025-08-14T21:48:27.6021155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6021255Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6021535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6021620Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6021951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.6022097Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.6022406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.6022497Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6022522Z 2025-08-14T21:48:27.6022654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6022877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6022949Z return mod(**inputs) 2025-08-14T21:48:27.6023180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6023267Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6023564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6023648Z outputs = self.layoutlm( 2025-08-14T21:48:27.6023912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6024004Z return func(*args, **kwargs) 2025-08-14T21:48:27.6024276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6024352Z return func(*args, **kwargs) 2025-08-14T21:48:27.6024584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6024672Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6024978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6025065Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6025323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6025395Z return func(*args, **kwargs) 2025-08-14T21:48:27.6025662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6025736Z return func(*args, **kwargs) 2025-08-14T21:48:27.6025999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6026073Z return func(*args, **kwargs) 2025-08-14T21:48:27.6026157Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6026396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6026475Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6026764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6026848Z layer_outputs = layer_module( 2025-08-14T21:48:27.6027085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6027181Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6027440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6027533Z return func(*args, **kwargs) 2025-08-14T21:48:27.6027798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6027871Z return func(*args, **kwargs) 2025-08-14T21:48:27.6028131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6028210Z return func(*args, **kwargs) 2025-08-14T21:48:27.6028509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6028604Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6028851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6028924Z return func(*args, **kwargs) 2025-08-14T21:48:27.6029177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6029284Z return func(*args, **kwargs) 2025-08-14T21:48:27.6029533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6029612Z return func(*args, **kwargs) 2025-08-14T21:48:27.6029909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.6029991Z self_outputs = self.self( 2025-08-14T21:48:27.6030235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6030305Z return func(*args, **kwargs) 2025-08-14T21:48:27.6030585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6030659Z return func(*args, **kwargs) 2025-08-14T21:48:27.6030914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6030987Z return func(*args, **kwargs) 2025-08-14T21:48:27.6031264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.6031425Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.6031429Z 2025-08-14T21:48:27.6031539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6031750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6031826Z return mod(**inputs) 2025-08-14T21:48:27.6032050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6032136Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6032415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6032492Z outputs = self.layoutlm( 2025-08-14T21:48:27.6032750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6032823Z return func(*args, **kwargs) 2025-08-14T21:48:27.6033072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6033152Z return func(*args, **kwargs) 2025-08-14T21:48:27.6033380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6033465Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6033765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6033862Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6034123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6034194Z return func(*args, **kwargs) 2025-08-14T21:48:27.6034452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6034524Z return func(*args, **kwargs) 2025-08-14T21:48:27.6034777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6034858Z return func(*args, **kwargs) 2025-08-14T21:48:27.6034941Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6035172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6035261Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6035549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6035715Z layer_outputs = layer_module( 2025-08-14T21:48:27.6035983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6036072Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6036336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6036410Z return func(*args, **kwargs) 2025-08-14T21:48:27.6036665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6036748Z return func(*args, **kwargs) 2025-08-14T21:48:27.6037029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6037110Z return func(*args, **kwargs) 2025-08-14T21:48:27.6037394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6037487Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6037744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6037818Z return func(*args, **kwargs) 2025-08-14T21:48:27.6038070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6038151Z return func(*args, **kwargs) 2025-08-14T21:48:27.6038400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6038482Z return func(*args, **kwargs) 2025-08-14T21:48:27.6038765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.6038843Z self_outputs = self.self( 2025-08-14T21:48:27.6039103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6039171Z return func(*args, **kwargs) 2025-08-14T21:48:27.6039405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6039480Z return func(*args, **kwargs) 2025-08-14T21:48:27.6039714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6039790Z return func(*args, **kwargs) 2025-08-14T21:48:27.6040055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.6040194Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.6040216Z 2025-08-14T21:48:27.6040330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6040528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6040603Z return mod(**inputs) 2025-08-14T21:48:27.6040814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6040887Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6041156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6041226Z outputs = self.layoutlm( 2025-08-14T21:48:27.6041461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6041535Z return func(*args, **kwargs) 2025-08-14T21:48:27.6041766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6041844Z return func(*args, **kwargs) 2025-08-14T21:48:27.6042103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6042183Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6042466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6042544Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6042802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6042874Z return func(*args, **kwargs) 2025-08-14T21:48:27.6043122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6043201Z return func(*args, **kwargs) 2025-08-14T21:48:27.6043467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6043542Z return func(*args, **kwargs) 2025-08-14T21:48:27.6043633Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6043856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6043939Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6044217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6044291Z layer_outputs = layer_module( 2025-08-14T21:48:27.6044524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6044606Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6044853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6044933Z return func(*args, **kwargs) 2025-08-14T21:48:27.6045180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6045260Z return func(*args, **kwargs) 2025-08-14T21:48:27.6045507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6045578Z return func(*args, **kwargs) 2025-08-14T21:48:27.6045862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6045949Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6046196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6046275Z return func(*args, **kwargs) 2025-08-14T21:48:27.6046523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6046631Z return func(*args, **kwargs) 2025-08-14T21:48:27.6046873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6046940Z return func(*args, **kwargs) 2025-08-14T21:48:27.6047216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.6047287Z self_outputs = self.self( 2025-08-14T21:48:27.6047527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6047602Z return func(*args, **kwargs) 2025-08-14T21:48:27.6047841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6047917Z return func(*args, **kwargs) 2025-08-14T21:48:27.6048156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6048244Z return func(*args, **kwargs) 2025-08-14T21:48:27.6048530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.6048676Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.6048680Z 2025-08-14T21:48:27.6048769Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.6048848Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.6048952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6049159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6049225Z return mod(**inputs) 2025-08-14T21:48:27.6049458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6049542Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6049804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6049881Z outputs = self.layoutlm( 2025-08-14T21:48:27.6050116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6050184Z return func(*args, **kwargs) 2025-08-14T21:48:27.6050425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6050491Z return func(*args, **kwargs) 2025-08-14T21:48:27.6050702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6050785Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6051048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6051130Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6051367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6051435Z return func(*args, **kwargs) 2025-08-14T21:48:27.6051674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6051742Z return func(*args, **kwargs) 2025-08-14T21:48:27.6051976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6052053Z return func(*args, **kwargs) 2025-08-14T21:48:27.6052129Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6052349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6052424Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6052703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6052788Z layer_outputs = layer_module( 2025-08-14T21:48:27.6053003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6053092Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6053324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6053392Z return func(*args, **kwargs) 2025-08-14T21:48:27.6053631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6053698Z return func(*args, **kwargs) 2025-08-14T21:48:27.6053930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6054006Z return func(*args, **kwargs) 2025-08-14T21:48:27.6054285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6054398Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6054635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6054702Z return func(*args, **kwargs) 2025-08-14T21:48:27.6054942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6055009Z return func(*args, **kwargs) 2025-08-14T21:48:27.6055250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6055328Z return func(*args, **kwargs) 2025-08-14T21:48:27.6055628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.6055783Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.6056051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.6056135Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6056139Z 2025-08-14T21:48:27.6056250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6056458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6056536Z return mod(**inputs) 2025-08-14T21:48:27.6056764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6056842Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6057129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6057206Z outputs = self.layoutlm( 2025-08-14T21:48:27.6057457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6057540Z return func(*args, **kwargs) 2025-08-14T21:48:27.6057789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6057869Z return func(*args, **kwargs) 2025-08-14T21:48:27.6058098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6058175Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6058473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6058551Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6058811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6058923Z return func(*args, **kwargs) 2025-08-14T21:48:27.6059184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6059263Z return func(*args, **kwargs) 2025-08-14T21:48:27.6059515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6059584Z return func(*args, **kwargs) 2025-08-14T21:48:27.6059672Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6059899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6059975Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6060267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6060342Z layer_outputs = layer_module( 2025-08-14T21:48:27.6060583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6060700Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6060949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6061027Z return func(*args, **kwargs) 2025-08-14T21:48:27.6061273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6061350Z return func(*args, **kwargs) 2025-08-14T21:48:27.6061595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6061666Z return func(*args, **kwargs) 2025-08-14T21:48:27.6061996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6062090Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6062365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6062453Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6062762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.6062898Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.6063179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.6063266Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6063270Z 2025-08-14T21:48:27.6063387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6063597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6063675Z return mod(**inputs) 2025-08-14T21:48:27.6063904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6063982Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6064268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6064342Z outputs = self.layoutlm( 2025-08-14T21:48:27.6064594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6064676Z return func(*args, **kwargs) 2025-08-14T21:48:27.6064923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6065002Z return func(*args, **kwargs) 2025-08-14T21:48:27.6065228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6065326Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6065624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6065703Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6065962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6066046Z return func(*args, **kwargs) 2025-08-14T21:48:27.6066298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6066380Z return func(*args, **kwargs) 2025-08-14T21:48:27.6066641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6066716Z return func(*args, **kwargs) 2025-08-14T21:48:27.6066811Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6067044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6067154Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6067442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6067518Z layer_outputs = layer_module( 2025-08-14T21:48:27.6067753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6067836Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6068083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6068161Z return func(*args, **kwargs) 2025-08-14T21:48:27.6068428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6068512Z return func(*args, **kwargs) 2025-08-14T21:48:27.6068767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6068840Z return func(*args, **kwargs) 2025-08-14T21:48:27.6069130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6069220Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6069496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6069586Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6069905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.6070042Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.6070325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.6070447Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.6070681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.6070756Z return self.act(input) 2025-08-14T21:48:27.6070760Z 2025-08-14T21:48:27.6070876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6071088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6071158Z return mod(**inputs) 2025-08-14T21:48:27.6071395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6071473Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6071776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6071880Z outputs = self.layoutlm( 2025-08-14T21:48:27.6072130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6072212Z return func(*args, **kwargs) 2025-08-14T21:48:27.6072459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6072530Z return func(*args, **kwargs) 2025-08-14T21:48:27.6072761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6072841Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6073143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6073232Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6073488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6073588Z return func(*args, **kwargs) 2025-08-14T21:48:27.6073861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6073935Z return func(*args, **kwargs) 2025-08-14T21:48:27.6074198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6074270Z return func(*args, **kwargs) 2025-08-14T21:48:27.6074353Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6074590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6074668Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6074992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6075072Z layer_outputs = layer_module( 2025-08-14T21:48:27.6075313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6075406Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6075735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6075816Z return func(*args, **kwargs) 2025-08-14T21:48:27.6076081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6076156Z return func(*args, **kwargs) 2025-08-14T21:48:27.6076420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6076492Z return func(*args, **kwargs) 2025-08-14T21:48:27.6076780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6076889Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6077145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6077229Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6077522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.6077656Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.6077926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.6078008Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6078012Z 2025-08-14T21:48:27.6078126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6078345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6078414Z return mod(**inputs) 2025-08-14T21:48:27.6078644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6078721Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6078983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6079066Z outputs = self.layoutlm( 2025-08-14T21:48:27.6079307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6079387Z return func(*args, **kwargs) 2025-08-14T21:48:27.6079639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6079714Z return func(*args, **kwargs) 2025-08-14T21:48:27.6079953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6080074Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6080359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6080446Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6080706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6080783Z return func(*args, **kwargs) 2025-08-14T21:48:27.6081020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6081091Z return func(*args, **kwargs) 2025-08-14T21:48:27.6081354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6081428Z return func(*args, **kwargs) 2025-08-14T21:48:27.6081508Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6081748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6081825Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6082114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6082191Z layer_outputs = layer_module( 2025-08-14T21:48:27.6082423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6082513Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6082767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6082838Z return func(*args, **kwargs) 2025-08-14T21:48:27.6083098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6083173Z return func(*args, **kwargs) 2025-08-14T21:48:27.6083432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6083504Z return func(*args, **kwargs) 2025-08-14T21:48:27.6083782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6083880Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6084134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6084213Z return func(*args, **kwargs) 2025-08-14T21:48:27.6084461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6084533Z return func(*args, **kwargs) 2025-08-14T21:48:27.6084804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6084879Z return func(*args, **kwargs) 2025-08-14T21:48:27.6085160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.6085243Z self_outputs = self.self( 2025-08-14T21:48:27.6085490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6085569Z return func(*args, **kwargs) 2025-08-14T21:48:27.6085817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6085888Z return func(*args, **kwargs) 2025-08-14T21:48:27.6086143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6086216Z return func(*args, **kwargs) 2025-08-14T21:48:27.6086509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.6086686Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.6086690Z 2025-08-14T21:48:27.6086799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6087017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6087088Z return mod(**inputs) 2025-08-14T21:48:27.6087316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6087403Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6087698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6087780Z outputs = self.layoutlm( 2025-08-14T21:48:27.6088030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6088104Z return func(*args, **kwargs) 2025-08-14T21:48:27.6088359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6088431Z return func(*args, **kwargs) 2025-08-14T21:48:27.6088653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6088738Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6089038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6089122Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6089370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6089444Z return func(*args, **kwargs) 2025-08-14T21:48:27.6089695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6089768Z return func(*args, **kwargs) 2025-08-14T21:48:27.6090015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6090095Z return func(*args, **kwargs) 2025-08-14T21:48:27.6090175Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6090403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6090481Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6090779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6090861Z layer_outputs = layer_module( 2025-08-14T21:48:27.6091111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6091196Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6091455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6091525Z return func(*args, **kwargs) 2025-08-14T21:48:27.6091777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6091849Z return func(*args, **kwargs) 2025-08-14T21:48:27.6092097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6092176Z return func(*args, **kwargs) 2025-08-14T21:48:27.6092477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6092570Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6092820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6092925Z return func(*args, **kwargs) 2025-08-14T21:48:27.6093186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6093257Z return func(*args, **kwargs) 2025-08-14T21:48:27.6093506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6093583Z return func(*args, **kwargs) 2025-08-14T21:48:27.6093882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.6093964Z self_outputs = self.self( 2025-08-14T21:48:27.6094232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6094306Z return func(*args, **kwargs) 2025-08-14T21:48:27.6094562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6094633Z return func(*args, **kwargs) 2025-08-14T21:48:27.6094879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6094959Z return func(*args, **kwargs) 2025-08-14T21:48:27.6095234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.6095386Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.6095390Z 2025-08-14T21:48:27.6095501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6095712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6095792Z return mod(**inputs) 2025-08-14T21:48:27.6096018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6096109Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6096388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6096462Z outputs = self.layoutlm( 2025-08-14T21:48:27.6096718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6096790Z return func(*args, **kwargs) 2025-08-14T21:48:27.6097036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6097115Z return func(*args, **kwargs) 2025-08-14T21:48:27.6097340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6097448Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6097740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6097817Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6098078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6098149Z return func(*args, **kwargs) 2025-08-14T21:48:27.6098402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6098479Z return func(*args, **kwargs) 2025-08-14T21:48:27.6098733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6098811Z return func(*args, **kwargs) 2025-08-14T21:48:27.6098893Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6099126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6099252Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6099531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6099605Z layer_outputs = layer_module( 2025-08-14T21:48:27.6099842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6099925Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6100182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6100253Z return func(*args, **kwargs) 2025-08-14T21:48:27.6100522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6100608Z return func(*args, **kwargs) 2025-08-14T21:48:27.6100858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6100932Z return func(*args, **kwargs) 2025-08-14T21:48:27.6101219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6101307Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6101562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6101635Z return func(*args, **kwargs) 2025-08-14T21:48:27.6101884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6101962Z return func(*args, **kwargs) 2025-08-14T21:48:27.6102214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6102295Z return func(*args, **kwargs) 2025-08-14T21:48:27.6102576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.6102653Z self_outputs = self.self( 2025-08-14T21:48:27.6102909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6102980Z return func(*args, **kwargs) 2025-08-14T21:48:27.6103231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6103311Z return func(*args, **kwargs) 2025-08-14T21:48:27.6103559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6103638Z return func(*args, **kwargs) 2025-08-14T21:48:27.6103919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.6104106Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.6104111Z 2025-08-14T21:48:27.6104206Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.6104289Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.6104399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6104614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6104684Z return mod(**inputs) 2025-08-14T21:48:27.6104916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6104994Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6105274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6105357Z outputs = self.layoutlm( 2025-08-14T21:48:27.6105604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6105718Z return func(*args, **kwargs) 2025-08-14T21:48:27.6105970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6106042Z return func(*args, **kwargs) 2025-08-14T21:48:27.6106274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6106352Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6106631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6106716Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6106984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6107067Z return func(*args, **kwargs) 2025-08-14T21:48:27.6107323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6107397Z return func(*args, **kwargs) 2025-08-14T21:48:27.6107653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6107723Z return func(*args, **kwargs) 2025-08-14T21:48:27.6107802Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6108034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6108111Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6108400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6108476Z layer_outputs = layer_module( 2025-08-14T21:48:27.6108846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6108952Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6109209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6109282Z return func(*args, **kwargs) 2025-08-14T21:48:27.6109540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6109615Z return func(*args, **kwargs) 2025-08-14T21:48:27.6109873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6109944Z return func(*args, **kwargs) 2025-08-14T21:48:27.6110226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6110367Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6110622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6110698Z return func(*args, **kwargs) 2025-08-14T21:48:27.6110961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6111035Z return func(*args, **kwargs) 2025-08-14T21:48:27.6111295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6111369Z return func(*args, **kwargs) 2025-08-14T21:48:27.6111654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.6111803Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.6112108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.6112242Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6112246Z 2025-08-14T21:48:27.6112385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6112604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6112685Z return mod(**inputs) 2025-08-14T21:48:27.6112916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6112996Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6113308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6113383Z outputs = self.layoutlm( 2025-08-14T21:48:27.6113675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6113752Z return func(*args, **kwargs) 2025-08-14T21:48:27.6114014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6114096Z return func(*args, **kwargs) 2025-08-14T21:48:27.6114335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6114422Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6114728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6114806Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6115075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6115150Z return func(*args, **kwargs) 2025-08-14T21:48:27.6115414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6115498Z return func(*args, **kwargs) 2025-08-14T21:48:27.6115819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6115906Z return func(*args, **kwargs) 2025-08-14T21:48:27.6115987Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6116217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6116305Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6116611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6116690Z layer_outputs = layer_module( 2025-08-14T21:48:27.6116942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6117028Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6117324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6117402Z return func(*args, **kwargs) 2025-08-14T21:48:27.6117661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6117743Z return func(*args, **kwargs) 2025-08-14T21:48:27.6118001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6118075Z return func(*args, **kwargs) 2025-08-14T21:48:27.6118370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6118465Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6118757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6118844Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6120131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.6120276Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.6120568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.6120667Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6120671Z 2025-08-14T21:48:27.6120784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6121002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6121081Z return mod(**inputs) 2025-08-14T21:48:27.6121334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6121418Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6121715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6121793Z outputs = self.layoutlm( 2025-08-14T21:48:27.6122059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6122134Z return func(*args, **kwargs) 2025-08-14T21:48:27.6122390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6122474Z return func(*args, **kwargs) 2025-08-14T21:48:27.6122705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6122786Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6123079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6123160Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6123427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6123501Z return func(*args, **kwargs) 2025-08-14T21:48:27.6123755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6123835Z return func(*args, **kwargs) 2025-08-14T21:48:27.6124087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6124167Z return func(*args, **kwargs) 2025-08-14T21:48:27.6124249Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6124479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6124567Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6124874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6124955Z layer_outputs = layer_module( 2025-08-14T21:48:27.6125199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6125284Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6125548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6125620Z return func(*args, **kwargs) 2025-08-14T21:48:27.6125873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6125953Z return func(*args, **kwargs) 2025-08-14T21:48:27.6126208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6126282Z return func(*args, **kwargs) 2025-08-14T21:48:27.6126594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6126723Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6127012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6127097Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6127427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.6127562Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.6127844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.6128002Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.6128226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.6128302Z return self.act(input) 2025-08-14T21:48:27.6128306Z 2025-08-14T21:48:27.6128423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6128632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6128701Z return mod(**inputs) 2025-08-14T21:48:27.6128934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6129012Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6129315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6129390Z outputs = self.layoutlm( 2025-08-14T21:48:27.6129640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6129721Z return func(*args, **kwargs) 2025-08-14T21:48:27.6129974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6130046Z return func(*args, **kwargs) 2025-08-14T21:48:27.6130279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6130358Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6130662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6130741Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6130991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6131074Z return func(*args, **kwargs) 2025-08-14T21:48:27.6131344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6131423Z return func(*args, **kwargs) 2025-08-14T21:48:27.6131673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6131743Z return func(*args, **kwargs) 2025-08-14T21:48:27.6131829Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6132057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6132135Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6132424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6132498Z layer_outputs = layer_module( 2025-08-14T21:48:27.6132735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6132820Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6133103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6133184Z return func(*args, **kwargs) 2025-08-14T21:48:27.6133431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6133504Z return func(*args, **kwargs) 2025-08-14T21:48:27.6133759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6133828Z return func(*args, **kwargs) 2025-08-14T21:48:27.6134114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6134208Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6134506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6134601Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6134931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.6135083Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.6135385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.6135474Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6135478Z 2025-08-14T21:48:27.6135594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6135801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6135873Z return mod(**inputs) 2025-08-14T21:48:27.6136115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6136196Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6136490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6136565Z outputs = self.layoutlm( 2025-08-14T21:48:27.6136820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6136901Z return func(*args, **kwargs) 2025-08-14T21:48:27.6137156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6137229Z return func(*args, **kwargs) 2025-08-14T21:48:27.6137470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6137550Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6137855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6137936Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6138185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6138265Z return func(*args, **kwargs) 2025-08-14T21:48:27.6138516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6138593Z return func(*args, **kwargs) 2025-08-14T21:48:27.6138840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6138911Z return func(*args, **kwargs) 2025-08-14T21:48:27.6138997Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6139223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6139303Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6139625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6139702Z layer_outputs = layer_module( 2025-08-14T21:48:27.6139949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6140033Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6140291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6140367Z return func(*args, **kwargs) 2025-08-14T21:48:27.6140631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6140705Z return func(*args, **kwargs) 2025-08-14T21:48:27.6140996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6141088Z return func(*args, **kwargs) 2025-08-14T21:48:27.6141389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6141479Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6141739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6141818Z return func(*args, **kwargs) 2025-08-14T21:48:27.6142086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6142159Z return func(*args, **kwargs) 2025-08-14T21:48:27.6142431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6142505Z return func(*args, **kwargs) 2025-08-14T21:48:27.6142814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.6142896Z self_outputs = self.self( 2025-08-14T21:48:27.6143155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6143235Z return func(*args, **kwargs) 2025-08-14T21:48:27.6143501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6143575Z return func(*args, **kwargs) 2025-08-14T21:48:27.6143850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6143921Z return func(*args, **kwargs) 2025-08-14T21:48:27.6144216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:48:27.6144393Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.6144399Z 2025-08-14T21:48:27.6144514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6144737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6144808Z return mod(**inputs) 2025-08-14T21:48:27.6145050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6145131Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6145429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6145513Z outputs = self.layoutlm( 2025-08-14T21:48:27.6145777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6145851Z return func(*args, **kwargs) 2025-08-14T21:48:27.6146133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6146246Z return func(*args, **kwargs) 2025-08-14T21:48:27.6146491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6146572Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6146871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6146959Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6147217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6147290Z return func(*args, **kwargs) 2025-08-14T21:48:27.6147583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6147660Z return func(*args, **kwargs) 2025-08-14T21:48:27.6147940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6148015Z return func(*args, **kwargs) 2025-08-14T21:48:27.6148099Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6148340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6148420Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6148748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6148826Z layer_outputs = layer_module( 2025-08-14T21:48:27.6149074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6149167Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6149425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6149501Z return func(*args, **kwargs) 2025-08-14T21:48:27.6149766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6149839Z return func(*args, **kwargs) 2025-08-14T21:48:27.6150102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6150175Z return func(*args, **kwargs) 2025-08-14T21:48:27.6150500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6150600Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6150860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6150932Z return func(*args, **kwargs) 2025-08-14T21:48:27.6151213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6151288Z return func(*args, **kwargs) 2025-08-14T21:48:27.6151548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6151621Z return func(*args, **kwargs) 2025-08-14T21:48:27.6151942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.6152027Z self_outputs = self.self( 2025-08-14T21:48:27.6152288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6152360Z return func(*args, **kwargs) 2025-08-14T21:48:27.6152622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6152697Z return func(*args, **kwargs) 2025-08-14T21:48:27.6152981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6153085Z return func(*args, **kwargs) 2025-08-14T21:48:27.6153380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:48:27.6153540Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.6153544Z 2025-08-14T21:48:27.6153656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6153881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6153952Z return mod(**inputs) 2025-08-14T21:48:27.6154209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6154300Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6154588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6154666Z outputs = self.layoutlm( 2025-08-14T21:48:27.6154930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6155002Z return func(*args, **kwargs) 2025-08-14T21:48:27.6155265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6155337Z return func(*args, **kwargs) 2025-08-14T21:48:27.6155568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6155741Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6156075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6156158Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6156427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6156503Z return func(*args, **kwargs) 2025-08-14T21:48:27.6156767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6156842Z return func(*args, **kwargs) 2025-08-14T21:48:27.6157096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6157179Z return func(*args, **kwargs) 2025-08-14T21:48:27.6157263Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6157504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6157587Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6157907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6157996Z layer_outputs = layer_module( 2025-08-14T21:48:27.6158235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6158321Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6158591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6158666Z return func(*args, **kwargs) 2025-08-14T21:48:27.6158930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6159002Z return func(*args, **kwargs) 2025-08-14T21:48:27.6159260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6159342Z return func(*args, **kwargs) 2025-08-14T21:48:27.6159627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6159757Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6160022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6160097Z return func(*args, **kwargs) 2025-08-14T21:48:27.6160356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6160431Z return func(*args, **kwargs) 2025-08-14T21:48:27.6160684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6160769Z return func(*args, **kwargs) 2025-08-14T21:48:27.6161076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:48:27.6161156Z self_outputs = self.self( 2025-08-14T21:48:27.6161424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6161497Z return func(*args, **kwargs) 2025-08-14T21:48:27.6161761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6161835Z return func(*args, **kwargs) 2025-08-14T21:48:27.6162090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6162197Z return func(*args, **kwargs) 2025-08-14T21:48:27.6162483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:48:27.6162652Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:48:27.6162658Z 2025-08-14T21:48:27.6162747Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.6162834Z cudagraph partition due to non gpu ops 2025-08-14T21:48:27.6162957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6163175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6163246Z return mod(**inputs) 2025-08-14T21:48:27.6163487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6163567Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6163861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6163937Z outputs = self.layoutlm( 2025-08-14T21:48:27.6164193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6164275Z return func(*args, **kwargs) 2025-08-14T21:48:27.6164549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6164624Z return func(*args, **kwargs) 2025-08-14T21:48:27.6164861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6164940Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6165233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6165312Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6165564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6165643Z return func(*args, **kwargs) 2025-08-14T21:48:27.6165900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6165974Z return func(*args, **kwargs) 2025-08-14T21:48:27.6166253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6166372Z return func(*args, **kwargs) 2025-08-14T21:48:27.6166460Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6166691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6166769Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6167063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6167141Z layer_outputs = layer_module( 2025-08-14T21:48:27.6167377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6167488Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6167749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6167832Z return func(*args, **kwargs) 2025-08-14T21:48:27.6168089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6168162Z return func(*args, **kwargs) 2025-08-14T21:48:27.6168423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6168495Z return func(*args, **kwargs) 2025-08-14T21:48:27.6168788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:48:27.6168880Z self_attention_outputs = self.attention( 2025-08-14T21:48:27.6169139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6169222Z return func(*args, **kwargs) 2025-08-14T21:48:27.6169487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6169561Z return func(*args, **kwargs) 2025-08-14T21:48:27.6169817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6169888Z return func(*args, **kwargs) 2025-08-14T21:48:27.6170171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:48:27.6170305Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:48:27.6170585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:48:27.6170681Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6170686Z 2025-08-14T21:48:27.6170794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6171032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6171105Z return mod(**inputs) 2025-08-14T21:48:27.6171331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6171417Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6171692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6171764Z outputs = self.layoutlm( 2025-08-14T21:48:27.6172015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6172086Z return func(*args, **kwargs) 2025-08-14T21:48:27.6172341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6172411Z return func(*args, **kwargs) 2025-08-14T21:48:27.6172656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6172758Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6173046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6173126Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6173386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6173459Z return func(*args, **kwargs) 2025-08-14T21:48:27.6173722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6173795Z return func(*args, **kwargs) 2025-08-14T21:48:27.6174072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6174157Z return func(*args, **kwargs) 2025-08-14T21:48:27.6174242Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6174488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6174573Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6174851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6174931Z layer_outputs = layer_module( 2025-08-14T21:48:27.6175161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6175246Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6175503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6175575Z return func(*args, **kwargs) 2025-08-14T21:48:27.6175839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6175913Z return func(*args, **kwargs) 2025-08-14T21:48:27.6176162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6176240Z return func(*args, **kwargs) 2025-08-14T21:48:27.6176521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6176613Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6176894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6176975Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6177296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.6177444Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.6177723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:48:27.6177820Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6177823Z 2025-08-14T21:48:27.6177932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6178146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6178214Z return mod(**inputs) 2025-08-14T21:48:27.6178440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6178525Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6178825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6178900Z outputs = self.layoutlm( 2025-08-14T21:48:27.6179180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6179270Z return func(*args, **kwargs) 2025-08-14T21:48:27.6179526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6179598Z return func(*args, **kwargs) 2025-08-14T21:48:27.6179827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6179910Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6180210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6180286Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6180570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6180645Z return func(*args, **kwargs) 2025-08-14T21:48:27.6180906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6180977Z return func(*args, **kwargs) 2025-08-14T21:48:27.6181225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6181305Z return func(*args, **kwargs) 2025-08-14T21:48:27.6181385Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6181612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6181699Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6181979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6182062Z layer_outputs = layer_module( 2025-08-14T21:48:27.6182293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6182379Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6182637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6182708Z return func(*args, **kwargs) 2025-08-14T21:48:27.6182956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6183035Z return func(*args, **kwargs) 2025-08-14T21:48:27.6183282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6183361Z return func(*args, **kwargs) 2025-08-14T21:48:27.6183645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6183756Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6184047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6184131Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6184463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:48:27.6184591Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:48:27.6184872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:48:27.6184999Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:48:27.6185220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:27.6185295Z return self.act(input) 2025-08-14T21:48:27.6185308Z 2025-08-14T21:48:27.6185418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6185662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6185741Z return mod(**inputs) 2025-08-14T21:48:27.6185968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6186047Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6186335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6186409Z outputs = self.layoutlm( 2025-08-14T21:48:27.6186666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6186738Z return func(*args, **kwargs) 2025-08-14T21:48:27.6187005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6187085Z return func(*args, **kwargs) 2025-08-14T21:48:27.6187315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6187395Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6187678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:48:27.6187755Z encoder_outputs = self.encoder( 2025-08-14T21:48:27.6188005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6188077Z return func(*args, **kwargs) 2025-08-14T21:48:27.6188322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6188398Z return func(*args, **kwargs) 2025-08-14T21:48:27.6188644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6188717Z return func(*args, **kwargs) 2025-08-14T21:48:27.6188805Z [Previous line repeated 1 more time] 2025-08-14T21:48:27.6189030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6189113Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6189389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:48:27.6189464Z layer_outputs = layer_module( 2025-08-14T21:48:27.6189703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:27.6189786Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:27.6190036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6190144Z return func(*args, **kwargs) 2025-08-14T21:48:27.6190408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6190489Z return func(*args, **kwargs) 2025-08-14T21:48:27.6190749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6190822Z return func(*args, **kwargs) 2025-08-14T21:48:27.6191122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:48:27.6191214Z layer_output = apply_chunking_to_forward( 2025-08-14T21:48:27.6191504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:48:27.6191587Z return forward_fn(*input_tensors) 2025-08-14T21:48:27.6191918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:48:27.6192107Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:48:27.6192396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:48:27.6192487Z hidden_states = self.dense(hidden_states) 2025-08-14T21:48:27.6192500Z 2025-08-14T21:48:27.6192615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6192829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6192908Z return mod(**inputs) 2025-08-14T21:48:27.6193141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6193221Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6193531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6193613Z outputs = self.layoutlm( 2025-08-14T21:48:27.6193884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6193960Z return func(*args, **kwargs) 2025-08-14T21:48:27.6194220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6194301Z return func(*args, **kwargs) 2025-08-14T21:48:27.6194529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6194610Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6194908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 654, in forward 2025-08-14T21:48:27.6195012Z pooled_output = self.pooler(sequence_output) 2025-08-14T21:48:27.6195308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 430, in forward 2025-08-14T21:48:27.6195415Z pooled_output = self.dense(first_token_tensor) 2025-08-14T21:48:27.6195419Z 2025-08-14T21:48:27.6195533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6195834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6195911Z return mod(**inputs) 2025-08-14T21:48:27.6196146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6196238Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6196544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:48:27.6196631Z outputs = self.layoutlm( 2025-08-14T21:48:27.6196890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6196988Z return func(*args, **kwargs) 2025-08-14T21:48:27.6197255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:48:27.6197330Z return func(*args, **kwargs) 2025-08-14T21:48:27.6197569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6197650Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6197954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 654, in forward 2025-08-14T21:48:27.6198061Z pooled_output = self.pooler(sequence_output) 2025-08-14T21:48:27.6198363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 431, in forward 2025-08-14T21:48:27.6198469Z pooled_output = self.activation(pooled_output) 2025-08-14T21:48:27.6198475Z 2025-08-14T21:48:27.6198597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6198849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6198929Z return mod(**inputs) 2025-08-14T21:48:27.6199161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6199240Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6199549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 891, in forward 2025-08-14T21:48:27.6199640Z logits = self.classifier(pooled_output) 2025-08-14T21:48:27.6199644Z 2025-08-14T21:48:27.6199762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6199989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6200063Z return mod(**inputs) 2025-08-14T21:48:27.6200304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6200387Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6200673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-08-14T21:48:27.6200830Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:48:27.6200834Z 2025-08-14T21:48:27.6200945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6201167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6201236Z return mod(**inputs) 2025-08-14T21:48:27.6201473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6201563Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6201856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-08-14T21:48:27.6201998Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:48:27.6202009Z 2025-08-14T21:48:27.6202120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:27.6202334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:27.6202412Z return mod(**inputs) 2025-08-14T21:48:27.6202648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:48:27.6202725Z output = func(self, *args, **kwargs) 2025-08-14T21:48:27.6203022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-08-14T21:48:27.6203156Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:48:27.6203186Z 2025-08-14T21:48:39.5571651Z Compilation time (from dynamo_timed): 19.241572571 2025-08-14T21:48:39.5571988Z pass 2025-08-14T21:48:39.5572449Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:39.5574002Z TIMING: _recursive_pre_grad_passes:0.01285 _recursive_joint_graph_passes:0.45633 _recursive_post_grad_passes:0.08442 async_compile.wait:0.66393 code_gen:7.51665 inductor_compile:8.71415 backend_compile:13.13049 gc:0.00193 entire_frame_compile:19.24157 total_wall_time:19.24157 2025-08-14T21:48:39.5575163Z STATS: call_* op count: 860 | FakeTensorMode.__torch_dispatch__:16781 | FakeTensor.__torch_dispatch__:4682 | ProxyTorchDispatchMode.__torch_dispatch__:5774 2025-08-14T21:48:39.5575790Z Dynamo produced 2 graphs covering 860 ops with 0 graph breaks (0 unique) 2025-08-14T21:48:44.9907417Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:48:44.9909545Z from pkg_resources import resource_filename 2025-08-14T21:48:45.5653344Z 2025-08-14T21:48:52.4958447Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:48:52.4958741Z loading model: 0it [00:06, ?it/s] 2025-08-14T21:48:52.4989879Z cpu eval M2M100ForConditionalGeneration 2025-08-14T21:48:53.4050622Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:53.7803991Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:54.1541920Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:11.5680946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5681777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5682160Z return mod(**inputs) 2025-08-14T21:49:11.5682628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5683068Z outputs = self.model( 2025-08-14T21:49:11.5683472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5683897Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5684324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-08-14T21:49:11.5684792Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-08-14T21:49:11.5685246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:49:11.5685631Z return func(*args, **kwargs) 2025-08-14T21:49:11.5686040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:49:11.5686609Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:49:11.5687229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 80, in create_position_ids_from_input_ids 2025-08-14T21:49:11.5687708Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:49:11.5687851Z 2025-08-14T21:49:11.5687967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5688339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5688724Z return mod(**inputs) 2025-08-14T21:49:11.5689124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5689612Z outputs = self.model( 2025-08-14T21:49:11.5690001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.5690430Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.5690837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-08-14T21:49:11.5691314Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-08-14T21:49:11.5691761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:49:11.5692128Z return func(*args, **kwargs) 2025-08-14T21:49:11.5692538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:49:11.5693097Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:49:11.5693763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 80, in create_position_ids_from_input_ids 2025-08-14T21:49:11.5694289Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:49:11.5694438Z 2025-08-14T21:49:11.5694533Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5694753Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5694972Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5695190Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5695407Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5695620Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5695845Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5696066Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5696278Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5696516Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5696734Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5696944Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5697195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5697583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5697939Z return mod(**inputs) 2025-08-14T21:49:11.5698339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5698752Z outputs = self.model( 2025-08-14T21:49:11.5699144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5699556Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5699971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-08-14T21:49:11.5700435Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-08-14T21:49:11.5700862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:49:11.5701239Z return func(*args, **kwargs) 2025-08-14T21:49:11.5701635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:49:11.5702195Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:49:11.5702817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-14T21:49:11.5703434Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:49:11.5703704Z 2025-08-14T21:49:11.5703828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5704214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5704586Z return mod(**inputs) 2025-08-14T21:49:11.5704990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5705401Z outputs = self.model( 2025-08-14T21:49:11.5705797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5706211Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5706625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-08-14T21:49:11.5707073Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-08-14T21:49:11.5707497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:49:11.5707887Z return func(*args, **kwargs) 2025-08-14T21:49:11.5708318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:49:11.5709123Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:49:11.5709744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-14T21:49:11.5710352Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:49:11.5710659Z 2025-08-14T21:49:11.5710945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5711343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5711690Z return mod(**inputs) 2025-08-14T21:49:11.5712165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5712583Z outputs = self.model( 2025-08-14T21:49:11.5712991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5713414Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5713831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5714249Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5714643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5715043Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5715478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5716176Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5716614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.5717121Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.5717352Z 2025-08-14T21:49:11.5717467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5717860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5718207Z return mod(**inputs) 2025-08-14T21:49:11.5718606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5719018Z outputs = self.model( 2025-08-14T21:49:11.5719418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5719831Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5720249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5720717Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5721096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5721497Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5721922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5722361Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5722790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.5723219Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.5723369Z 2025-08-14T21:49:11.5723494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5723885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5724233Z return mod(**inputs) 2025-08-14T21:49:11.5724709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5725124Z outputs = self.model( 2025-08-14T21:49:11.5725509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5725927Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5726341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5726757Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5727131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5727551Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5727980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5728410Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5728844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.5729282Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.5729438Z 2025-08-14T21:49:11.5729533Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5729779Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5730016Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5730246Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5730494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5730887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5731241Z return mod(**inputs) 2025-08-14T21:49:11.5731637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5732051Z outputs = self.model( 2025-08-14T21:49:11.5732446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5732878Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5733283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5733738Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5734154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5734606Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5735025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5735484Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5735922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5736369Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5736854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.5737381Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.5737582Z 2025-08-14T21:49:11.5737704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5738093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5738451Z return mod(**inputs) 2025-08-14T21:49:11.5738874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5739294Z outputs = self.model( 2025-08-14T21:49:11.5739749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5740177Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5740593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5741011Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5741385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5741773Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5742190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5742659Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5743091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5743538Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5744016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.5744515Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.5744698Z 2025-08-14T21:49:11.5744813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5745209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5745560Z return mod(**inputs) 2025-08-14T21:49:11.5745968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5746385Z outputs = self.model( 2025-08-14T21:49:11.5746851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5747265Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5747668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5748071Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5748436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5748807Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5749209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5749642Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5750059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.5750503Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.5750654Z 2025-08-14T21:49:11.5750768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5751155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5751488Z return mod(**inputs) 2025-08-14T21:49:11.5751873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5752304Z outputs = self.model( 2025-08-14T21:49:11.5752695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5753116Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5753529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5753955Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5754329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5754762Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5755185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5755894Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5756088Z 2025-08-14T21:49:11.5756202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5756599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5756969Z return mod(**inputs) 2025-08-14T21:49:11.5757368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5757789Z outputs = self.model( 2025-08-14T21:49:11.5758212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5758646Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5759057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5759477Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5759861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5760253Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5760671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5761139Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5761328Z 2025-08-14T21:49:11.5761449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5761842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5762190Z return mod(**inputs) 2025-08-14T21:49:11.5762591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5763005Z outputs = self.model( 2025-08-14T21:49:11.5763397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5763820Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5764232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5764650Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5765022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5765415Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5765859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.5766283Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.5766442Z 2025-08-14T21:49:11.5766563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5766957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5767308Z return mod(**inputs) 2025-08-14T21:49:11.5767700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5768115Z outputs = self.model( 2025-08-14T21:49:11.5768509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5768920Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5769334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5769767Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5770156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5770535Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5770944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5771383Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5771818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.5772316Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.5772545Z 2025-08-14T21:49:11.5772674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5773071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5773410Z return mod(**inputs) 2025-08-14T21:49:11.5773805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5774227Z outputs = self.model( 2025-08-14T21:49:11.5774609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5775035Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5775474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5775886Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5776257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5776690Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5777098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5777529Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5777942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.5778368Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.5778519Z 2025-08-14T21:49:11.5778628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5779007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5779344Z return mod(**inputs) 2025-08-14T21:49:11.5779727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5780150Z outputs = self.model( 2025-08-14T21:49:11.5780556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5780975Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5781373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5781777Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5782133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5782514Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5782922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5783341Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5783752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.5784166Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.5784311Z 2025-08-14T21:49:11.5784438Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5784677Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5784891Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5785106Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5785360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5785731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5786072Z return mod(**inputs) 2025-08-14T21:49:11.5786455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5786849Z outputs = self.model( 2025-08-14T21:49:11.5787252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5787663Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5788069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5788470Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5788839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5789221Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5789622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5790046Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5790477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5790918Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5791391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.5791915Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.5792120Z 2025-08-14T21:49:11.5792232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5792632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5792977Z return mod(**inputs) 2025-08-14T21:49:11.5793375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5793805Z outputs = self.model( 2025-08-14T21:49:11.5794201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5794624Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5795040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5795488Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5795945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5796341Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5796808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5797259Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5797683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5798128Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5798615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.5799109Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.5799287Z 2025-08-14T21:49:11.5799425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5799832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5800186Z return mod(**inputs) 2025-08-14T21:49:11.5800580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5801005Z outputs = self.model( 2025-08-14T21:49:11.5801410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5801835Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5802235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5802670Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5803054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5803453Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5803885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5804325Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5804763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.5805191Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.5805349Z 2025-08-14T21:49:11.5805462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5805856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5806226Z return mod(**inputs) 2025-08-14T21:49:11.5806643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5807051Z outputs = self.model( 2025-08-14T21:49:11.5807437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5807846Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5808255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5808785Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5809177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5809559Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5809987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5810509Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5810691Z 2025-08-14T21:49:11.5810812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5811182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5811521Z return mod(**inputs) 2025-08-14T21:49:11.5811900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5812305Z outputs = self.model( 2025-08-14T21:49:11.5812688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5813105Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5813506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5813914Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5814281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5814735Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5815150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5815612Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5815795Z 2025-08-14T21:49:11.5815904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5816277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5816611Z return mod(**inputs) 2025-08-14T21:49:11.5816991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5817403Z outputs = self.model( 2025-08-14T21:49:11.5817814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5818223Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5818630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5819036Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5819397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5819792Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5820207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.5820625Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.5820775Z 2025-08-14T21:49:11.5820888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5821284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5821645Z return mod(**inputs) 2025-08-14T21:49:11.5822041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5822455Z outputs = self.model( 2025-08-14T21:49:11.5822854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5823268Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5823678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5824103Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5824473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5824859Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5825291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:49:11.5825723Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.5825871Z 2025-08-14T21:49:11.5825991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5826373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5826730Z return mod(**inputs) 2025-08-14T21:49:11.5827131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5827543Z outputs = self.model( 2025-08-14T21:49:11.5827928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5828349Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5828762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5829172Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5829653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5830048Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5830471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5830897Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5831329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.5831840Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.5832059Z 2025-08-14T21:49:11.5832180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5832577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5832931Z return mod(**inputs) 2025-08-14T21:49:11.5833325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5833729Z outputs = self.model( 2025-08-14T21:49:11.5834122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5834548Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5834956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5835373Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5835825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5836233Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5836656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5837090Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5837525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.5837964Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.5838111Z 2025-08-14T21:49:11.5838223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5838613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5838983Z return mod(**inputs) 2025-08-14T21:49:11.5839370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5839736Z outputs = self.model( 2025-08-14T21:49:11.5840093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5840495Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5840855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5841241Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5841586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5841945Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5842331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5842754Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5843174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.5843595Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.5843736Z 2025-08-14T21:49:11.5843836Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5844066Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5844349Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5844556Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5844797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5845178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5845519Z return mod(**inputs) 2025-08-14T21:49:11.5845898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5846301Z outputs = self.model( 2025-08-14T21:49:11.5846710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5847115Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5847523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5847910Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5848258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5848614Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5849023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5849445Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5849858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5850291Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5850771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.5851260Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.5851447Z 2025-08-14T21:49:11.5851551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5851913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5852238Z return mod(**inputs) 2025-08-14T21:49:11.5852618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5853015Z outputs = self.model( 2025-08-14T21:49:11.5853399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5853808Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5854207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5854627Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5854976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5855336Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5855726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5856150Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5856569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5856997Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5857465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.5857949Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.5858116Z 2025-08-14T21:49:11.5858262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5858627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5858954Z return mod(**inputs) 2025-08-14T21:49:11.5859317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5859697Z outputs = self.model( 2025-08-14T21:49:11.5860065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5860475Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5860868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5861264Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5861616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5861979Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5862363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5862755Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5863155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.5863549Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.5863684Z 2025-08-14T21:49:11.5863797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5864147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5864471Z return mod(**inputs) 2025-08-14T21:49:11.5864833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5865209Z outputs = self.model( 2025-08-14T21:49:11.5865570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5865956Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5866331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5866703Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5867050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5867406Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5867781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5868228Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5868403Z 2025-08-14T21:49:11.5868508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5868869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5869186Z return mod(**inputs) 2025-08-14T21:49:11.5869550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5869926Z outputs = self.model( 2025-08-14T21:49:11.5870289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5870667Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5871047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5871430Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5871773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5872168Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5872554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5872981Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5873150Z 2025-08-14T21:49:11.5873253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5873616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5873955Z return mod(**inputs) 2025-08-14T21:49:11.5874342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5874712Z outputs = self.model( 2025-08-14T21:49:11.5875090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5875488Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5875953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5876377Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5876762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5877157Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5877578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.5877975Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.5878114Z 2025-08-14T21:49:11.5878227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5878581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5878944Z return mod(**inputs) 2025-08-14T21:49:11.5879348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5879766Z outputs = self.model( 2025-08-14T21:49:11.5880152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5880603Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5881015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5881459Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5881831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5882241Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5882683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5883117Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5883558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.5884084Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.5884309Z 2025-08-14T21:49:11.5884432Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5884821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5885174Z return mod(**inputs) 2025-08-14T21:49:11.5885567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5885976Z outputs = self.model( 2025-08-14T21:49:11.5886373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5886814Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5887252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5887660Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5888032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5888387Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5888771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5889164Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5889592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.5889982Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.5890116Z 2025-08-14T21:49:11.5890222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5890578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5890902Z return mod(**inputs) 2025-08-14T21:49:11.5891264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5891634Z outputs = self.model( 2025-08-14T21:49:11.5891992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5892375Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5892745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5893129Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5893487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5893869Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5894266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5894690Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5895083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.5895473Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.5895615Z 2025-08-14T21:49:11.5895696Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5895910Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5896121Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5896319Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5896577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5896938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5897265Z return mod(**inputs) 2025-08-14T21:49:11.5897623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5898001Z outputs = self.model( 2025-08-14T21:49:11.5898366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5898746Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5899127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5899512Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5899864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5900216Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5900632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5901034Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5901424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5901832Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5902275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.5902756Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.5902940Z 2025-08-14T21:49:11.5903043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5903417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5903744Z return mod(**inputs) 2025-08-14T21:49:11.5904106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5904477Z outputs = self.model( 2025-08-14T21:49:11.5904841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5905225Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5905592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5905975Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5906327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5906688Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5907064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5907466Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5907868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5908271Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5908829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.5909295Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.5909459Z 2025-08-14T21:49:11.5909572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5909929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5910280Z return mod(**inputs) 2025-08-14T21:49:11.5910666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5911125Z outputs = self.model( 2025-08-14T21:49:11.5911506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5911904Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5912309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5912713Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5913086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5913474Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5913894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5914315Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5914768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.5915209Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.5915354Z 2025-08-14T21:49:11.5915470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5915899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5916252Z return mod(**inputs) 2025-08-14T21:49:11.5916637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5917023Z outputs = self.model( 2025-08-14T21:49:11.5917391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5917803Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5918179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5918547Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5918890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5919247Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5919618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5920037Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5920210Z 2025-08-14T21:49:11.5920312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5920661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5920971Z return mod(**inputs) 2025-08-14T21:49:11.5921326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5921699Z outputs = self.model( 2025-08-14T21:49:11.5922055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5922427Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5922798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5923175Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5923509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5923866Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5924246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5924678Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5924843Z 2025-08-14T21:49:11.5924947Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5925299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5925619Z return mod(**inputs) 2025-08-14T21:49:11.5925973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5926335Z outputs = self.model( 2025-08-14T21:49:11.5926685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5927063Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5927410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5927782Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5928120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5928513Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5928884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.5929264Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.5929395Z 2025-08-14T21:49:11.5929504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5929844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5930161Z return mod(**inputs) 2025-08-14T21:49:11.5930518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5930893Z outputs = self.model( 2025-08-14T21:49:11.5931267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5931644Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5932017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5932388Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5932719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5933071Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5933451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:49:11.5951078Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.5951259Z 2025-08-14T21:49:11.5951380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5951793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5952137Z return mod(**inputs) 2025-08-14T21:49:11.5952544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5952944Z outputs = self.model( 2025-08-14T21:49:11.5953318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5953727Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5954142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5954550Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5954923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5955320Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5955847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5956404Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5956841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.5957310Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.5957517Z 2025-08-14T21:49:11.5957638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5957999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5958332Z return mod(**inputs) 2025-08-14T21:49:11.5958702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5959088Z outputs = self.model( 2025-08-14T21:49:11.5959451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5959874Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5960293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5960673Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5961024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5961389Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5961779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5962177Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5962604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.5963004Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.5963141Z 2025-08-14T21:49:11.5963260Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5963621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5963948Z return mod(**inputs) 2025-08-14T21:49:11.5964317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5964696Z outputs = self.model( 2025-08-14T21:49:11.5965067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5965456Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5965842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5966219Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5966572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5966941Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5967319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5967723Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5968121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.5968521Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.5968663Z 2025-08-14T21:49:11.5968750Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5968970Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5969179Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5969383Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.5969612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5969991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5970324Z return mod(**inputs) 2025-08-14T21:49:11.5970681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5971061Z outputs = self.model( 2025-08-14T21:49:11.5971423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5971810Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5972179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5972560Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5972909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5973265Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5973692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5974097Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5974496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5974899Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5975349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.5975835Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.5976020Z 2025-08-14T21:49:11.5976133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5976503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5976829Z return mod(**inputs) 2025-08-14T21:49:11.5977204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5977588Z outputs = self.model( 2025-08-14T21:49:11.5977956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5978355Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5978729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5979113Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5979468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5979835Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5980220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5980632Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5981039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.5981464Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.5981899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.5982354Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.5982515Z 2025-08-14T21:49:11.5982631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5983002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5983328Z return mod(**inputs) 2025-08-14T21:49:11.5983702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5984105Z outputs = self.model( 2025-08-14T21:49:11.5984456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5984834Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5985203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5985576Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5985919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5986282Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5986671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.5987074Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.5987571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.5988004Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.5988142Z 2025-08-14T21:49:11.5988258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5988614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5988943Z return mod(**inputs) 2025-08-14T21:49:11.5989306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5989688Z outputs = self.model( 2025-08-14T21:49:11.5990049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5990452Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5990840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5991224Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5991581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5991961Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5992378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5992802Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5992989Z 2025-08-14T21:49:11.5993102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5993482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5993845Z return mod(**inputs) 2025-08-14T21:49:11.5994220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5994630Z outputs = self.model( 2025-08-14T21:49:11.5995014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.5995424Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.5995910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.5996336Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.5996724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.5997126Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.5997548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.5997998Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.5998179Z 2025-08-14T21:49:11.5998286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.5998645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.5998964Z return mod(**inputs) 2025-08-14T21:49:11.5999325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.5999704Z outputs = self.model( 2025-08-14T21:49:11.6000057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6000439Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6000816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6001194Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6001536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6001928Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6002311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.6002696Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6002831Z 2025-08-14T21:49:11.6002935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6003291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6003613Z return mod(**inputs) 2025-08-14T21:49:11.6003965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6004380Z outputs = self.model( 2025-08-14T21:49:11.6004777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6005193Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6005588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6005996Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6006368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6006746Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6007136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6007539Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6007941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6008398Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6008611Z 2025-08-14T21:49:11.6008859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6009239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6009569Z return mod(**inputs) 2025-08-14T21:49:11.6009932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6010321Z outputs = self.model( 2025-08-14T21:49:11.6010695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6011080Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6011467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6011860Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6012280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6012639Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6013025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6013430Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6013831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6014215Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6014361Z 2025-08-14T21:49:11.6014467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6014828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6015150Z return mod(**inputs) 2025-08-14T21:49:11.6015520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6015965Z outputs = self.model( 2025-08-14T21:49:11.6016376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6016756Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6017134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6017516Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6017856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6018220Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6018640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6019030Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6019413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6019805Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6019948Z 2025-08-14T21:49:11.6020039Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6020254Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6020456Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6020663Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6020896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6021247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6021571Z return mod(**inputs) 2025-08-14T21:49:11.6021942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6022308Z outputs = self.model( 2025-08-14T21:49:11.6022676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6023066Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6023434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6023800Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6024140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6024504Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6024889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6025285Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6025702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6026153Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6026618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6027127Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6027328Z 2025-08-14T21:49:11.6027446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6027805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6028123Z return mod(**inputs) 2025-08-14T21:49:11.6028486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6028868Z outputs = self.model( 2025-08-14T21:49:11.6029229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6029633Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6030032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6030416Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6030757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6031120Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6031509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6031932Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6032404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6032811Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6033264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6033743Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6033914Z 2025-08-14T21:49:11.6034025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6034410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6034772Z return mod(**inputs) 2025-08-14T21:49:11.6035155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6035575Z outputs = self.model( 2025-08-14T21:49:11.6036024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6036457Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6036868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6037278Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6037650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6038028Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6038430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6038833Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6039230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6039615Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6039761Z 2025-08-14T21:49:11.6039867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6040252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6040581Z return mod(**inputs) 2025-08-14T21:49:11.6040936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6041319Z outputs = self.model( 2025-08-14T21:49:11.6041682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6042059Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6042439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6042821Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6043173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6043529Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6043932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6044388Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6044566Z 2025-08-14T21:49:11.6044675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6045046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6045379Z return mod(**inputs) 2025-08-14T21:49:11.6045750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6046132Z outputs = self.model( 2025-08-14T21:49:11.6046510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6046910Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6047277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6047643Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6047988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6048349Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6048723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6049151Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6049328Z 2025-08-14T21:49:11.6049444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6049792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6050102Z return mod(**inputs) 2025-08-14T21:49:11.6050452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6050823Z outputs = self.model( 2025-08-14T21:49:11.6051170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6051541Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6051908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6052272Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6052601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6052946Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6053320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.6053790Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6053923Z 2025-08-14T21:49:11.6054025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6054371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6054685Z return mod(**inputs) 2025-08-14T21:49:11.6055026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6055394Z outputs = self.model( 2025-08-14T21:49:11.6055743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6056116Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6056475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6056850Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6057193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6057574Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6057962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:49:11.6058353Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6058487Z 2025-08-14T21:49:11.6058600Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6058951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6059286Z return mod(**inputs) 2025-08-14T21:49:11.6059640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6060012Z outputs = self.model( 2025-08-14T21:49:11.6060376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6060752Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6061130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6061503Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6061849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6062207Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6062584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6062981Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6063377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6063835Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6064036Z 2025-08-14T21:49:11.6064147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6064509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6064848Z return mod(**inputs) 2025-08-14T21:49:11.6065215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6065586Z outputs = self.model( 2025-08-14T21:49:11.6065949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6066355Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6066765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6067172Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6067572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6067956Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6068355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6068762Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6069182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6069605Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6069747Z 2025-08-14T21:49:11.6069857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6070237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6070585Z return mod(**inputs) 2025-08-14T21:49:11.6070971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6071389Z outputs = self.model( 2025-08-14T21:49:11.6071794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6072205Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6072603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6073008Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6073377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6073760Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6074184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6074607Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6075029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6075455Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6075680Z 2025-08-14T21:49:11.6075780Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6076017Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6076246Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6077101Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6077350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6077734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6078076Z return mod(**inputs) 2025-08-14T21:49:11.6078455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6078859Z outputs = self.model( 2025-08-14T21:49:11.6079241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6079639Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6080037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6080440Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6080810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6081187Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6081573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6081977Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6082374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6082820Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6083271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6083752Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6083938Z 2025-08-14T21:49:11.6084044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6084424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6084768Z return mod(**inputs) 2025-08-14T21:49:11.6085153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6085547Z outputs = self.model( 2025-08-14T21:49:11.6085936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6086338Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6086755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6087138Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6087484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6087847Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6088226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6088624Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6089022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6089445Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6089886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6090340Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6090512Z 2025-08-14T21:49:11.6090616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6090974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6091288Z return mod(**inputs) 2025-08-14T21:49:11.6091648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6092026Z outputs = self.model( 2025-08-14T21:49:11.6092380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6092766Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6093144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6093528Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6093749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6093828Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6094080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6094169Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6094421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6094503Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6094507Z 2025-08-14T21:49:11.6094610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6094836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6094905Z return mod(**inputs) 2025-08-14T21:49:11.6095157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6095232Z outputs = self.model( 2025-08-14T21:49:11.6095483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6095561Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6095807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6095879Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6096109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6096188Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6096441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6096599Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6096603Z 2025-08-14T21:49:11.6096706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6096908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6096972Z return mod(**inputs) 2025-08-14T21:49:11.6097223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6097297Z outputs = self.model( 2025-08-14T21:49:11.6097543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6097655Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6097904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6097979Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6098215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6098290Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6098537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6098649Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6098652Z 2025-08-14T21:49:11.6098752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6098955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6099019Z return mod(**inputs) 2025-08-14T21:49:11.6099268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6099345Z outputs = self.model( 2025-08-14T21:49:11.6099593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6099672Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6099920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6099990Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6100214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6100290Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6100537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.6100646Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6100650Z 2025-08-14T21:49:11.6100752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6100956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6101020Z return mod(**inputs) 2025-08-14T21:49:11.6101266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6101342Z outputs = self.model( 2025-08-14T21:49:11.6101591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6101668Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6101912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6101984Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6102214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6102312Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6102568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6102665Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6102904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6103058Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6103062Z 2025-08-14T21:49:11.6103162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6103355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6103458Z return mod(**inputs) 2025-08-14T21:49:11.6103707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6103784Z outputs = self.model( 2025-08-14T21:49:11.6104035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6104105Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6104354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6104423Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6104640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6104728Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6104975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6105074Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6105323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6105403Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6105406Z 2025-08-14T21:49:11.6105516Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6105719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6105794Z return mod(**inputs) 2025-08-14T21:49:11.6106061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6106131Z outputs = self.model( 2025-08-14T21:49:11.6106407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6106482Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6106749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6106829Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6107046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6107131Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6107379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6107469Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6107731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6107814Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6107817Z 2025-08-14T21:49:11.6107906Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6107985Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6108060Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6108172Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6108272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6108463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6108534Z return mod(**inputs) 2025-08-14T21:49:11.6108910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6108981Z outputs = self.model( 2025-08-14T21:49:11.6109231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6109300Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6109609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6109686Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6109910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6110000Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6110265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6110367Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6110630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6110735Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6111056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6111205Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6111213Z 2025-08-14T21:49:11.6111330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6111547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6111617Z return mod(**inputs) 2025-08-14T21:49:11.6111908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6111981Z outputs = self.model( 2025-08-14T21:49:11.6112258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6112346Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6112617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6112704Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6112972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6113061Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6113336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6113435Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6113713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6113826Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6114147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6114272Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6114276Z 2025-08-14T21:49:11.6114388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6114602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6114708Z return mod(**inputs) 2025-08-14T21:49:11.6115023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6115106Z outputs = self.model( 2025-08-14T21:49:11.6115386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6115464Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6115835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6115920Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6116198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6116298Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6116569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6116676Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6116955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6117043Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6117047Z 2025-08-14T21:49:11.6117168Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6117380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6117461Z return mod(**inputs) 2025-08-14T21:49:11.6117743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6117819Z outputs = self.model( 2025-08-14T21:49:11.6118112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6118195Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6118477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6118560Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6118807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6118901Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6119181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6119313Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6119317Z 2025-08-14T21:49:11.6119440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6119679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6119759Z return mod(**inputs) 2025-08-14T21:49:11.6120045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6120115Z outputs = self.model( 2025-08-14T21:49:11.6120400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6120477Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6120756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6120844Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6121089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6121182Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6121494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6121643Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6121647Z 2025-08-14T21:49:11.6121766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6121977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6122056Z return mod(**inputs) 2025-08-14T21:49:11.6122338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6122410Z outputs = self.model( 2025-08-14T21:49:11.6122696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6122802Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6123090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6123176Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6123412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6123503Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6123780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.6123867Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6123871Z 2025-08-14T21:49:11.6123988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6124202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6124274Z return mod(**inputs) 2025-08-14T21:49:11.6124569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6124655Z outputs = self.model( 2025-08-14T21:49:11.6124925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6125002Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6125260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6125343Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6125577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6125661Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6125909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:49:11.6125988Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6126009Z 2025-08-14T21:49:11.6126118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6126315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6126379Z return mod(**inputs) 2025-08-14T21:49:11.6126633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6126698Z outputs = self.model( 2025-08-14T21:49:11.6126953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6127024Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6127273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6127354Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6127583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6127681Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6129039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6129155Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6129428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6129588Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6129593Z 2025-08-14T21:49:11.6129703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6129919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6130010Z return mod(**inputs) 2025-08-14T21:49:11.6130288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6130363Z outputs = self.model( 2025-08-14T21:49:11.6130628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6130715Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6130976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6131057Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6131291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6131375Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6131649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6131747Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6132013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6132107Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6132111Z 2025-08-14T21:49:11.6132220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6132437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6132505Z return mod(**inputs) 2025-08-14T21:49:11.6132775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6132856Z outputs = self.model( 2025-08-14T21:49:11.6133123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6133200Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6133492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6133569Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6133806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6133889Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6134161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6134258Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6134501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6134595Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6134598Z 2025-08-14T21:49:11.6134683Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6134765Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6134849Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6134947Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6135066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6135275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6135339Z return mod(**inputs) 2025-08-14T21:49:11.6135604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6135674Z outputs = self.model( 2025-08-14T21:49:11.6135938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6136022Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6136297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6136375Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6136617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6136699Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6136969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6137064Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6137328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6137439Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6137741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6137883Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6137888Z 2025-08-14T21:49:11.6137990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6138187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6138259Z return mod(**inputs) 2025-08-14T21:49:11.6138510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6138577Z outputs = self.model( 2025-08-14T21:49:11.6138833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6138904Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6139159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6139230Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6139447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6139561Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6139811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6139908Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6140152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6140247Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6140541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6140650Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6140654Z 2025-08-14T21:49:11.6140755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6140959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6141044Z return mod(**inputs) 2025-08-14T21:49:11.6141317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6141385Z outputs = self.model( 2025-08-14T21:49:11.6141632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6141713Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6141970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6142053Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6142302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6142388Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6142660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6142753Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6143017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6143110Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6143113Z 2025-08-14T21:49:11.6143218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6143431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6143499Z return mod(**inputs) 2025-08-14T21:49:11.6143768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6143843Z outputs = self.model( 2025-08-14T21:49:11.6144095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6144167Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6144437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6144510Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6144747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6144828Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6145089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6145224Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6145228Z 2025-08-14T21:49:11.6145334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6145565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6145634Z return mod(**inputs) 2025-08-14T21:49:11.6145898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6145974Z outputs = self.model( 2025-08-14T21:49:11.6146236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6146313Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6146581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6146654Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6146893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6146975Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6147284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6147432Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6147436Z 2025-08-14T21:49:11.6147541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6147754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6147823Z return mod(**inputs) 2025-08-14T21:49:11.6148086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6148167Z outputs = self.model( 2025-08-14T21:49:11.6148428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6148525Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6148804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6148881Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6149120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6149203Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6149468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.6149562Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6149566Z 2025-08-14T21:49:11.6149670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6149887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6149955Z return mod(**inputs) 2025-08-14T21:49:11.6150224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6150306Z outputs = self.model( 2025-08-14T21:49:11.6150576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6150651Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6150925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6151000Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6151239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6151321Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6151582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6151684Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6151967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6152131Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6152142Z 2025-08-14T21:49:11.6152250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6152459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6152533Z return mod(**inputs) 2025-08-14T21:49:11.6152797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6152872Z outputs = self.model( 2025-08-14T21:49:11.6153138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6153214Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6153478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6153595Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6153826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6153915Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6154174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6154267Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6154535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6154619Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6154623Z 2025-08-14T21:49:11.6154756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6154965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6155035Z return mod(**inputs) 2025-08-14T21:49:11.6155307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6155377Z outputs = self.model( 2025-08-14T21:49:11.6155733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6155820Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6156095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6156179Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6156434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6156521Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6156814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6156912Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6157201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6157294Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6157298Z 2025-08-14T21:49:11.6157384Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6157486Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6157566Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6157648Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6157764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6157975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6158083Z return mod(**inputs) 2025-08-14T21:49:11.6158347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6158420Z outputs = self.model( 2025-08-14T21:49:11.6158689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6158764Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6159042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6159115Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6159355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6159443Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6159714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6159829Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6160116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6160220Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6160535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6160675Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6160679Z 2025-08-14T21:49:11.6160785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6161000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6161068Z return mod(**inputs) 2025-08-14T21:49:11.6161372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6161444Z outputs = self.model( 2025-08-14T21:49:11.6161691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6161768Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6162034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6162106Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6162350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6162432Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6162708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6162801Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6163062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6163184Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6163470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6163585Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6163589Z 2025-08-14T21:49:11.6163691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6163896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6163973Z return mod(**inputs) 2025-08-14T21:49:11.6164235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6164308Z outputs = self.model( 2025-08-14T21:49:11.6164614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6164694Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6164977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6165052Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6165290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6165381Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6165662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6165757Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6166034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6166120Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6166145Z 2025-08-14T21:49:11.6166276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6166486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6166554Z return mod(**inputs) 2025-08-14T21:49:11.6166829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6166900Z outputs = self.model( 2025-08-14T21:49:11.6167170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6167246Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6167526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6167614Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6167846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6167930Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6168201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6168325Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6168329Z 2025-08-14T21:49:11.6168444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6168652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6168720Z return mod(**inputs) 2025-08-14T21:49:11.6168996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6169069Z outputs = self.model( 2025-08-14T21:49:11.6169347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6169429Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6169692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6169773Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6170000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6170080Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6170349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6170471Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6170475Z 2025-08-14T21:49:11.6170594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6170822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6170893Z return mod(**inputs) 2025-08-14T21:49:11.6171168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6171238Z outputs = self.model( 2025-08-14T21:49:11.6171507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6171584Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6171842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6171922Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6172150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6172234Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6172518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.6172620Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6172624Z 2025-08-14T21:49:11.6172736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6172942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6173011Z return mod(**inputs) 2025-08-14T21:49:11.6173281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6173353Z outputs = self.model( 2025-08-14T21:49:11.6173623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6173723Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6174005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6174089Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6174317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6174398Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6174664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:49:11.6174747Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6174751Z 2025-08-14T21:49:11.6174862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6175068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6175135Z return mod(**inputs) 2025-08-14T21:49:11.6175406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6175480Z outputs = self.model( 2025-08-14T21:49:11.6175741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6175825Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6176082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6176160Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6176388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6176467Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6176733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6176828Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6177124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6177287Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6177291Z 2025-08-14T21:49:11.6177398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6177611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6177681Z return mod(**inputs) 2025-08-14T21:49:11.6177942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6178022Z outputs = self.model( 2025-08-14T21:49:11.6178282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6178365Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6178625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6178740Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6178980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6179061Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6179328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6179423Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6179683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6179773Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6179776Z 2025-08-14T21:49:11.6179900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6180111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6180190Z return mod(**inputs) 2025-08-14T21:49:11.6180457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6180533Z outputs = self.model( 2025-08-14T21:49:11.6180797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6180871Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6181140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6181215Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6181449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6181541Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6181808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6181911Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6182172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6182262Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6182266Z 2025-08-14T21:49:11.6182358Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6182440Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6182527Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6182606Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6182714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6182931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6183024Z return mod(**inputs) 2025-08-14T21:49:11.6183302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6183385Z outputs = self.model( 2025-08-14T21:49:11.6183663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6183747Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6184037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6184112Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6184354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6184434Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6184704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6184808Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6185115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6185232Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6185548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6185685Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6185690Z 2025-08-14T21:49:11.6185804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6186008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6186085Z return mod(**inputs) 2025-08-14T21:49:11.6186368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6186446Z outputs = self.model( 2025-08-14T21:49:11.6186723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6186799Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6187061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6187144Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6187374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6187463Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6187731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6187824Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6188100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6188204Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6188519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6188635Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6188638Z 2025-08-14T21:49:11.6188745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6188961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6189030Z return mod(**inputs) 2025-08-14T21:49:11.6189295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6189375Z outputs = self.model( 2025-08-14T21:49:11.6189657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6189741Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6190002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6190077Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6190312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6190392Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6190658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6190750Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6191010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6191102Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6191122Z 2025-08-14T21:49:11.6191247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6191457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6191532Z return mod(**inputs) 2025-08-14T21:49:11.6191792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6191867Z outputs = self.model( 2025-08-14T21:49:11.6192131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6192206Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6192493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6192572Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6192804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6192899Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6193169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6193305Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6193309Z 2025-08-14T21:49:11.6193418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6193632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6193710Z return mod(**inputs) 2025-08-14T21:49:11.6193982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6194065Z outputs = self.model( 2025-08-14T21:49:11.6194340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6194422Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6194700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6194776Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6195016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6195107Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6195378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6195514Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6195518Z 2025-08-14T21:49:11.6195735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6196029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6196113Z return mod(**inputs) 2025-08-14T21:49:11.6196390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6196473Z outputs = self.model( 2025-08-14T21:49:11.6196751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6196829Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6197109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6197186Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6197429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6197527Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6197818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.6197933Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6197938Z 2025-08-14T21:49:11.6198048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6198263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6198344Z return mod(**inputs) 2025-08-14T21:49:11.6198644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6198725Z outputs = self.model( 2025-08-14T21:49:11.6199022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6199128Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6199435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6199519Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6199755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6199848Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6200139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6200243Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6200534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6200696Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6200701Z 2025-08-14T21:49:11.6200819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6201036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6201122Z return mod(**inputs) 2025-08-14T21:49:11.6201416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6201488Z outputs = self.model( 2025-08-14T21:49:11.6201787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6201864Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6202159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6202244Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6202505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6202619Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6202915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6203014Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6203303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6203388Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6203391Z 2025-08-14T21:49:11.6203507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6203721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6203792Z return mod(**inputs) 2025-08-14T21:49:11.6204089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6204163Z outputs = self.model( 2025-08-14T21:49:11.6204440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6204564Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6204859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6204942Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6205199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6205283Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6205577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6205672Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6205980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6206085Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6206090Z 2025-08-14T21:49:11.6206177Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6206267Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6206348Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6206430Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6206546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6206758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6206828Z return mod(**inputs) 2025-08-14T21:49:11.6207108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6207178Z outputs = self.model( 2025-08-14T21:49:11.6207469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6207551Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6207839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6207924Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6208157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6208246Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6208516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6208609Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6209007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6209119Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6209482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6209636Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6209640Z 2025-08-14T21:49:11.6209749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6209967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6210037Z return mod(**inputs) 2025-08-14T21:49:11.6210301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6210383Z outputs = self.model( 2025-08-14T21:49:11.6210647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6210731Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6210992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6211130Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6211370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6211454Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6211714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6211816Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6212077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6212187Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6212516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6212641Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6212646Z 2025-08-14T21:49:11.6212769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6212980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6213059Z return mod(**inputs) 2025-08-14T21:49:11.6213342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6213414Z outputs = self.model( 2025-08-14T21:49:11.6213685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6213761Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6214023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6214107Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6214337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6214428Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6214691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:49:11.6214784Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:49:11.6215051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6215135Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6215139Z 2025-08-14T21:49:11.6215250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6215456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6215524Z return mod(**inputs) 2025-08-14T21:49:11.6215825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6215899Z outputs = self.model( 2025-08-14T21:49:11.6216159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6216243Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6216503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6216584Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6216815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6216896Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6217166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6217293Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6217315Z 2025-08-14T21:49:11.6217448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6217656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6217724Z return mod(**inputs) 2025-08-14T21:49:11.6217999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6218070Z outputs = self.model( 2025-08-14T21:49:11.6218336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6218419Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6218698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6218783Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6219013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6219096Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6219365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:49:11.6219488Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6219492Z 2025-08-14T21:49:11.6219598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6219811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6219879Z return mod(**inputs) 2025-08-14T21:49:11.6220149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6220221Z outputs = self.model( 2025-08-14T21:49:11.6220484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6220570Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6220832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6220913Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6221142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6221222Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6221488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:49:11.6221572Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6221576Z 2025-08-14T21:49:11.6221684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6221914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6221983Z return mod(**inputs) 2025-08-14T21:49:11.6222254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6222324Z outputs = self.model( 2025-08-14T21:49:11.6222586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:49:11.6222668Z encoder_outputs = self.encoder( 2025-08-14T21:49:11.6222924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:49:11.6223005Z layer_outputs = encoder_layer( 2025-08-14T21:49:11.6223233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6223314Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6223582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:49:11.6223708Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6223713Z 2025-08-14T21:49:11.6223820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6224034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6224101Z return mod(**inputs) 2025-08-14T21:49:11.6224370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6224440Z outputs = self.model( 2025-08-14T21:49:11.6224701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6224812Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6225078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-08-14T21:49:11.6225257Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-08-14T21:49:11.6225507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:49:11.6225583Z return func(*args, **kwargs) 2025-08-14T21:49:11.6225850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:49:11.6226075Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:49:11.6226407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-14T21:49:11.6226664Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:49:11.6226670Z 2025-08-14T21:49:11.6226782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6226998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6227068Z return mod(**inputs) 2025-08-14T21:49:11.6227343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6227421Z outputs = self.model( 2025-08-14T21:49:11.6227683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6227767Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6228033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-08-14T21:49:11.6228212Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-08-14T21:49:11.6228493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:49:11.6228587Z return func(*args, **kwargs) 2025-08-14T21:49:11.6228850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:49:11.6229079Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:49:11.6229407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-14T21:49:11.6229608Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:49:11.6229612Z 2025-08-14T21:49:11.6229721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6229933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6230011Z return mod(**inputs) 2025-08-14T21:49:11.6230316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6230395Z outputs = self.model( 2025-08-14T21:49:11.6230658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6230733Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6231002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6231077Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6231318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6231422Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6231696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6231817Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6232092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6232258Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6232270Z 2025-08-14T21:49:11.6232381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6232597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6232675Z return mod(**inputs) 2025-08-14T21:49:11.6232946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6233020Z outputs = self.model( 2025-08-14T21:49:11.6233303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6233385Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6233664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6233740Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6233979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6234074Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6234344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6234452Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6234734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6234842Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6234847Z 2025-08-14T21:49:11.6234968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6235185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6235254Z return mod(**inputs) 2025-08-14T21:49:11.6235543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6235676Z outputs = self.model( 2025-08-14T21:49:11.6236007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6236087Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6236361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6236448Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6236684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6236818Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6237098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6237212Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6237482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6237571Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6237576Z 2025-08-14T21:49:11.6237660Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6237754Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6237843Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6237934Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6238048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6238249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6238322Z return mod(**inputs) 2025-08-14T21:49:11.6238569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6238637Z outputs = self.model( 2025-08-14T21:49:11.6238895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6238967Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6239222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6239303Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6239535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6239626Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6239909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6240010Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6240297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6240398Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6240720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6240863Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6240867Z 2025-08-14T21:49:11.6240976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6241195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6241285Z return mod(**inputs) 2025-08-14T21:49:11.6241556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6241634Z outputs = self.model( 2025-08-14T21:49:11.6241910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6241993Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6242270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6242342Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6242579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6242663Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6242950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6243094Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6243369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6243478Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6243788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6243902Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6243913Z 2025-08-14T21:49:11.6244018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6244223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6244318Z return mod(**inputs) 2025-08-14T21:49:11.6244606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6244680Z outputs = self.model( 2025-08-14T21:49:11.6244964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6245040Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6245322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6245396Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6245635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6245725Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6245990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6246092Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6246380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6246464Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6246467Z 2025-08-14T21:49:11.6246581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6246790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6246858Z return mod(**inputs) 2025-08-14T21:49:11.6247126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6247196Z outputs = self.model( 2025-08-14T21:49:11.6247478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6247593Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6247874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6247956Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6248200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6248282Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6248567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6248681Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6248967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6249124Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6249130Z 2025-08-14T21:49:11.6249237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6249519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6249589Z return mod(**inputs) 2025-08-14T21:49:11.6249859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6249929Z outputs = self.model( 2025-08-14T21:49:11.6250197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6250281Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6250542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6250617Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6250872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6250957Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6251229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6251343Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6251604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6251698Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6251701Z 2025-08-14T21:49:11.6251807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6252026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6252094Z return mod(**inputs) 2025-08-14T21:49:11.6252365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6252446Z outputs = self.model( 2025-08-14T21:49:11.6252720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6252795Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6253073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6253147Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6253388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6253470Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6253736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6253860Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6254140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6254233Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6254243Z 2025-08-14T21:49:11.6254325Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6254406Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6254494Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6254573Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6254679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6254892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6254960Z return mod(**inputs) 2025-08-14T21:49:11.6255222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6255299Z outputs = self.model( 2025-08-14T21:49:11.6255562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6255683Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6255947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6256021Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6256258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6256341Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6256608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6256719Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6256997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6257110Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6257418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6257557Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6257570Z 2025-08-14T21:49:11.6257675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6257882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6257957Z return mod(**inputs) 2025-08-14T21:49:11.6258224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6258294Z outputs = self.model( 2025-08-14T21:49:11.6258564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6258639Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6258911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6258987Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6259217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6259307Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6259570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6259682Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6259954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6260058Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6260392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6260508Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6260512Z 2025-08-14T21:49:11.6260619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6260837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6260906Z return mod(**inputs) 2025-08-14T21:49:11.6261183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6261253Z outputs = self.model( 2025-08-14T21:49:11.6261532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6261618Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6261897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6261992Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6262254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6262340Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6262619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6262728Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6263005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6263101Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6263104Z 2025-08-14T21:49:11.6263233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6263456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6263538Z return mod(**inputs) 2025-08-14T21:49:11.6263812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6263890Z outputs = self.model( 2025-08-14T21:49:11.6264160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6264235Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6264517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6264590Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6264873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6264958Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6265218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6265356Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6265360Z 2025-08-14T21:49:11.6265465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6265688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6265758Z return mod(**inputs) 2025-08-14T21:49:11.6266084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6266164Z outputs = self.model( 2025-08-14T21:49:11.6266448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6266527Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6266819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6266916Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6267172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6267256Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6267541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6267677Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6267681Z 2025-08-14T21:49:11.6267792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6268018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6268088Z return mod(**inputs) 2025-08-14T21:49:11.6268376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6268477Z outputs = self.model( 2025-08-14T21:49:11.6268775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6268855Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6269144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6269223Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6269475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6269559Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6269835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6269949Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6269954Z 2025-08-14T21:49:11.6270065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6270281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6270357Z return mod(**inputs) 2025-08-14T21:49:11.6270646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6270726Z outputs = self.model( 2025-08-14T21:49:11.6271007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6271083Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6271362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6271437Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6271681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6271767Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6272041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6272155Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6272424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6272588Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6272600Z 2025-08-14T21:49:11.6272712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6272929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6273006Z return mod(**inputs) 2025-08-14T21:49:11.6273281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6273383Z outputs = self.model( 2025-08-14T21:49:11.6273669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6273745Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6274021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6274096Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6274335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6274426Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6274696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6274802Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6275097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6275201Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6275204Z 2025-08-14T21:49:11.6275322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6275535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6275683Z return mod(**inputs) 2025-08-14T21:49:11.6275976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6276049Z outputs = self.model( 2025-08-14T21:49:11.6276328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6276431Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6276706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6276796Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6277033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6277120Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6277400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6277507Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6277783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6277875Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6277880Z 2025-08-14T21:49:11.6277969Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6278068Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6278151Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6278237Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6278362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6278578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6278656Z return mod(**inputs) 2025-08-14T21:49:11.6278918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6278985Z outputs = self.model( 2025-08-14T21:49:11.6279252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6279326Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6279591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6279698Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6279932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6280021Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6280285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6280386Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6280657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6280758Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6281068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6281209Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6281214Z 2025-08-14T21:49:11.6281320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6281579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6281650Z return mod(**inputs) 2025-08-14T21:49:11.6281917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6281996Z outputs = self.model( 2025-08-14T21:49:11.6282261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6282344Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6282607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6282696Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6282937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6283021Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6283289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6283391Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6283648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6283758Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6284058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6284171Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6284183Z 2025-08-14T21:49:11.6284292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6284501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6284578Z return mod(**inputs) 2025-08-14T21:49:11.6284840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6284909Z outputs = self.model( 2025-08-14T21:49:11.6285177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6285252Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6285518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6285593Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6285826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6285937Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6286197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6286299Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6286568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6286652Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6286656Z 2025-08-14T21:49:11.6286770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6286978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6287045Z return mod(**inputs) 2025-08-14T21:49:11.6287315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6287386Z outputs = self.model( 2025-08-14T21:49:11.6287656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6287767Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6288033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6288114Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6288345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6288425Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6288692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-08-14T21:49:11.6288774Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6288777Z 2025-08-14T21:49:11.6288908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6289118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6289188Z return mod(**inputs) 2025-08-14T21:49:11.6289459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6289529Z outputs = self.model( 2025-08-14T21:49:11.6289791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6289874Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6290135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6290214Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6290445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6290526Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6290797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6290914Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6291183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6291342Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6291346Z 2025-08-14T21:49:11.6291453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6291669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6291736Z return mod(**inputs) 2025-08-14T21:49:11.6292008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6292099Z outputs = self.model( 2025-08-14T21:49:11.6292363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6292447Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6292714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6292787Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6293024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6293104Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6293372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6293484Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6293746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6293857Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6293861Z 2025-08-14T21:49:11.6293984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6294198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6294266Z return mod(**inputs) 2025-08-14T21:49:11.6294529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6294608Z outputs = self.model( 2025-08-14T21:49:11.6294868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6294944Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6295237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6295314Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6295552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6295637Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6295897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6296018Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6296276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6296364Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6296376Z 2025-08-14T21:49:11.6296458Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6296541Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6296627Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6296710Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6296816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6297034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6297101Z return mod(**inputs) 2025-08-14T21:49:11.6297361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6297439Z outputs = self.model( 2025-08-14T21:49:11.6297702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6297785Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6298045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6298121Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6298379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6298462Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6298728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6298849Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6299111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6299219Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6299520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6299659Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6299671Z 2025-08-14T21:49:11.6299780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6299991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6300107Z return mod(**inputs) 2025-08-14T21:49:11.6300371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6300439Z outputs = self.model( 2025-08-14T21:49:11.6300706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6300780Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6301045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6301119Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6301363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6301455Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6301716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6301825Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6302093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6302193Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6302500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6302612Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6302615Z 2025-08-14T21:49:11.6302720Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6302933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6303003Z return mod(**inputs) 2025-08-14T21:49:11.6303294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6303364Z outputs = self.model( 2025-08-14T21:49:11.6303637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6303721Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6303993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6304066Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6304312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6304394Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6304674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6304804Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6305077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6305168Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6305172Z 2025-08-14T21:49:11.6305278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6305495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6305563Z return mod(**inputs) 2025-08-14T21:49:11.6305836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6305913Z outputs = self.model( 2025-08-14T21:49:11.6306187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6306265Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6306579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6306656Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6306901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6306982Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6307255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6307388Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6307391Z 2025-08-14T21:49:11.6307499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6307724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6307803Z return mod(**inputs) 2025-08-14T21:49:11.6308091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6308169Z outputs = self.model( 2025-08-14T21:49:11.6308504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6308581Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6309028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6309109Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6309357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6309439Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6309702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6309841Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6309844Z 2025-08-14T21:49:11.6309951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6310160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6310238Z return mod(**inputs) 2025-08-14T21:49:11.6310512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6310590Z outputs = self.model( 2025-08-14T21:49:11.6310863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6310942Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6311214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6311417Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6311656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6311737Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6311998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6312093Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6312097Z 2025-08-14T21:49:11.6312205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6312413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6312491Z return mod(**inputs) 2025-08-14T21:49:11.6312814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6312894Z outputs = self.model( 2025-08-14T21:49:11.6313197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6313304Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6313582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6313657Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6313890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6313980Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6314251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6314364Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6314660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6314823Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6314827Z 2025-08-14T21:49:11.6314942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6315152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6315227Z return mod(**inputs) 2025-08-14T21:49:11.6315496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6315568Z outputs = self.model( 2025-08-14T21:49:11.6316079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6316163Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6316438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6316525Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6316762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6316857Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6317126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6317232Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6317509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6317595Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6317601Z 2025-08-14T21:49:11.6317719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6317934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6318028Z return mod(**inputs) 2025-08-14T21:49:11.6318309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6318381Z outputs = self.model( 2025-08-14T21:49:11.6318650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6318738Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6319008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6319091Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6319330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6319417Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6319694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6319852Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6320129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6320221Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6320225Z 2025-08-14T21:49:11.6320312Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6320407Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6320489Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6320571Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6320688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6320901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6321000Z return mod(**inputs) 2025-08-14T21:49:11.6321275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6321351Z outputs = self.model( 2025-08-14T21:49:11.6321625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6321704Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6321974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6322057Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6322292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6322383Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6322654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6322761Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6323040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6323145Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6323469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6323611Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6323615Z 2025-08-14T21:49:11.6323726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6323945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6324016Z return mod(**inputs) 2025-08-14T21:49:11.6324289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6324391Z outputs = self.model( 2025-08-14T21:49:11.6324666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6324749Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6325019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6325096Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6325343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6325426Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6325695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6325807Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6326083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6326235Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6326547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6326663Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6326667Z 2025-08-14T21:49:11.6326783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6326995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6327072Z return mod(**inputs) 2025-08-14T21:49:11.6327343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6327414Z outputs = self.model( 2025-08-14T21:49:11.6327709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6327804Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6328079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6328151Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6328383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6328471Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6328738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6328838Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6329116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6329202Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6329206Z 2025-08-14T21:49:11.6329321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6329531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6329600Z return mod(**inputs) 2025-08-14T21:49:11.6329875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6329944Z outputs = self.model( 2025-08-14T21:49:11.6330212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6330295Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6330562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6330645Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6330896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6330981Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6331251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6331362Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6331631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6331788Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6331792Z 2025-08-14T21:49:11.6331899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6332114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6332185Z return mod(**inputs) 2025-08-14T21:49:11.6332452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6332566Z outputs = self.model( 2025-08-14T21:49:11.6332834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6332918Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6333185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6333254Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6333482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6333559Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6333834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6333944Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6334196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6334284Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6334287Z 2025-08-14T21:49:11.6334392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6334589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6334665Z return mod(**inputs) 2025-08-14T21:49:11.6334912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6334992Z outputs = self.model( 2025-08-14T21:49:11.6335241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6335317Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6335574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6335649Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6335875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6335956Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6336205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6336320Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6336571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6339307Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6339313Z 2025-08-14T21:49:11.6339407Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6339486Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6339567Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6339655Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6339758Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6339964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6340039Z return mod(**inputs) 2025-08-14T21:49:11.6340293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6340372Z outputs = self.model( 2025-08-14T21:49:11.6340623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6340695Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6340990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6341092Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6341333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6341412Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6341662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6341768Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6342020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6342118Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6342418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6342561Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6342566Z 2025-08-14T21:49:11.6342668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6342867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6342940Z return mod(**inputs) 2025-08-14T21:49:11.6343192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6343267Z outputs = self.model( 2025-08-14T21:49:11.6343522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6343599Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6343874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6343949Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6344193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6344280Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6344549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6344662Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6344935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6345046Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6345357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6345534Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6345538Z 2025-08-14T21:49:11.6345642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6345844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6345916Z return mod(**inputs) 2025-08-14T21:49:11.6346168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6346241Z outputs = self.model( 2025-08-14T21:49:11.6346493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6346570Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6346844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6346919Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6347154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6347241Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6347556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6347676Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6347949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6348032Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6348035Z 2025-08-14T21:49:11.6348150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6348358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6348432Z return mod(**inputs) 2025-08-14T21:49:11.6348736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6348811Z outputs = self.model( 2025-08-14T21:49:11.6349086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6349162Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6349433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6349514Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6349753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6349840Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6350111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-08-14T21:49:11.6350198Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6350202Z 2025-08-14T21:49:11.6350317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6350526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6350600Z return mod(**inputs) 2025-08-14T21:49:11.6350903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6350973Z outputs = self.model( 2025-08-14T21:49:11.6351245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6351319Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6351590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6351671Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6351938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6352027Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6352293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6352418Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6352422Z 2025-08-14T21:49:11.6352535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6352742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6352815Z return mod(**inputs) 2025-08-14T21:49:11.6353089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6353159Z outputs = self.model( 2025-08-14T21:49:11.6353434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6353510Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6353806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6353908Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6354134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6354221Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6354493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6354617Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6354620Z 2025-08-14T21:49:11.6354735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6354962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6355035Z return mod(**inputs) 2025-08-14T21:49:11.6355312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6355382Z outputs = self.model( 2025-08-14T21:49:11.6355741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6355826Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6356089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6356176Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6356411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6356504Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6356778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6356866Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6356873Z 2025-08-14T21:49:11.6356993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6357206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6357289Z return mod(**inputs) 2025-08-14T21:49:11.6357559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6357629Z outputs = self.model( 2025-08-14T21:49:11.6357899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6357977Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6358240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6358361Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6358592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6358685Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6358947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6359051Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6359317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6359474Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6359478Z 2025-08-14T21:49:11.6359585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6359803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6359879Z return mod(**inputs) 2025-08-14T21:49:11.6360152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6360235Z outputs = self.model( 2025-08-14T21:49:11.6360481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6360561Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6360813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6360887Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6361095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6361171Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6361432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6361529Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6361777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6361869Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6361873Z 2025-08-14T21:49:11.6361980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6362194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6362262Z return mod(**inputs) 2025-08-14T21:49:11.6362524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6362603Z outputs = self.model( 2025-08-14T21:49:11.6362871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6362955Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6363229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6363303Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6363546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6363629Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6363896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6364003Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6364255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6364370Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6364375Z 2025-08-14T21:49:11.6364455Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6364532Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6364617Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6364690Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6364791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6364992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6365056Z return mod(**inputs) 2025-08-14T21:49:11.6365311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6365377Z outputs = self.model( 2025-08-14T21:49:11.6365623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6365703Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6365947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6366050Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6366277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6366354Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6366612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6366707Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6366955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6367057Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6367363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6367502Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6367507Z 2025-08-14T21:49:11.6367610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6367805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6367877Z return mod(**inputs) 2025-08-14T21:49:11.6368125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6368192Z outputs = self.model( 2025-08-14T21:49:11.6368447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6368516Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6368772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6368843Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6369059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6369145Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6369392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6369493Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6369740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6369833Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6370124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6370268Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6370272Z 2025-08-14T21:49:11.6370373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6370584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6370653Z return mod(**inputs) 2025-08-14T21:49:11.6370921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6370991Z outputs = self.model( 2025-08-14T21:49:11.6371258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6371339Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6371587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6371667Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6371885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6371981Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6372248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6372346Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6372606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6372696Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6372700Z 2025-08-14T21:49:11.6372805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6373018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6373086Z return mod(**inputs) 2025-08-14T21:49:11.6373367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6373449Z outputs = self.model( 2025-08-14T21:49:11.6373713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6373796Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6374054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6374126Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6374360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6374442Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6374701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6374826Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6375090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6375251Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6375255Z 2025-08-14T21:49:11.6375356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6375554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6375629Z return mod(**inputs) 2025-08-14T21:49:11.6375891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6375969Z outputs = self.model( 2025-08-14T21:49:11.6376230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6376336Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6376605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6376684Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6376909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6376999Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6377257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6377378Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6377638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6377721Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6377729Z 2025-08-14T21:49:11.6377846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6378053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6378453Z return mod(**inputs) 2025-08-14T21:49:11.6378739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6378812Z outputs = self.model( 2025-08-14T21:49:11.6379083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6379160Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6379422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6379505Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6379748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6379844Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6380110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6380225Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6380497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6380587Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6380591Z 2025-08-14T21:49:11.6380682Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6380765Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6380846Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6380932Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6381041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6381253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6381327Z return mod(**inputs) 2025-08-14T21:49:11.6381594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6381665Z outputs = self.model( 2025-08-14T21:49:11.6381938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6382012Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6382281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6382355Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6382593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6382684Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6382980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6383103Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6383376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6383482Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6383803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6383946Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6383950Z 2025-08-14T21:49:11.6384060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6384283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6384355Z return mod(**inputs) 2025-08-14T21:49:11.6384637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6384730Z outputs = self.model( 2025-08-14T21:49:11.6385022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6385112Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6385389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6385473Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6385710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6385795Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6386073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6386193Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6386445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6386551Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6386836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6386950Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6386953Z 2025-08-14T21:49:11.6387053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6387248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6387324Z return mod(**inputs) 2025-08-14T21:49:11.6387574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6387652Z outputs = self.model( 2025-08-14T21:49:11.6387902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6387979Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6388237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6388308Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6388526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6388613Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6388861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6388974Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6389248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6389330Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6389335Z 2025-08-14T21:49:11.6389444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6389642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6389714Z return mod(**inputs) 2025-08-14T21:49:11.6389964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6390030Z outputs = self.model( 2025-08-14T21:49:11.6390289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6390360Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6390610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6390690Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6390940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6391025Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6391272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6391394Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6391397Z 2025-08-14T21:49:11.6391505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6391699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6391770Z return mod(**inputs) 2025-08-14T21:49:11.6392040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6392109Z outputs = self.model( 2025-08-14T21:49:11.6392365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6392440Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6392684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6392764Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6392978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6393064Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6393319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6393442Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6393448Z 2025-08-14T21:49:11.6393565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6393770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6393853Z return mod(**inputs) 2025-08-14T21:49:11.6394106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6394172Z outputs = self.model( 2025-08-14T21:49:11.6394424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6394495Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6394741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6394817Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6395033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6395142Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6395399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6395486Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6395490Z 2025-08-14T21:49:11.6395691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6395916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6395988Z return mod(**inputs) 2025-08-14T21:49:11.6396269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6396340Z outputs = self.model( 2025-08-14T21:49:11.6396629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6396711Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6396972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6397104Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6397335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6397423Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6397684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-08-14T21:49:11.6397768Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6397771Z 2025-08-14T21:49:11.6397885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6398102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6398185Z return mod(**inputs) 2025-08-14T21:49:11.6398446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6398515Z outputs = self.model( 2025-08-14T21:49:11.6398767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6398839Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6399084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6399161Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6399378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6399450Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6399691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6399785Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6400032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6400178Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6400181Z 2025-08-14T21:49:11.6400279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6400480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6400542Z return mod(**inputs) 2025-08-14T21:49:11.6400797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6400864Z outputs = self.model( 2025-08-14T21:49:11.6401121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6401219Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6401464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6401533Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6401756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6401830Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6402084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6402178Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6402425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6402510Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6402517Z 2025-08-14T21:49:11.6402617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6402816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6402912Z return mod(**inputs) 2025-08-14T21:49:11.6403157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6403230Z outputs = self.model( 2025-08-14T21:49:11.6403473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6403543Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6403795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6403863Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6404106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6404192Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6404464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6404579Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6404844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6404944Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6404948Z 2025-08-14T21:49:11.6405036Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6405133Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6405224Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6405305Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6405416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6405635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6405707Z return mod(**inputs) 2025-08-14T21:49:11.6405992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6406062Z outputs = self.model( 2025-08-14T21:49:11.6406313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6406395Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6406657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6406728Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6406949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6407045Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6407289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6407386Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6407625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6407724Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6408007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6408146Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6408149Z 2025-08-14T21:49:11.6408251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6408444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6408519Z return mod(**inputs) 2025-08-14T21:49:11.6408901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6409042Z outputs = self.model( 2025-08-14T21:49:11.6409332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6409407Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6409711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6409779Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6409991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6410075Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6410346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6410444Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6410695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6410789Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6411084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6411191Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6411195Z 2025-08-14T21:49:11.6411294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6411495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6411560Z return mod(**inputs) 2025-08-14T21:49:11.6411818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6411886Z outputs = self.model( 2025-08-14T21:49:11.6412154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6412240Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6412511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6412590Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6412825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6412906Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6413180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6413279Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6413586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6413681Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6413686Z 2025-08-14T21:49:11.6413793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6414008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6414071Z return mod(**inputs) 2025-08-14T21:49:11.6414319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6414390Z outputs = self.model( 2025-08-14T21:49:11.6414636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6414707Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6414961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6415032Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6415304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6415388Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6415660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6415780Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6416057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6416223Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6416227Z 2025-08-14T21:49:11.6416334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6416562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6416643Z return mod(**inputs) 2025-08-14T21:49:11.6416931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6417001Z outputs = self.model( 2025-08-14T21:49:11.6417278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6417347Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6417595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6417665Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6417883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6417975Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6418221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6418344Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6418625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6418709Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6418712Z 2025-08-14T21:49:11.6418828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6419033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6419100Z return mod(**inputs) 2025-08-14T21:49:11.6419380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6419471Z outputs = self.model( 2025-08-14T21:49:11.6419746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6419822Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6420091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6420174Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6420404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6420492Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6420754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6420866Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6421142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6421232Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6421236Z 2025-08-14T21:49:11.6421341Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6421445Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6421527Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6421612Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6421720Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6421925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6422004Z return mod(**inputs) 2025-08-14T21:49:11.6422269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6422338Z outputs = self.model( 2025-08-14T21:49:11.6422628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6422706Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6422979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6423054Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6423287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6423377Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6423644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6423754Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6424029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6424134Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6424460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6424615Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6424619Z 2025-08-14T21:49:11.6424724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6424939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6425009Z return mod(**inputs) 2025-08-14T21:49:11.6425284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6425355Z outputs = self.model( 2025-08-14T21:49:11.6425623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6425707Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6425995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6426070Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6426306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6426388Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6426657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6426766Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6427022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6427130Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6427432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6427552Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6427574Z 2025-08-14T21:49:11.6427701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6427908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6427984Z return mod(**inputs) 2025-08-14T21:49:11.6428249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6428329Z outputs = self.model( 2025-08-14T21:49:11.6428594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6428674Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6428945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6429045Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6429280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6429371Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6429634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6429752Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6430015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6430101Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6430104Z 2025-08-14T21:49:11.6430220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6430428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6430500Z return mod(**inputs) 2025-08-14T21:49:11.6430771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6430845Z outputs = self.model( 2025-08-14T21:49:11.6431116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6431194Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6431462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6431543Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6431761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6431845Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6432100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6432248Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6432253Z 2025-08-14T21:49:11.6432368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6432576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6432645Z return mod(**inputs) 2025-08-14T21:49:11.6432917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6432987Z outputs = self.model( 2025-08-14T21:49:11.6433256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6433332Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6433597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6433680Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6433911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6434032Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6434294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6434419Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6434422Z 2025-08-14T21:49:11.6434535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6434742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6434811Z return mod(**inputs) 2025-08-14T21:49:11.6435078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6435169Z outputs = self.model( 2025-08-14T21:49:11.6435449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6435529Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6435941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6436042Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6436281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6436376Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6436649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6436736Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6436740Z 2025-08-14T21:49:11.6436868Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6437083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6437156Z return mod(**inputs) 2025-08-14T21:49:11.6437434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6437501Z outputs = self.model( 2025-08-14T21:49:11.6437762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6437834Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6438083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6438162Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6438379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6438484Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6438740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6438841Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6439094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6439245Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6439249Z 2025-08-14T21:49:11.6439349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6439553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6439618Z return mod(**inputs) 2025-08-14T21:49:11.6439872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6439942Z outputs = self.model( 2025-08-14T21:49:11.6440190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6440314Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6440562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6440633Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6440856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6440934Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6441187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6441283Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6441547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6441637Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6441642Z 2025-08-14T21:49:11.6441745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6441948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6442012Z return mod(**inputs) 2025-08-14T21:49:11.6442262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6442334Z outputs = self.model( 2025-08-14T21:49:11.6442582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6442653Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6442911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6442982Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6443204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6443286Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6443536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6443638Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6443887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6443978Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6443982Z 2025-08-14T21:49:11.6444061Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6444149Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6444276Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6444351Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6444448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6444651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6444714Z return mod(**inputs) 2025-08-14T21:49:11.6444976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6445046Z outputs = self.model( 2025-08-14T21:49:11.6445309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6445392Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6445656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6445724Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6445951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6446029Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6446334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6446434Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6446678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6446782Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6447068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6447207Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6447210Z 2025-08-14T21:49:11.6447329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6447534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6447610Z return mod(**inputs) 2025-08-14T21:49:11.6447852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6447918Z outputs = self.model( 2025-08-14T21:49:11.6448166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6448236Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6448480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6448550Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6448760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6448847Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6449085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6449185Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6449430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6449523Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6449807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6449912Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6449915Z 2025-08-14T21:49:11.6450014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6450212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6450296Z return mod(**inputs) 2025-08-14T21:49:11.6450548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6450619Z outputs = self.model( 2025-08-14T21:49:11.6450859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6450936Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6451181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6451248Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6451465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6451541Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6451799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6451895Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6452179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6452272Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6452276Z 2025-08-14T21:49:11.6452386Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6452584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6452649Z return mod(**inputs) 2025-08-14T21:49:11.6452896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6452967Z outputs = self.model( 2025-08-14T21:49:11.6453230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6453306Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6453556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6453627Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6453844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6453919Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6454159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-08-14T21:49:11.6454243Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6454247Z 2025-08-14T21:49:11.6454344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6454543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6454609Z return mod(**inputs) 2025-08-14T21:49:11.6454851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6454926Z outputs = self.model( 2025-08-14T21:49:11.6455169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6455236Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6455487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6455552Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6455769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6455843Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6456084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6456216Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6456458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6456610Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6456614Z 2025-08-14T21:49:11.6456711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6456903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6456974Z return mod(**inputs) 2025-08-14T21:49:11.6457217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6457281Z outputs = self.model( 2025-08-14T21:49:11.6457531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6457601Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6457883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6457953Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6458163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6458247Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6458498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6458605Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6458842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6458932Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6458936Z 2025-08-14T21:49:11.6459040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6459228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6459290Z return mod(**inputs) 2025-08-14T21:49:11.6459535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6459597Z outputs = self.model( 2025-08-14T21:49:11.6459841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6459910Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6460145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6460220Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6460431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6460505Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6460752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6460855Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6461094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6461175Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6461178Z 2025-08-14T21:49:11.6461253Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6461338Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6461412Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6461494Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6461626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6461818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6461893Z return mod(**inputs) 2025-08-14T21:49:11.6462142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6462208Z outputs = self.model( 2025-08-14T21:49:11.6462460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6462533Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6462799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6462873Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6463100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6463194Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6463454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6463602Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6463872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6463977Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6464288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6464427Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6464431Z 2025-08-14T21:49:11.6464537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6464782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6464856Z return mod(**inputs) 2025-08-14T21:49:11.6465140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6465212Z outputs = self.model( 2025-08-14T21:49:11.6465475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6465557Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6465817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6465892Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6466130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6466212Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6466484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6466595Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6466860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6466971Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6467272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6467391Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6467395Z 2025-08-14T21:49:11.6467502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6467710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6467789Z return mod(**inputs) 2025-08-14T21:49:11.6468077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6468149Z outputs = self.model( 2025-08-14T21:49:11.6468420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6468496Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6468763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6468836Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6469063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6469153Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6469413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6469532Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6469791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6469913Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6469918Z 2025-08-14T21:49:11.6470034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6470241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6470309Z return mod(**inputs) 2025-08-14T21:49:11.6470580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6470650Z outputs = self.model( 2025-08-14T21:49:11.6470922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6471018Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6471288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6471373Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6471616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6471704Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6471980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6472108Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6472112Z 2025-08-14T21:49:11.6472226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6472435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6472505Z return mod(**inputs) 2025-08-14T21:49:11.6472794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6472866Z outputs = self.model( 2025-08-14T21:49:11.6473150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6473225Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6473506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6473589Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6473837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6473922Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6474217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6474367Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6474371Z 2025-08-14T21:49:11.6474488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6474706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6474778Z return mod(**inputs) 2025-08-14T21:49:11.6475074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6475149Z outputs = self.model( 2025-08-14T21:49:11.6475439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6475521Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6475922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6476025Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6476265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6476375Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6476682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6476776Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6476780Z 2025-08-14T21:49:11.6476906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6477128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6477204Z return mod(**inputs) 2025-08-14T21:49:11.6477507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6477581Z outputs = self.model( 2025-08-14T21:49:11.6477887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6477966Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6478244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6478327Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6478569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6478650Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6478930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6479037Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6479318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6479482Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6479486Z 2025-08-14T21:49:11.6479593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6479808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6479877Z return mod(**inputs) 2025-08-14T21:49:11.6480212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6480292Z outputs = self.model( 2025-08-14T21:49:11.6480542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6480621Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6480870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6480961Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6481191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6481271Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6481529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6481627Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6481879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6481966Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6481969Z 2025-08-14T21:49:11.6482067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6482273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6482339Z return mod(**inputs) 2025-08-14T21:49:11.6482594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6482687Z outputs = self.model( 2025-08-14T21:49:11.6482964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6483042Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6483312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6483385Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6483618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6483699Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6483984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6484099Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6484363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6484462Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6484465Z 2025-08-14T21:49:11.6484548Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6484631Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6484720Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6484798Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6484904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6485119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6485188Z return mod(**inputs) 2025-08-14T21:49:11.6485450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6485529Z outputs = self.model( 2025-08-14T21:49:11.6485792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6485876Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6486139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6486209Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6486432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6486511Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6486765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6486861Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6487128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6487234Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6487520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6487652Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6487662Z 2025-08-14T21:49:11.6487760Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6487966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6488042Z return mod(**inputs) 2025-08-14T21:49:11.6488309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6488378Z outputs = self.model( 2025-08-14T21:49:11.6488649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6488745Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6489028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6489102Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6489329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6489416Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6489678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6489786Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6490057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6490155Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6490446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6490555Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6490558Z 2025-08-14T21:49:11.6490658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6490859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6490924Z return mod(**inputs) 2025-08-14T21:49:11.6491181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6491249Z outputs = self.model( 2025-08-14T21:49:11.6491496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6491576Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6491823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6491897Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6492125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6492207Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6492479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6492579Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6492841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6492934Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6492957Z 2025-08-14T21:49:11.6493067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6493284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6493357Z return mod(**inputs) 2025-08-14T21:49:11.6493617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6493695Z outputs = self.model( 2025-08-14T21:49:11.6493954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6494027Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6494296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6494366Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6494589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6494669Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6494935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6495068Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6495315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6495473Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6495476Z 2025-08-14T21:49:11.6495578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6495778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6495850Z return mod(**inputs) 2025-08-14T21:49:11.6496117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6496187Z outputs = self.model( 2025-08-14T21:49:11.6496444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6496516Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6496769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6496838Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6497054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6497140Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6497400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6497519Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6497783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6497869Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6497873Z 2025-08-14T21:49:11.6497988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6498195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6498262Z return mod(**inputs) 2025-08-14T21:49:11.6498533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6498603Z outputs = self.model( 2025-08-14T21:49:11.6498872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6498947Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6499233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6499314Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6499545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6499626Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6499892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6500002Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6500268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6500356Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6500360Z 2025-08-14T21:49:11.6500442Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6500534Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6500616Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6500701Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6500829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6501054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6501133Z return mod(**inputs) 2025-08-14T21:49:11.6501406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6501476Z outputs = self.model( 2025-08-14T21:49:11.6501745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6501820Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6502100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6502191Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6502418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6502509Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6502769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6502880Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6503148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6503251Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6503558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6503696Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6503703Z 2025-08-14T21:49:11.6503808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6504018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6504089Z return mod(**inputs) 2025-08-14T21:49:11.6504355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6504425Z outputs = self.model( 2025-08-14T21:49:11.6504685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6504768Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6505025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6505100Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6505336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6505442Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6505712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6505824Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6506087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6506194Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6506498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6506618Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6506622Z 2025-08-14T21:49:11.6506732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6506943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6507020Z return mod(**inputs) 2025-08-14T21:49:11.6507325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6507401Z outputs = self.model( 2025-08-14T21:49:11.6507676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6507752Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6508026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6508101Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6508335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6508445Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6508848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6508979Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6509244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6509331Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6509335Z 2025-08-14T21:49:11.6509450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6509657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6509727Z return mod(**inputs) 2025-08-14T21:49:11.6510000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6510072Z outputs = self.model( 2025-08-14T21:49:11.6510347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6510427Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6510692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6510779Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6511010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6511102Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6511365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-08-14T21:49:11.6511450Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6511454Z 2025-08-14T21:49:11.6511571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6511836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6511905Z return mod(**inputs) 2025-08-14T21:49:11.6512178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6512250Z outputs = self.model( 2025-08-14T21:49:11.6512518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6512594Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6512854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6512937Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6513164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6513247Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6513520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6513707Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6513712Z 2025-08-14T21:49:11.6513828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6514037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6514109Z return mod(**inputs) 2025-08-14T21:49:11.6514399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6514471Z outputs = self.model( 2025-08-14T21:49:11.6514762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6514839Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6515150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6515236Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6515477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6515563Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6516073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6516284Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6516292Z 2025-08-14T21:49:11.6516471Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6516827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6516926Z return mod(**inputs) 2025-08-14T21:49:11.6517422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6517526Z outputs = self.model( 2025-08-14T21:49:11.6518020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6518126Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6518446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6518533Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6518777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6518862Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6519141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6519298Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6519302Z 2025-08-14T21:49:11.6519419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6519632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6519705Z return mod(**inputs) 2025-08-14T21:49:11.6519981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6520053Z outputs = self.model( 2025-08-14T21:49:11.6520339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6520415Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6520695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6520777Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6521014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6521099Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6521440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6521552Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6521837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6522001Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6522004Z 2025-08-14T21:49:11.6522114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6522338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6522407Z return mod(**inputs) 2025-08-14T21:49:11.6522713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6522788Z outputs = self.model( 2025-08-14T21:49:11.6523080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6523162Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6523448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6523523Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6523780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6523863Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6524153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6524263Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6524538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6524637Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6524640Z 2025-08-14T21:49:11.6524752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6524972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6525043Z return mod(**inputs) 2025-08-14T21:49:11.6525313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6525392Z outputs = self.model( 2025-08-14T21:49:11.6525667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6525753Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6526050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6526125Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6526362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6526444Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6526711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6526815Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6527062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6527146Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6527155Z 2025-08-14T21:49:11.6527235Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6527316Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6527400Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6527474Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6527633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6527841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6527908Z return mod(**inputs) 2025-08-14T21:49:11.6528172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6528260Z outputs = self.model( 2025-08-14T21:49:11.6528526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6528604Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6528868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6528941Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6529171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6529249Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6529500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6529595Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6529843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6529947Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6530237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6530372Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6530382Z 2025-08-14T21:49:11.6530482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6530684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6530755Z return mod(**inputs) 2025-08-14T21:49:11.6531008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6531074Z outputs = self.model( 2025-08-14T21:49:11.6531335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6531406Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6531673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6531742Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6531978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6532064Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6532309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6532402Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6532649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6532743Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6533031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6533133Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6533137Z 2025-08-14T21:49:11.6533238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6533439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6533533Z return mod(**inputs) 2025-08-14T21:49:11.6533809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6533877Z outputs = self.model( 2025-08-14T21:49:11.6534123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6534200Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6534444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6534512Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6534732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6534825Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6535077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6535173Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6535426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6535518Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6535521Z 2025-08-14T21:49:11.6535627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6535838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6535904Z return mod(**inputs) 2025-08-14T21:49:11.6536162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6536242Z outputs = self.model( 2025-08-14T21:49:11.6536503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6536579Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6536846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6536920Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6537155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6537246Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6537491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6537605Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6537862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6538035Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6538041Z 2025-08-14T21:49:11.6538141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6538331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6538401Z return mod(**inputs) 2025-08-14T21:49:11.6538652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6538721Z outputs = self.model( 2025-08-14T21:49:11.6538972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6539044Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6539297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6539371Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6539584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6539701Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6539942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6540051Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6540287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6540364Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6540367Z 2025-08-14T21:49:11.6540472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6540679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6540746Z return mod(**inputs) 2025-08-14T21:49:11.6541002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6541074Z outputs = self.model( 2025-08-14T21:49:11.6541326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6541397Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6541643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6541721Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6541937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6542014Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6542270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6542375Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6542639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6542724Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6542727Z 2025-08-14T21:49:11.6542805Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6542891Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6542965Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6543045Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6543145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6543340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6543411Z return mod(**inputs) 2025-08-14T21:49:11.6543689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6543754Z outputs = self.model( 2025-08-14T21:49:11.6544013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6544083Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6544335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6544404Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6544619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6544705Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6544951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6545059Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6545315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6545470Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6545769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6545902Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6545906Z 2025-08-14T21:49:11.6546006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6546208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6546274Z return mod(**inputs) 2025-08-14T21:49:11.6546530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6546615Z outputs = self.model( 2025-08-14T21:49:11.6546863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6546946Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6547193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6547262Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6547486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6547562Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6547816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6547918Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6548167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6548273Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6548563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6548676Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6548679Z 2025-08-14T21:49:11.6548780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6548975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6549048Z return mod(**inputs) 2025-08-14T21:49:11.6549294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6549360Z outputs = self.model( 2025-08-14T21:49:11.6549614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6549704Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6549963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6550033Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6550249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6550334Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6550583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6550695Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6550943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6551027Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6551031Z 2025-08-14T21:49:11.6551138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6551366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6551433Z return mod(**inputs) 2025-08-14T21:49:11.6551691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6551757Z outputs = self.model( 2025-08-14T21:49:11.6552010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6552081Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6552326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6552404Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6552639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6552723Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6552974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6553097Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6553101Z 2025-08-14T21:49:11.6553215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6553421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6553489Z return mod(**inputs) 2025-08-14T21:49:11.6553760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6553829Z outputs = self.model( 2025-08-14T21:49:11.6554101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6554178Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6554442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6554524Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6554757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6554839Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6555107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6555230Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6555234Z 2025-08-14T21:49:11.6555350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6555584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6555768Z return mod(**inputs) 2025-08-14T21:49:11.6556062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6556135Z outputs = self.model( 2025-08-14T21:49:11.6556414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6556492Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6556764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6556850Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6557079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6557157Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6557418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6557499Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6557530Z 2025-08-14T21:49:11.6557661Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6557858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6557923Z return mod(**inputs) 2025-08-14T21:49:11.6558187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6558252Z outputs = self.model( 2025-08-14T21:49:11.6558504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6558576Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6558840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6558922Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6559140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6559217Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6559470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-08-14T21:49:11.6559549Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6559552Z 2025-08-14T21:49:11.6559659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6559857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6559921Z return mod(**inputs) 2025-08-14T21:49:11.6560176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6560244Z outputs = self.model( 2025-08-14T21:49:11.6560497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6560575Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6560824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6560900Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6561125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6561199Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6561446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6561543Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6561809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6561956Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6561962Z 2025-08-14T21:49:11.6562062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6562256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6562320Z return mod(**inputs) 2025-08-14T21:49:11.6562566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6562631Z outputs = self.model( 2025-08-14T21:49:11.6562869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6562945Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6563195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6563264Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6563522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6563601Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6563854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6563952Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6564196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6564280Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6564283Z 2025-08-14T21:49:11.6564384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6564601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6564667Z return mod(**inputs) 2025-08-14T21:49:11.6564924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6564995Z outputs = self.model( 2025-08-14T21:49:11.6565238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6565307Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6565558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6565624Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6565842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6565917Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6566159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6566260Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6566500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6566582Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6566593Z 2025-08-14T21:49:11.6566671Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6566746Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6566827Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6566898Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6566996Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6567191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6567273Z return mod(**inputs) 2025-08-14T21:49:11.6567522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6567594Z outputs = self.model( 2025-08-14T21:49:11.6567843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6567922Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6568173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6568243Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6568471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6568550Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6568809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6568910Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6569180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6569300Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6569588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6569721Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6569732Z 2025-08-14T21:49:11.6569835Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6570030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6570103Z return mod(**inputs) 2025-08-14T21:49:11.6570366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6570438Z outputs = self.model( 2025-08-14T21:49:11.6570692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6570768Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6571028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6571096Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6571305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6571388Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6571626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6571718Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6571969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6572060Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6572348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6572453Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6572457Z 2025-08-14T21:49:11.6572554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6572749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6572811Z return mod(**inputs) 2025-08-14T21:49:11.6573060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6573124Z outputs = self.model( 2025-08-14T21:49:11.6573384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6573460Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6573701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6573769Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6573984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6574059Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6574304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6574398Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6574635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6574723Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6574727Z 2025-08-14T21:49:11.6574825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6575057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6575124Z return mod(**inputs) 2025-08-14T21:49:11.6575364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6575438Z outputs = self.model( 2025-08-14T21:49:11.6575680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6575748Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6575997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6576067Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6576308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6576387Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6576625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6576735Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6576976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6577119Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6577131Z 2025-08-14T21:49:11.6577228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6577417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6577491Z return mod(**inputs) 2025-08-14T21:49:11.6577738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6577807Z outputs = self.model( 2025-08-14T21:49:11.6578061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6578131Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6578384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6578454Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6578669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6578753Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6579001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6579126Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6579378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6579456Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6579460Z 2025-08-14T21:49:11.6579566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6579760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6579824Z return mod(**inputs) 2025-08-14T21:49:11.6580078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6580144Z outputs = self.model( 2025-08-14T21:49:11.6580395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6580470Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6580714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6580824Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6581044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6581122Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6581376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6581480Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6581733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6581818Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6581823Z 2025-08-14T21:49:11.6581921Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6582007Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6582082Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6582159Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6582268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6582463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6582536Z return mod(**inputs) 2025-08-14T21:49:11.6582786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6582853Z outputs = self.model( 2025-08-14T21:49:11.6583111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6583183Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6583438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6583509Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6583726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6583811Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6584057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6584162Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6584417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6584514Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6584807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6584959Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6584963Z 2025-08-14T21:49:11.6585064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6585270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6585335Z return mod(**inputs) 2025-08-14T21:49:11.6585592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6585659Z outputs = self.model( 2025-08-14T21:49:11.6585906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6585985Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6586231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6586303Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6586541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6586651Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6586906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6587010Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6587257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6587361Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6587644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6587756Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6587762Z 2025-08-14T21:49:11.6587879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6588077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6588152Z return mod(**inputs) 2025-08-14T21:49:11.6588400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6588466Z outputs = self.model( 2025-08-14T21:49:11.6588721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6588792Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6589046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6589117Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6589335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6589422Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6589672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6589777Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6590031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6590111Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6590114Z 2025-08-14T21:49:11.6590223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6590418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6590483Z return mod(**inputs) 2025-08-14T21:49:11.6590739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6590824Z outputs = self.model( 2025-08-14T21:49:11.6591080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6591155Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6591408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6591484Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6591705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6591783Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6592041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6592163Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6592170Z 2025-08-14T21:49:11.6592286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6592495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6592600Z return mod(**inputs) 2025-08-14T21:49:11.6592876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6592945Z outputs = self.model( 2025-08-14T21:49:11.6593211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6593286Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6593547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6593627Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6593872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6593957Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6594232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6594355Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6594359Z 2025-08-14T21:49:11.6594475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6594687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6594758Z return mod(**inputs) 2025-08-14T21:49:11.6595033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6595106Z outputs = self.model( 2025-08-14T21:49:11.6595381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6595462Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6595810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6595902Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6596141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6596225Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6596505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6596602Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6596606Z 2025-08-14T21:49:11.6596723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6596928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6597027Z return mod(**inputs) 2025-08-14T21:49:11.6597303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6597372Z outputs = self.model( 2025-08-14T21:49:11.6597628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6597701Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6597957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6598034Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6598244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6598320Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6598572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6598670Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6598962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6599112Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6599115Z 2025-08-14T21:49:11.6599216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6599413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6599477Z return mod(**inputs) 2025-08-14T21:49:11.6599727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6599791Z outputs = self.model( 2025-08-14T21:49:11.6600048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6600129Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6600379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6600450Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6600673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6600749Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6601000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6601095Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6601340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6601426Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6601431Z 2025-08-14T21:49:11.6601530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6601727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6601793Z return mod(**inputs) 2025-08-14T21:49:11.6602039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6602111Z outputs = self.model( 2025-08-14T21:49:11.6602356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6602426Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6602674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6602742Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6602983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6603060Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6603309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6603415Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6603670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6603753Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6603763Z 2025-08-14T21:49:11.6603839Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6603916Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6603998Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6604072Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6604175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6604384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6604448Z return mod(**inputs) 2025-08-14T21:49:11.6604732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6604808Z outputs = self.model( 2025-08-14T21:49:11.6605058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6605136Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6605394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6605462Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6605683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6605775Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6606024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6606120Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6606357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6606455Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6606733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6606863Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6606873Z 2025-08-14T21:49:11.6606970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6607163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6607237Z return mod(**inputs) 2025-08-14T21:49:11.6607485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6607554Z outputs = self.model( 2025-08-14T21:49:11.6607810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6607881Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6608132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6608201Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6608414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6608500Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6608870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6609029Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6609292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6609389Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6609692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6609797Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6609801Z 2025-08-14T21:49:11.6609899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6610094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6610157Z return mod(**inputs) 2025-08-14T21:49:11.6610413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6610483Z outputs = self.model( 2025-08-14T21:49:11.6610791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6610874Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6611127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6611197Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6611423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6611502Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6611757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6611881Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6612132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6612225Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6612229Z 2025-08-14T21:49:11.6612332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6612534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6612599Z return mod(**inputs) 2025-08-14T21:49:11.6612846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6612920Z outputs = self.model( 2025-08-14T21:49:11.6613169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6613240Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6613499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6613569Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6613796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6613875Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6614126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-08-14T21:49:11.6614215Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6614218Z 2025-08-14T21:49:11.6614319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6614516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6614588Z return mod(**inputs) 2025-08-14T21:49:11.6614837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6614943Z outputs = self.model( 2025-08-14T21:49:11.6615192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6615266Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6615519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6615589Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6615810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6615888Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6616132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6616247Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6616496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6616679Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6616691Z 2025-08-14T21:49:11.6616794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6616989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6617062Z return mod(**inputs) 2025-08-14T21:49:11.6617310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6617377Z outputs = self.model( 2025-08-14T21:49:11.6617632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6617702Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6617971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6618045Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6618264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6618349Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6618598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6618705Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6618966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6619044Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6619047Z 2025-08-14T21:49:11.6619155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6619355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6619421Z return mod(**inputs) 2025-08-14T21:49:11.6619682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6619747Z outputs = self.model( 2025-08-14T21:49:11.6620003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6620074Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6620326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6620401Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6620621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6620717Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6620981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6621089Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6621343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6621429Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6621432Z 2025-08-14T21:49:11.6621512Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6621599Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6621675Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6621749Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6621857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6622051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6622125Z return mod(**inputs) 2025-08-14T21:49:11.6622376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6622492Z outputs = self.model( 2025-08-14T21:49:11.6622754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6622825Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6623078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6623154Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6623371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6623455Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6623721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6623830Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6624087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6624186Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6624482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6624615Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6624619Z 2025-08-14T21:49:11.6624719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6624921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6624986Z return mod(**inputs) 2025-08-14T21:49:11.6625248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6625321Z outputs = self.model( 2025-08-14T21:49:11.6625570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6625649Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6625891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6625959Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6626177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6626253Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6626501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6626627Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6626866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6626970Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6627257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6627363Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6627374Z 2025-08-14T21:49:11.6627474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6627669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6627743Z return mod(**inputs) 2025-08-14T21:49:11.6627991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6628061Z outputs = self.model( 2025-08-14T21:49:11.6628321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6628434Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6628716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6628792Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6629030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6629120Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6629398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6629507Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6629806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6629895Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6629900Z 2025-08-14T21:49:11.6630015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6630222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6630290Z return mod(**inputs) 2025-08-14T21:49:11.6630571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6630640Z outputs = self.model( 2025-08-14T21:49:11.6630916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6630992Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6631268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6631351Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6631589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6631674Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6631953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6632080Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6632084Z 2025-08-14T21:49:11.6632195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6632402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6632468Z return mod(**inputs) 2025-08-14T21:49:11.6632748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6632840Z outputs = self.model( 2025-08-14T21:49:11.6633122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6633201Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6633482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6633566Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6633813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6633897Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6634189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6634319Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6634323Z 2025-08-14T21:49:11.6634446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6634660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6634750Z return mod(**inputs) 2025-08-14T21:49:11.6635050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6635125Z outputs = self.model( 2025-08-14T21:49:11.6635413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6635491Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6635847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6635939Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6636188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6636301Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6636595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6636698Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6636703Z 2025-08-14T21:49:11.6636819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6637028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6637097Z return mod(**inputs) 2025-08-14T21:49:11.6637375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6637445Z outputs = self.model( 2025-08-14T21:49:11.6637716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6637803Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6638076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6638158Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6638398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6638481Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6638750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6638853Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6639131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6639288Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6639340Z 2025-08-14T21:49:11.6639452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6639665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6639736Z return mod(**inputs) 2025-08-14T21:49:11.6640011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6640084Z outputs = self.model( 2025-08-14T21:49:11.6640334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6640410Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6640658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6640727Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6640949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6641026Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6641297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6641412Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6641661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6641753Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6641757Z 2025-08-14T21:49:11.6641864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6642070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6642144Z return mod(**inputs) 2025-08-14T21:49:11.6642420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6642514Z outputs = self.model( 2025-08-14T21:49:11.6642779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6642859Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6643132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6643207Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6643447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6643529Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6643795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6643900Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6644154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6644240Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6644244Z 2025-08-14T21:49:11.6644337Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6644417Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6644502Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6644577Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6644680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6644886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6644951Z return mod(**inputs) 2025-08-14T21:49:11.6645199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6645273Z outputs = self.model( 2025-08-14T21:49:11.6645550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6645630Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6645878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6645949Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6646172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6646256Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6646505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6646609Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6646859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6646965Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6647251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6647439Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6647443Z 2025-08-14T21:49:11.6647554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6647751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6647822Z return mod(**inputs) 2025-08-14T21:49:11.6648074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6648140Z outputs = self.model( 2025-08-14T21:49:11.6648398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6648489Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6648743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6648822Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6649040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6649123Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6649369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6649467Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6649720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6649817Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6650112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6650222Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6650227Z 2025-08-14T21:49:11.6650331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6650545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6650611Z return mod(**inputs) 2025-08-14T21:49:11.6650875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6650953Z outputs = self.model( 2025-08-14T21:49:11.6651214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6651297Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6651559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6651650Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6651891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6651974Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6652241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6652349Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6652597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6652686Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6652690Z 2025-08-14T21:49:11.6652789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6652997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6653074Z return mod(**inputs) 2025-08-14T21:49:11.6653357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6653451Z outputs = self.model( 2025-08-14T21:49:11.6653713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6653786Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6654053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6654126Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6654362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6654445Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6654726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6654858Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6655108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6655257Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6655267Z 2025-08-14T21:49:11.6655368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6655564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6655635Z return mod(**inputs) 2025-08-14T21:49:11.6655883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6655948Z outputs = self.model( 2025-08-14T21:49:11.6656204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6656275Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6656530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6656601Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6656816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6656902Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6657149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6657254Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6657508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6657609Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6657612Z 2025-08-14T21:49:11.6657722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6657920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6657984Z return mod(**inputs) 2025-08-14T21:49:11.6658240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6658305Z outputs = self.model( 2025-08-14T21:49:11.6658558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6658630Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6658874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6658953Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6659168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6659263Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6659531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6659639Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6659891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6659976Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6659979Z 2025-08-14T21:49:11.6660056Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6660144Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6660220Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6660295Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6660420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6660615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6660687Z return mod(**inputs) 2025-08-14T21:49:11.6660937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6661003Z outputs = self.model( 2025-08-14T21:49:11.6661255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6661327Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6661574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6661651Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6661868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6661965Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6662204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6662307Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6662553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6662649Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6662932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6663062Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6663065Z 2025-08-14T21:49:11.6663164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6663384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6663448Z return mod(**inputs) 2025-08-14T21:49:11.6663689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6663764Z outputs = self.model( 2025-08-14T21:49:11.6664005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6664084Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6664322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6664390Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6664608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6664684Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6664933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6665051Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6665305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6665408Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6665682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6665785Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6665795Z 2025-08-14T21:49:11.6665892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6666078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6666149Z return mod(**inputs) 2025-08-14T21:49:11.6666410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6666480Z outputs = self.model( 2025-08-14T21:49:11.6666733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6666804Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6667060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6667130Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6667352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6667437Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6667688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6667795Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6668057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6668139Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6668143Z 2025-08-14T21:49:11.6668250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6668449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6668512Z return mod(**inputs) 2025-08-14T21:49:11.6668781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6668846Z outputs = self.model( 2025-08-14T21:49:11.6669097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6669188Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6669426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6669502Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6669710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6669784Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6670032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-08-14T21:49:11.6670107Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6670110Z 2025-08-14T21:49:11.6670213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6670402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6670465Z return mod(**inputs) 2025-08-14T21:49:11.6670713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6670796Z outputs = self.model( 2025-08-14T21:49:11.6671060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6671134Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6671384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6671462Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6671683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6671761Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6672018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6672157Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6672162Z 2025-08-14T21:49:11.6672272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6672471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6672537Z return mod(**inputs) 2025-08-14T21:49:11.6672793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6672858Z outputs = self.model( 2025-08-14T21:49:11.6673105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6673183Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6673430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6673514Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6673744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6673828Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6674100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6674225Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6674229Z 2025-08-14T21:49:11.6674345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6674554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6674622Z return mod(**inputs) 2025-08-14T21:49:11.6674897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6674966Z outputs = self.model( 2025-08-14T21:49:11.6675249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6675335Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6675596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6675761Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6676001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6676083Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6676353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6676439Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6676443Z 2025-08-14T21:49:11.6676561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6676776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6676847Z return mod(**inputs) 2025-08-14T21:49:11.6677158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6677237Z outputs = self.model( 2025-08-14T21:49:11.6677477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6677557Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6677802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6677877Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6678086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6678161Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6678438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6678547Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6678818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6678978Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6678982Z 2025-08-14T21:49:11.6679090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6679303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6679370Z return mod(**inputs) 2025-08-14T21:49:11.6679636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6679713Z outputs = self.model( 2025-08-14T21:49:11.6679976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6680060Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6680334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6680409Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6680645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6680727Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6681049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6681153Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6681425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6681536Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6681540Z 2025-08-14T21:49:11.6681646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6681853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6681929Z return mod(**inputs) 2025-08-14T21:49:11.6682238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6682312Z outputs = self.model( 2025-08-14T21:49:11.6682582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6682657Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6682922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6682999Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6683226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6683347Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6683623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6683731Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6684004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6684093Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6684097Z 2025-08-14T21:49:11.6684188Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6684272Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6684359Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6684458Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6684565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6684780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6684852Z return mod(**inputs) 2025-08-14T21:49:11.6685116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6685193Z outputs = self.model( 2025-08-14T21:49:11.6685461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6685543Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6685816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6685889Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6686128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6686213Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6686502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6686615Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6686885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6686997Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6687310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6687451Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6687454Z 2025-08-14T21:49:11.6687568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6687797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6687882Z return mod(**inputs) 2025-08-14T21:49:11.6688135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6688202Z outputs = self.model( 2025-08-14T21:49:11.6688461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6688545Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6688785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6688862Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6689074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6689162Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6689403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6689538Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6689788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6689880Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6690165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6690271Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6690275Z 2025-08-14T21:49:11.6690374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6690572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6690653Z return mod(**inputs) 2025-08-14T21:49:11.6690894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6690968Z outputs = self.model( 2025-08-14T21:49:11.6691209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6691286Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6691526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6691594Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6691810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6691883Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6692131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:49:11.6692226Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:11.6692471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6692558Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6692561Z 2025-08-14T21:49:11.6692658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6692848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6692918Z return mod(**inputs) 2025-08-14T21:49:11.6693157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6693229Z outputs = self.model( 2025-08-14T21:49:11.6693469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6693564Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6693813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6693883Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6694095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6694178Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6694416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6694526Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6694769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:49:11.6694914Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:11.6694920Z 2025-08-14T21:49:11.6695028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6695219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6695320Z return mod(**inputs) 2025-08-14T21:49:11.6695566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6695630Z outputs = self.model( 2025-08-14T21:49:11.6695878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6695946Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6696187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6696263Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6696496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6696584Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6696832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6696937Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6697192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:49:11.6697271Z key_states = self.k_proj(current_states) 2025-08-14T21:49:11.6697274Z 2025-08-14T21:49:11.6697380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6697575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6697638Z return mod(**inputs) 2025-08-14T21:49:11.6697896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6697965Z outputs = self.model( 2025-08-14T21:49:11.6698222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6698301Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6698552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6698629Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6698837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6698912Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6699160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6699262Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6699529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:49:11.6699613Z value_states = self.v_proj(current_states) 2025-08-14T21:49:11.6699618Z 2025-08-14T21:49:11.6699695Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6699779Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6699853Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6699924Z cudagraph partition due to non gpu ops 2025-08-14T21:49:11.6700029Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6700218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6700288Z return mod(**inputs) 2025-08-14T21:49:11.6700530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6700596Z outputs = self.model( 2025-08-14T21:49:11.6700845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6700931Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6701185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6701266Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6701475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6701556Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6701796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6701897Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6702158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6702255Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6702542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:11.6702669Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:11.6702673Z 2025-08-14T21:49:11.6702771Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6702968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6703033Z return mod(**inputs) 2025-08-14T21:49:11.6703278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6703353Z outputs = self.model( 2025-08-14T21:49:11.6703605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6703689Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6703949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6704022Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6704241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6704321Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6704571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6704676Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6704919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:49:11.6705045Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:11.6705331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:11.6705445Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:11.6705455Z 2025-08-14T21:49:11.6705557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6705754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6705825Z return mod(**inputs) 2025-08-14T21:49:11.6706072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6706139Z outputs = self.model( 2025-08-14T21:49:11.6706393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6706464Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6706720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6706809Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6707045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6707133Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6707386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:49:11.6707492Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:11.6707751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:49:11.6707832Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:11.6707835Z 2025-08-14T21:49:11.6707946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6708158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6708223Z return mod(**inputs) 2025-08-14T21:49:11.6708485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6708551Z outputs = self.model( 2025-08-14T21:49:11.6708982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6709061Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6709308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6709388Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6709609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6709693Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6709953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6710077Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6710081Z 2025-08-14T21:49:11.6710191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6710386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6710453Z return mod(**inputs) 2025-08-14T21:49:11.6710709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6710776Z outputs = self.model( 2025-08-14T21:49:11.6711029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6711110Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6711408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6711485Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6711703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6711780Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6712038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:49:11.6712153Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:11.6712156Z 2025-08-14T21:49:11.6712262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6712469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6712538Z return mod(**inputs) 2025-08-14T21:49:11.6712812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6712883Z outputs = self.model( 2025-08-14T21:49:11.6713214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6713300Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6713558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6713638Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6713864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6713946Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6714214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:49:11.6714326Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:11.6714331Z 2025-08-14T21:49:11.6714445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6714658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6714725Z return mod(**inputs) 2025-08-14T21:49:11.6714993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:49:11.6715062Z outputs = self.model( 2025-08-14T21:49:11.6715323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:49:11.6715408Z decoder_outputs = self.decoder( 2025-08-14T21:49:11.6715743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:49:11.6715833Z layer_outputs = decoder_layer( 2025-08-14T21:49:11.6716083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:11.6716165Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:11.6716437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-08-14T21:49:11.6716521Z hidden_states = residual + hidden_states 2025-08-14T21:49:11.6716525Z 2025-08-14T21:49:11.6716639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6716844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6716911Z return mod(**inputs) 2025-08-14T21:49:11.6717180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1422, in forward 2025-08-14T21:49:11.6717264Z lm_logits = self.lm_head(outputs[0]) 2025-08-14T21:49:11.6717268Z 2025-08-14T21:49:11.6717401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:11.6717618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:11.6717684Z return mod(**inputs) 2025-08-14T21:49:11.6717941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1429, in forward 2025-08-14T21:49:11.6718111Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:49:11.6718115Z 2025-08-14T21:49:23.7699123Z Compilation time (from dynamo_timed): 28.10535142 2025-08-14T21:49:23.7792319Z pass 2025-08-14T21:49:23.7792727Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:23.7793599Z TIMING: _recursive_pre_grad_passes:0.01506 _recursive_joint_graph_passes:1.16729 _recursive_post_grad_passes:0.16604 async_compile.wait:0.78512 code_gen:11.64543 inductor_compile:14.62509 backend_compile:22.11098 gc:0.00028 entire_frame_compile:28.10535 total_wall_time:28.10535 2025-08-14T21:49:23.7794592Z STATS: call_* op count: 1014 | FakeTensorMode.__torch_dispatch__:33764 | FakeTensor.__torch_dispatch__:11261 | ProxyTorchDispatchMode.__torch_dispatch__:12417 2025-08-14T21:49:23.7795453Z Dynamo produced 1 graphs covering 1014 ops with 0 graph breaks (0 unique) 2025-08-14T21:49:29.6570098Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:49:29.6572257Z from pkg_resources import resource_filename 2025-08-14T21:49:30.2609356Z 2025-08-14T21:49:33.1725018Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:49:33.1725840Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:49:33.1747748Z cpu eval MBartForCausalLM 2025-08-14T21:49:34.8685670Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:35.5051040Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:36.1404555Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:43.9506150Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9510155Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9510410Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9510644Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9510877Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9511105Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9511339Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9511710Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9512040Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9512436Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9512712Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9512935Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9513340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9513813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9514200Z return mod(**inputs) 2025-08-14T21:49:43.9514669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9515136Z outputs = self.model.decoder( 2025-08-14T21:49:43.9515575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9516172Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9516570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9517285Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9517714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9518176Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9518627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9519137Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9519371Z 2025-08-14T21:49:43.9519489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9519887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9520237Z return mod(**inputs) 2025-08-14T21:49:43.9520635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9521063Z outputs = self.model.decoder( 2025-08-14T21:49:43.9521478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9521991Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9522374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9522768Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9523187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9523625Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9524067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9524494Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9524638Z 2025-08-14T21:49:43.9524802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9525181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9525528Z return mod(**inputs) 2025-08-14T21:49:43.9525916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9526334Z outputs = self.model.decoder( 2025-08-14T21:49:43.9526752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9527175Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9527550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9527933Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9528352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9528806Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9529240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9529680Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9529838Z 2025-08-14T21:49:43.9529926Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9530155Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9530365Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9530584Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9530830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9531202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9531548Z return mod(**inputs) 2025-08-14T21:49:43.9531944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9532436Z outputs = self.model.decoder( 2025-08-14T21:49:43.9532844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9533256Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9533621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9533990Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9534400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9534830Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9535258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9535690Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9536168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9536709Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9536926Z 2025-08-14T21:49:43.9537047Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9537419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9537763Z return mod(**inputs) 2025-08-14T21:49:43.9538146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9538573Z outputs = self.model.decoder( 2025-08-14T21:49:43.9538976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9539459Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9539856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9540233Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9540654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9541096Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9541542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9541999Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9542473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9542963Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9543137Z 2025-08-14T21:49:43.9543252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9543646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9543996Z return mod(**inputs) 2025-08-14T21:49:43.9544391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9544800Z outputs = self.model.decoder( 2025-08-14T21:49:43.9545208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9545620Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9545994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9546374Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9546791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9547316Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9547748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9548177Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9548335Z 2025-08-14T21:49:43.9548454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9548847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9549191Z return mod(**inputs) 2025-08-14T21:49:43.9563715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9564173Z outputs = self.model.decoder( 2025-08-14T21:49:43.9564609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9565034Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9565434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9565808Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9566350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9566792Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9566973Z 2025-08-14T21:49:43.9567089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9567715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9568053Z return mod(**inputs) 2025-08-14T21:49:43.9568428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9568816Z outputs = self.model.decoder( 2025-08-14T21:49:43.9569236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9569620Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9569962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9570320Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9570702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9571123Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9571494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9571829Z return self.act(input) 2025-08-14T21:49:43.9571942Z 2025-08-14T21:49:43.9572058Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9572430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9572756Z return mod(**inputs) 2025-08-14T21:49:43.9573115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9573499Z outputs = self.model.decoder( 2025-08-14T21:49:43.9573862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9574239Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9574580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9574933Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9575305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9575686Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9575821Z 2025-08-14T21:49:43.9575965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9576324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9576655Z return mod(**inputs) 2025-08-14T21:49:43.9577025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9577422Z outputs = self.model.decoder( 2025-08-14T21:49:43.9577794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9578172Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9578516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9578868Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9579242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9579652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9580057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9580543Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9580751Z 2025-08-14T21:49:43.9580855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9581205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9581526Z return mod(**inputs) 2025-08-14T21:49:43.9581861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9582238Z outputs = self.model.decoder( 2025-08-14T21:49:43.9582607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9582999Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9583330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9583688Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9584065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9584459Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9584856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9585237Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9585370Z 2025-08-14T21:49:43.9585482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9585834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9586145Z return mod(**inputs) 2025-08-14T21:49:43.9586490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9586853Z outputs = self.model.decoder( 2025-08-14T21:49:43.9587216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9587591Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9587929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9588272Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9588645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9589044Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9589435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9589837Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9589980Z 2025-08-14T21:49:43.9590062Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9590278Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9590478Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9590684Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9590916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9591271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9591601Z return mod(**inputs) 2025-08-14T21:49:43.9591970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9592371Z outputs = self.model.decoder( 2025-08-14T21:49:43.9592784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9593210Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9593591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9594016Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9594419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9594854Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9595279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9595809Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9596309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9596886Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9597103Z 2025-08-14T21:49:43.9597218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9597581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9597920Z return mod(**inputs) 2025-08-14T21:49:43.9598289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9598687Z outputs = self.model.decoder( 2025-08-14T21:49:43.9599059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9599444Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9599792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9600144Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9600537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9600950Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9601362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9601770Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9602219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9602680Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9602843Z 2025-08-14T21:49:43.9602958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9603315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9603666Z return mod(**inputs) 2025-08-14T21:49:43.9604070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9604476Z outputs = self.model.decoder( 2025-08-14T21:49:43.9604892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9605311Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9605687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9606064Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9606449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9606857Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9607250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9607659Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9607794Z 2025-08-14T21:49:43.9607898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9608288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9608622Z return mod(**inputs) 2025-08-14T21:49:43.9609276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9609669Z outputs = self.model.decoder( 2025-08-14T21:49:43.9610101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9610489Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9610833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9611195Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9611659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9612094Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9612270Z 2025-08-14T21:49:43.9612374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9612733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9613059Z return mod(**inputs) 2025-08-14T21:49:43.9613414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9613806Z outputs = self.model.decoder( 2025-08-14T21:49:43.9614186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9614573Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9614916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9615282Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9615677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9616093Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9616465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9616800Z return self.act(input) 2025-08-14T21:49:43.9616907Z 2025-08-14T21:49:43.9617015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9617356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9617669Z return mod(**inputs) 2025-08-14T21:49:43.9618023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9618429Z outputs = self.model.decoder( 2025-08-14T21:49:43.9618788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9619160Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9619492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9619835Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9620211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9620593Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9620725Z 2025-08-14T21:49:43.9620832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9621174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9621491Z return mod(**inputs) 2025-08-14T21:49:43.9621838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9622237Z outputs = self.model.decoder( 2025-08-14T21:49:43.9622661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9623036Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9623375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9623728Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9624111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:49:43.9624504Z hidden_states = residual + hidden_states 2025-08-14T21:49:43.9624637Z 2025-08-14T21:49:43.9624753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9625138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9625481Z return mod(**inputs) 2025-08-14T21:49:43.9625844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9626223Z outputs = self.model.decoder( 2025-08-14T21:49:43.9626613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9626984Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9627316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9627655Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9628026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9628424Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9628810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9629259Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9629462Z 2025-08-14T21:49:43.9629562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9629911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9630211Z return mod(**inputs) 2025-08-14T21:49:43.9630559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9630931Z outputs = self.model.decoder( 2025-08-14T21:49:43.9631295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9631681Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9632027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9632393Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9632781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9633196Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9633614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9634038Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9634184Z 2025-08-14T21:49:43.9634297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9634683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9635032Z return mod(**inputs) 2025-08-14T21:49:43.9635417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9635918Z outputs = self.model.decoder( 2025-08-14T21:49:43.9636351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9636769Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9637139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9637531Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9637920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9638326Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9638721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9639185Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9639329Z 2025-08-14T21:49:43.9639419Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9639632Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9639844Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9640050Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9640279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9640641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9640964Z return mod(**inputs) 2025-08-14T21:49:43.9641336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9641762Z outputs = self.model.decoder( 2025-08-14T21:49:43.9642139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9642534Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9642898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9643275Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9643659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9644068Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9644469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9644912Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9645380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9645889Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9646107Z 2025-08-14T21:49:43.9646216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9646592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9646923Z return mod(**inputs) 2025-08-14T21:49:43.9647288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9647677Z outputs = self.model.decoder( 2025-08-14T21:49:43.9648057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9648447Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9648791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9649156Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9649547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9649962Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9650401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9650810Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9651249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9651703Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9651871Z 2025-08-14T21:49:43.9651975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9652336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9652665Z return mod(**inputs) 2025-08-14T21:49:43.9653037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9653434Z outputs = self.model.decoder( 2025-08-14T21:49:43.9653824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9654213Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9654557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9654924Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9655317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9655727Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9656134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9656535Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9656673Z 2025-08-14T21:49:43.9656787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9657144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9657467Z return mod(**inputs) 2025-08-14T21:49:43.9657836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9658217Z outputs = self.model.decoder( 2025-08-14T21:49:43.9658582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9658954Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9659292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9659637Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9660034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9660452Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9660622Z 2025-08-14T21:49:43.9660731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9661072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9661386Z return mod(**inputs) 2025-08-14T21:49:43.9661739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9662112Z outputs = self.model.decoder( 2025-08-14T21:49:43.9662482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9662858Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9663206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9663561Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9663962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9664411Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9664799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9665131Z return self.act(input) 2025-08-14T21:49:43.9665249Z 2025-08-14T21:49:43.9665353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9665709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9666021Z return mod(**inputs) 2025-08-14T21:49:43.9666379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9666780Z outputs = self.model.decoder( 2025-08-14T21:49:43.9667157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9667540Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9667879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9668229Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9668602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9668993Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9669138Z 2025-08-14T21:49:43.9669242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9669598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9669916Z return mod(**inputs) 2025-08-14T21:49:43.9670277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9670669Z outputs = self.model.decoder( 2025-08-14T21:49:43.9671044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9671429Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9671803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9672190Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9672586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9673029Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9673470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9674001Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9674224Z 2025-08-14T21:49:43.9674339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9674728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9675079Z return mod(**inputs) 2025-08-14T21:49:43.9675476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9675993Z outputs = self.model.decoder( 2025-08-14T21:49:43.9676429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9676859Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9677236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9677605Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9677998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9678452Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9678862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9679259Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9679395Z 2025-08-14T21:49:43.9679507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9679866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9680183Z return mod(**inputs) 2025-08-14T21:49:43.9680540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9680927Z outputs = self.model.decoder( 2025-08-14T21:49:43.9681309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9681697Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9682046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9682409Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9682791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9683201Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9683609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9684012Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9684158Z 2025-08-14T21:49:43.9684243Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9684472Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9684689Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9684895Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9685143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9685512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9685836Z return mod(**inputs) 2025-08-14T21:49:43.9686204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9686601Z outputs = self.model.decoder( 2025-08-14T21:49:43.9686984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9687365Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9687725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9688092Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9688467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9688872Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9689267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9689674Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9690106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9690584Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9690765Z 2025-08-14T21:49:43.9690879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9691237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9691561Z return mod(**inputs) 2025-08-14T21:49:43.9691950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9692362Z outputs = self.model.decoder( 2025-08-14T21:49:43.9692734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9693162Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9693536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9693926Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9694329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9694764Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9695213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9695647Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9696123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9696614Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9696786Z 2025-08-14T21:49:43.9696905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9697279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9697620Z return mod(**inputs) 2025-08-14T21:49:43.9698011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9698428Z outputs = self.model.decoder( 2025-08-14T21:49:43.9698833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9699235Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9699602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9699971Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9700379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9700816Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9701240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9701645Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9701796Z 2025-08-14T21:49:43.9701904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9702314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9702677Z return mod(**inputs) 2025-08-14T21:49:43.9703072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9703504Z outputs = self.model.decoder( 2025-08-14T21:49:43.9703919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9704344Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9704722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9705136Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9705549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9706017Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9706213Z 2025-08-14T21:49:43.9706327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9706716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9707106Z return mod(**inputs) 2025-08-14T21:49:43.9707486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9707904Z outputs = self.model.decoder( 2025-08-14T21:49:43.9708341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9708962Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9709340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9709730Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9710188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9710642Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9711057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9711418Z return self.act(input) 2025-08-14T21:49:43.9711537Z 2025-08-14T21:49:43.9711656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9712029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9712378Z return mod(**inputs) 2025-08-14T21:49:43.9712769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9713181Z outputs = self.model.decoder( 2025-08-14T21:49:43.9713597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9714017Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9714394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9714781Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9715200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9715626Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9715848Z 2025-08-14T21:49:43.9715966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9716360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9716714Z return mod(**inputs) 2025-08-14T21:49:43.9717105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9717556Z outputs = self.model.decoder( 2025-08-14T21:49:43.9717959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9718368Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9718725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9719107Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9719515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:49:43.9719925Z hidden_states = residual + hidden_states 2025-08-14T21:49:43.9720068Z 2025-08-14T21:49:43.9720176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9720555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9720894Z return mod(**inputs) 2025-08-14T21:49:43.9721275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9721672Z outputs = self.model.decoder( 2025-08-14T21:49:43.9722106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9722487Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9722824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9723182Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9723563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9723983Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9724404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9724906Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9725118Z 2025-08-14T21:49:43.9725236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9725635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9725958Z return mod(**inputs) 2025-08-14T21:49:43.9726323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9726715Z outputs = self.model.decoder( 2025-08-14T21:49:43.9727094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9727487Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9727835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9728204Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9728588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9729006Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9729417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9729807Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9729953Z 2025-08-14T21:49:43.9730060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9730423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9730750Z return mod(**inputs) 2025-08-14T21:49:43.9731109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9731498Z outputs = self.model.decoder( 2025-08-14T21:49:43.9731898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9732283Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9732624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9732985Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9733371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9733778Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9734207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9734630Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9734781Z 2025-08-14T21:49:43.9734874Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9735102Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9735328Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9735560Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9735809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9736181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9736511Z return mod(**inputs) 2025-08-14T21:49:43.9736870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9737249Z outputs = self.model.decoder( 2025-08-14T21:49:43.9737627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9738009Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9738342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9738718Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9739109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9739528Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9739918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9740325Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9740759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9741242Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9741426Z 2025-08-14T21:49:43.9741530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9741892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9742224Z return mod(**inputs) 2025-08-14T21:49:43.9742573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9742954Z outputs = self.model.decoder( 2025-08-14T21:49:43.9743326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9743700Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9744026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9744375Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9744747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9745141Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9745547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9745977Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9746406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9746839Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9747003Z 2025-08-14T21:49:43.9747103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9747449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9747761Z return mod(**inputs) 2025-08-14T21:49:43.9748102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9748477Z outputs = self.model.decoder( 2025-08-14T21:49:43.9748848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9749209Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9749562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9749926Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9750297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9750684Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9751083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9751479Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9751613Z 2025-08-14T21:49:43.9751725Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9752104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9752452Z return mod(**inputs) 2025-08-14T21:49:43.9752839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9753261Z outputs = self.model.decoder( 2025-08-14T21:49:43.9753680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9754096Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9754469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9754862Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9755282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9755846Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9756045Z 2025-08-14T21:49:43.9756163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9756557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9756912Z return mod(**inputs) 2025-08-14T21:49:43.9757308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9757721Z outputs = self.model.decoder( 2025-08-14T21:49:43.9758139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9758557Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9758932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9759318Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9759727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9760291Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9760693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9761051Z return self.act(input) 2025-08-14T21:49:43.9761171Z 2025-08-14T21:49:43.9761278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9761653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9761986Z return mod(**inputs) 2025-08-14T21:49:43.9762376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9762783Z outputs = self.model.decoder( 2025-08-14T21:49:43.9763172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9763579Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9763947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9764367Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9764765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9765164Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9765295Z 2025-08-14T21:49:43.9765401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9765743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9766049Z return mod(**inputs) 2025-08-14T21:49:43.9766395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9766768Z outputs = self.model.decoder( 2025-08-14T21:49:43.9767143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9767522Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9767874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9768236Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9768619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9769028Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9769436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9769892Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9770103Z 2025-08-14T21:49:43.9770210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9770575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9770900Z return mod(**inputs) 2025-08-14T21:49:43.9771269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9771647Z outputs = self.model.decoder( 2025-08-14T21:49:43.9772016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9772393Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9772724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9773075Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9773463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9773886Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9774293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9774689Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9774826Z 2025-08-14T21:49:43.9774937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9775291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9775613Z return mod(**inputs) 2025-08-14T21:49:43.9775975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9776353Z outputs = self.model.decoder( 2025-08-14T21:49:43.9776742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9777115Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9777453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9777811Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9778203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9778605Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9779006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9779392Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9779541Z 2025-08-14T21:49:43.9779624Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9779841Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9780044Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9780251Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9780509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9780859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9781170Z return mod(**inputs) 2025-08-14T21:49:43.9781521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9781891Z outputs = self.model.decoder( 2025-08-14T21:49:43.9782250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9782618Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9782951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9783297Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9783662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9784059Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9784448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9784839Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9785268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9785734Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9785910Z 2025-08-14T21:49:43.9786017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9786355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9786670Z return mod(**inputs) 2025-08-14T21:49:43.9787023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9787452Z outputs = self.model.decoder( 2025-08-14T21:49:43.9787817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9788197Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9788536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9788880Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9789261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9789667Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9790075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9790479Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9790931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9791402Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9791578Z 2025-08-14T21:49:43.9791690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9792044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9792382Z return mod(**inputs) 2025-08-14T21:49:43.9792774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9793201Z outputs = self.model.decoder( 2025-08-14T21:49:43.9793622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9794043Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9795233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9795638Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9796156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9796612Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9797052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9797450Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9797591Z 2025-08-14T21:49:43.9797694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9798055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9798374Z return mod(**inputs) 2025-08-14T21:49:43.9798741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9799137Z outputs = self.model.decoder( 2025-08-14T21:49:43.9799519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9799894Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9800240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9800668Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9801045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9801478Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9801658Z 2025-08-14T21:49:43.9801761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9802116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9802463Z return mod(**inputs) 2025-08-14T21:49:43.9802822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9803208Z outputs = self.model.decoder( 2025-08-14T21:49:43.9803579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9803964Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9804313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9804678Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9805055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9805485Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9805877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9806219Z return self.act(input) 2025-08-14T21:49:43.9806352Z 2025-08-14T21:49:43.9806473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9806833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9807157Z return mod(**inputs) 2025-08-14T21:49:43.9807509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9807898Z outputs = self.model.decoder( 2025-08-14T21:49:43.9808277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9808797Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9809148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9809569Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9809949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9810325Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9810470Z 2025-08-14T21:49:43.9810573Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9810927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9811247Z return mod(**inputs) 2025-08-14T21:49:43.9811597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9811983Z outputs = self.model.decoder( 2025-08-14T21:49:43.9812372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9812745Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9813075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9813428Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9813804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:49:43.9814185Z hidden_states = residual + hidden_states 2025-08-14T21:49:43.9814325Z 2025-08-14T21:49:43.9814428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9814780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9815103Z return mod(**inputs) 2025-08-14T21:49:43.9815454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9815837Z outputs = self.model.decoder( 2025-08-14T21:49:43.9816255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9816617Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9816957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9817305Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9817677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9818067Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9818459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9818906Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9819101Z 2025-08-14T21:49:43.9819209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9819553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9819864Z return mod(**inputs) 2025-08-14T21:49:43.9820269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9820649Z outputs = self.model.decoder( 2025-08-14T21:49:43.9821029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9821412Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9821756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9822118Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9822496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9822920Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9823319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9823721Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9823864Z 2025-08-14T21:49:43.9823968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9824325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9824643Z return mod(**inputs) 2025-08-14T21:49:43.9825006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9825397Z outputs = self.model.decoder( 2025-08-14T21:49:43.9825777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9826161Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9826514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9826879Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9827265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9827682Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9828090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9828497Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9828639Z 2025-08-14T21:49:43.9828721Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9828942Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9829154Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9829356Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9829593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9829979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9830306Z return mod(**inputs) 2025-08-14T21:49:43.9830677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9831087Z outputs = self.model.decoder( 2025-08-14T21:49:43.9831488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9831900Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9832267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9832658Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9833069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9833499Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9833924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9834401Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9834887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9835410Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9835620Z 2025-08-14T21:49:43.9835800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9836205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9836553Z return mod(**inputs) 2025-08-14T21:49:43.9836948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9837403Z outputs = self.model.decoder( 2025-08-14T21:49:43.9837786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9838165Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9838513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9838872Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9839249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9839657Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9840063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9840472Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9840911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9841377Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9841558Z 2025-08-14T21:49:43.9841669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9842049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9842384Z return mod(**inputs) 2025-08-14T21:49:43.9842765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9843192Z outputs = self.model.decoder( 2025-08-14T21:49:43.9843559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9843954Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9844322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9844732Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9845130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9845560Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9845986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9846402Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9846544Z 2025-08-14T21:49:43.9846657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9847037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9847382Z return mod(**inputs) 2025-08-14T21:49:43.9847758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9848169Z outputs = self.model.decoder( 2025-08-14T21:49:43.9848568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9849053Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9849415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9849802Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9850221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9850691Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9850886Z 2025-08-14T21:49:43.9851004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9851394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9851761Z return mod(**inputs) 2025-08-14T21:49:43.9852143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9852562Z outputs = self.model.decoder( 2025-08-14T21:49:43.9852963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9853372Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9853725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9854103Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9854505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9854947Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9855353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9855711Z return self.act(input) 2025-08-14T21:49:43.9855827Z 2025-08-14T21:49:43.9855942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9856312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9856652Z return mod(**inputs) 2025-08-14T21:49:43.9857028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9857439Z outputs = self.model.decoder( 2025-08-14T21:49:43.9857840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9858264Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9858640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9859061Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9859482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9859927Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9860075Z 2025-08-14T21:49:43.9860196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9860574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9860924Z return mod(**inputs) 2025-08-14T21:49:43.9861318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9861740Z outputs = self.model.decoder( 2025-08-14T21:49:43.9862160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9862586Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9862976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9863370Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9863841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9864299Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9864739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9865247Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9865477Z 2025-08-14T21:49:43.9865592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9865982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9866329Z return mod(**inputs) 2025-08-14T21:49:43.9866741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9867162Z outputs = self.model.decoder( 2025-08-14T21:49:43.9867574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9867990Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9868369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9868757Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9869168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9869622Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9870062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9870910Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9871060Z 2025-08-14T21:49:43.9871175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9871570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9871927Z return mod(**inputs) 2025-08-14T21:49:43.9872314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9872751Z outputs = self.model.decoder( 2025-08-14T21:49:43.9873168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9873590Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9873963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9874359Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9874816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9875260Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9875746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9876309Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9876467Z 2025-08-14T21:49:43.9876567Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9876796Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9877029Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9877259Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9877519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9877910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9878279Z return mod(**inputs) 2025-08-14T21:49:43.9878695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9879138Z outputs = self.model.decoder( 2025-08-14T21:49:43.9879568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9879988Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9880362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9880745Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9881160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9881603Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9882052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9882495Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9882979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9883500Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9883680Z 2025-08-14T21:49:43.9883780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9884134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9884456Z return mod(**inputs) 2025-08-14T21:49:43.9884818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9885207Z outputs = self.model.decoder( 2025-08-14T21:49:43.9885584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9885971Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9886308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9886672Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9887056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9887466Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9887861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9888269Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9888711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9889171Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9889347Z 2025-08-14T21:49:43.9889448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9889792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9890105Z return mod(**inputs) 2025-08-14T21:49:43.9890445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9890818Z outputs = self.model.decoder( 2025-08-14T21:49:43.9891185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9891557Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9891885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9892234Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9892605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9892993Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9893420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9893806Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9893942Z 2025-08-14T21:49:43.9894054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9894405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9894734Z return mod(**inputs) 2025-08-14T21:49:43.9895096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9895485Z outputs = self.model.decoder( 2025-08-14T21:49:43.9895880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9896257Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9896591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9896940Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9897320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9897741Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9897911Z 2025-08-14T21:49:43.9898023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9898374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9898699Z return mod(**inputs) 2025-08-14T21:49:43.9899061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9899436Z outputs = self.model.decoder( 2025-08-14T21:49:43.9899804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9900185Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9900530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9900883Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9901268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9901699Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9902084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9902418Z return self.act(input) 2025-08-14T21:49:43.9902537Z 2025-08-14T21:49:43.9902660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9903021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9903341Z return mod(**inputs) 2025-08-14T21:49:43.9903711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9904133Z outputs = self.model.decoder( 2025-08-14T21:49:43.9904530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9904911Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9905262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9905627Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9906010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9906410Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9906553Z 2025-08-14T21:49:43.9906656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9907043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9907358Z return mod(**inputs) 2025-08-14T21:49:43.9907713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9908095Z outputs = self.model.decoder( 2025-08-14T21:49:43.9908469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9909003Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9909365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9909731Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9910168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:49:43.9910570Z hidden_states = residual + hidden_states 2025-08-14T21:49:43.9910712Z 2025-08-14T21:49:43.9910820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9911020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9911093Z return mod(**inputs) 2025-08-14T21:49:43.9911348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9911422Z outputs = self.model.decoder( 2025-08-14T21:49:43.9911685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9911757Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9911989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9912071Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9912343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9912460Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9912728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9912890Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9912902Z 2025-08-14T21:49:43.9913014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9913229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9913307Z return mod(**inputs) 2025-08-14T21:49:43.9913580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9913687Z outputs = self.model.decoder( 2025-08-14T21:49:43.9913958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9914035Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9914270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9914352Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9914622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9914733Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9914991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9915077Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9915083Z 2025-08-14T21:49:43.9915198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9915448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9915551Z return mod(**inputs) 2025-08-14T21:49:43.9915882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9915966Z outputs = self.model.decoder( 2025-08-14T21:49:43.9916235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9916309Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9916538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9916629Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9916924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9917034Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9917302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9917393Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9917398Z 2025-08-14T21:49:43.9917492Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9917575Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9917663Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9917743Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9917852Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9918066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9918135Z return mod(**inputs) 2025-08-14T21:49:43.9918415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9918502Z outputs = self.model.decoder( 2025-08-14T21:49:43.9918788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9918873Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9919101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9919183Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9919448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9919548Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9919807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9919948Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9920255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9920410Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9920414Z 2025-08-14T21:49:43.9920520Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9920725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9920801Z return mod(**inputs) 2025-08-14T21:49:43.9921074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9921157Z outputs = self.model.decoder( 2025-08-14T21:49:43.9921431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9921509Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9921742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9921843Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9922121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9922231Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9922488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9922595Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9922906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9923021Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9923027Z 2025-08-14T21:49:43.9923159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9923366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9923443Z return mod(**inputs) 2025-08-14T21:49:43.9923718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9923794Z outputs = self.model.decoder( 2025-08-14T21:49:43.9924068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9924142Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9924378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9924467Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9924728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9924837Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9925099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9925185Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9925189Z 2025-08-14T21:49:43.9925303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9925507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9925584Z return mod(**inputs) 2025-08-14T21:49:43.9925861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9925939Z outputs = self.model.decoder( 2025-08-14T21:49:43.9926220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9926316Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9926543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9926635Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9926897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9927029Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9927033Z 2025-08-14T21:49:43.9927138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9927345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9927425Z return mod(**inputs) 2025-08-14T21:49:43.9927696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9927779Z outputs = self.model.decoder( 2025-08-14T21:49:43.9928031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9928124Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9928361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9928445Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9928703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9928837Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9929065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9929142Z return self.act(input) 2025-08-14T21:49:43.9929145Z 2025-08-14T21:49:43.9929245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9929454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9929529Z return mod(**inputs) 2025-08-14T21:49:43.9929779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9929851Z outputs = self.model.decoder( 2025-08-14T21:49:43.9930105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9930177Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9930411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9930488Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9930726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9930816Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9930820Z 2025-08-14T21:49:43.9930921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9931123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9931190Z return mod(**inputs) 2025-08-14T21:49:43.9931438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9931526Z outputs = self.model.decoder( 2025-08-14T21:49:43.9931765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9931833Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9932051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9932127Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9932398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9932493Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9932739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9932898Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9932901Z 2025-08-14T21:49:43.9933001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9933200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9933266Z return mod(**inputs) 2025-08-14T21:49:43.9933511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9933590Z outputs = self.model.decoder( 2025-08-14T21:49:43.9933844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9933913Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9934173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9934254Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9934508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9934605Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9934862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9934954Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9934958Z 2025-08-14T21:49:43.9935062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9935287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9935368Z return mod(**inputs) 2025-08-14T21:49:43.9935622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9935703Z outputs = self.model.decoder( 2025-08-14T21:49:43.9935953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9936023Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9936251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9936327Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9936596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9936691Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9936934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9937028Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9937031Z 2025-08-14T21:49:43.9937110Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9937186Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9937265Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9937340Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9937445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9937636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9937698Z return mod(**inputs) 2025-08-14T21:49:43.9937947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9938018Z outputs = self.model.decoder( 2025-08-14T21:49:43.9938290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9938367Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9938579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9938662Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9938901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9938995Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9939244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9939337Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9939626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9939758Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9939762Z 2025-08-14T21:49:43.9939879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9940117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9940184Z return mod(**inputs) 2025-08-14T21:49:43.9940427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9940506Z outputs = self.model.decoder( 2025-08-14T21:49:43.9940744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9940820Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9941028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9941128Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9941377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9941473Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9941720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9941814Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9942090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9942207Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9942211Z 2025-08-14T21:49:43.9942311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9942498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9942574Z return mod(**inputs) 2025-08-14T21:49:43.9942815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9942898Z outputs = self.model.decoder( 2025-08-14T21:49:43.9943142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9943211Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9943429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9943504Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9943751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9943843Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9944082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9944185Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9944190Z 2025-08-14T21:49:43.9944291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9944482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9944552Z return mod(**inputs) 2025-08-14T21:49:43.9944794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9944872Z outputs = self.model.decoder( 2025-08-14T21:49:43.9945114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9945182Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9945400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9945480Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9945721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9945877Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9945881Z 2025-08-14T21:49:43.9945984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9946186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9946250Z return mod(**inputs) 2025-08-14T21:49:43.9946506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9946584Z outputs = self.model.decoder( 2025-08-14T21:49:43.9946825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9946917Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9947126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9947204Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9947450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9947563Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9947782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9947864Z return self.act(input) 2025-08-14T21:49:43.9947868Z 2025-08-14T21:49:43.9947972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9948183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9948251Z return mod(**inputs) 2025-08-14T21:49:43.9948515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9948599Z outputs = self.model.decoder( 2025-08-14T21:49:43.9948861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9948939Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9949165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9949247Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9949513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9949596Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9949600Z 2025-08-14T21:49:43.9949707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9949943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9950010Z return mod(**inputs) 2025-08-14T21:49:43.9950279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9950356Z outputs = self.model.decoder( 2025-08-14T21:49:43.9950616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9950697Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9950921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9951001Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9951268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:49:43.9951349Z hidden_states = residual + hidden_states 2025-08-14T21:49:43.9951354Z 2025-08-14T21:49:43.9951468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9951671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9951774Z return mod(**inputs) 2025-08-14T21:49:43.9952049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9952125Z outputs = self.model.decoder( 2025-08-14T21:49:43.9952391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9952466Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9952693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9952783Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9953073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9953181Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9953462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9953626Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9953630Z 2025-08-14T21:49:43.9953744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9953959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9954027Z return mod(**inputs) 2025-08-14T21:49:43.9954308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9954386Z outputs = self.model.decoder( 2025-08-14T21:49:43.9954665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9954741Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9954979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9955073Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9955346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9955450Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9955814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9955908Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9955912Z 2025-08-14T21:49:43.9956031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9956251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9956355Z return mod(**inputs) 2025-08-14T21:49:43.9956637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9956720Z outputs = self.model.decoder( 2025-08-14T21:49:43.9957000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9957078Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9957310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9957403Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9957672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9957777Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9958067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9958159Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9958178Z 2025-08-14T21:49:43.9958287Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9958374Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9958456Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9958547Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9958651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9958855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9958932Z return mod(**inputs) 2025-08-14T21:49:43.9959192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9959274Z outputs = self.model.decoder( 2025-08-14T21:49:43.9959569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9959646Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9959888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9959970Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9960232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9960341Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9960602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9960711Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9961018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9961164Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9961167Z 2025-08-14T21:49:43.9961282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9961493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9961568Z return mod(**inputs) 2025-08-14T21:49:43.9961833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9961912Z outputs = self.model.decoder( 2025-08-14T21:49:43.9962193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9962266Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9962497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9962605Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9962870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9962973Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9963213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9963306Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9963596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9963703Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9963706Z 2025-08-14T21:49:43.9963812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9964006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9964074Z return mod(**inputs) 2025-08-14T21:49:43.9964334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9964422Z outputs = self.model.decoder( 2025-08-14T21:49:43.9964726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9964811Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9965049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9965142Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9965411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9965508Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9965774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9965857Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9965861Z 2025-08-14T21:49:43.9965969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9966165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9966230Z return mod(**inputs) 2025-08-14T21:49:43.9966488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9966560Z outputs = self.model.decoder( 2025-08-14T21:49:43.9966820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9966896Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9967105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9967192Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9967434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9967549Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9967552Z 2025-08-14T21:49:43.9967659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9967851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9967922Z return mod(**inputs) 2025-08-14T21:49:43.9968166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9968238Z outputs = self.model.decoder( 2025-08-14T21:49:43.9968487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9968576Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9968788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9968873Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9969113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9969233Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9969435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9969503Z return self.act(input) 2025-08-14T21:49:43.9969506Z 2025-08-14T21:49:43.9969611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9969798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9969869Z return mod(**inputs) 2025-08-14T21:49:43.9970115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9970187Z outputs = self.model.decoder( 2025-08-14T21:49:43.9970472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9970543Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9970754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9970838Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9971078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9971163Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9971167Z 2025-08-14T21:49:43.9971264Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9971472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9971548Z return mod(**inputs) 2025-08-14T21:49:43.9971796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9971868Z outputs = self.model.decoder( 2025-08-14T21:49:43.9972119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9972187Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9972404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9972480Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9972724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9972823Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9973068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:43.9973220Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:43.9973226Z 2025-08-14T21:49:43.9973324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9973512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9973582Z return mod(**inputs) 2025-08-14T21:49:43.9973825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9973901Z outputs = self.model.decoder( 2025-08-14T21:49:43.9974156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9974228Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9974456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9974560Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9974831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9974941Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9975212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:43.9975299Z key_states = self.k_proj(current_states) 2025-08-14T21:49:43.9975302Z 2025-08-14T21:49:43.9975404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9975604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9975676Z return mod(**inputs) 2025-08-14T21:49:43.9975931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9976005Z outputs = self.model.decoder( 2025-08-14T21:49:43.9976287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9976373Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9976596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9976672Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9976920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9977023Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9977274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:43.9977365Z value_states = self.v_proj(current_states) 2025-08-14T21:49:43.9977370Z 2025-08-14T21:49:43.9977460Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9977536Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9977618Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9977692Z cudagraph partition due to non gpu ops 2025-08-14T21:49:43.9977790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9977986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9978050Z return mod(**inputs) 2025-08-14T21:49:43.9978294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9978371Z outputs = self.model.decoder( 2025-08-14T21:49:43.9978624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9978701Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9978922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9979019Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9979282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9979376Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9979628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9979724Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9980013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:43.9980153Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:43.9980156Z 2025-08-14T21:49:43.9980257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9980505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9980570Z return mod(**inputs) 2025-08-14T21:49:43.9980819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9980898Z outputs = self.model.decoder( 2025-08-14T21:49:43.9981147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9981216Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9981438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9981517Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9981769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9981867Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9982113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:43.9982255Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:43.9982544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:43.9982653Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:43.9982664Z 2025-08-14T21:49:43.9982765Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9982961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9983033Z return mod(**inputs) 2025-08-14T21:49:43.9983284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9983378Z outputs = self.model.decoder( 2025-08-14T21:49:43.9983665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9983741Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9983978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9984061Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9984321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:43.9984430Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:43.9984692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:43.9984777Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:43.9984789Z 2025-08-14T21:49:43.9984900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9985108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9985194Z return mod(**inputs) 2025-08-14T21:49:43.9985445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9985518Z outputs = self.model.decoder( 2025-08-14T21:49:43.9985774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9985844Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9986067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9986144Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9986390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9986533Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9986536Z 2025-08-14T21:49:43.9986638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9986834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9986906Z return mod(**inputs) 2025-08-14T21:49:43.9987151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9987232Z outputs = self.model.decoder( 2025-08-14T21:49:43.9987477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9987549Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9987770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9987850Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9988105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:43.9988254Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:43.9988462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:43.9988539Z return self.act(input) 2025-08-14T21:49:43.9988543Z 2025-08-14T21:49:43.9988640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9988834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9988907Z return mod(**inputs) 2025-08-14T21:49:43.9989156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9989236Z outputs = self.model.decoder( 2025-08-14T21:49:43.9989496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9989568Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9989794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9989872Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9990116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:43.9990203Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:43.9990207Z 2025-08-14T21:49:43.9990307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9990511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9990576Z return mod(**inputs) 2025-08-14T21:49:43.9990824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:49:43.9990907Z outputs = self.model.decoder( 2025-08-14T21:49:43.9991205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:43.9991284Z layer_outputs = decoder_layer( 2025-08-14T21:49:43.9991502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:43.9991579Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:43.9991833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:49:43.9991913Z hidden_states = residual + hidden_states 2025-08-14T21:49:43.9991916Z 2025-08-14T21:49:43.9992017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9992215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9992300Z return mod(**inputs) 2025-08-14T21:49:43.9992575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1880, in forward 2025-08-14T21:49:43.9992659Z logits = self.lm_head(outputs[0]) 2025-08-14T21:49:43.9992663Z 2025-08-14T21:49:43.9992767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:43.9992979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:43.9993047Z return mod(**inputs) 2025-08-14T21:49:43.9993327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1886, in forward 2025-08-14T21:49:43.9993483Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:49:43.9993487Z 2025-08-14T21:49:53.8328840Z Compilation time (from dynamo_timed): 15.747412488 2025-08-14T21:49:53.8620927Z pass 2025-08-14T21:49:53.8621657Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:53.8623552Z TIMING: _recursive_pre_grad_passes:0.00761 _recursive_joint_graph_passes:0.65816 _recursive_post_grad_passes:0.08419 async_compile.wait:0.66925 code_gen:8.44675 inductor_compile:9.74173 backend_compile:13.06078 gc:0.00162 entire_frame_compile:15.74741 total_wall_time:15.74741 2025-08-14T21:49:53.8624655Z STATS: call_* op count: 373 | FakeTensorMode.__torch_dispatch__:13266 | FakeTensor.__torch_dispatch__:4931 | ProxyTorchDispatchMode.__torch_dispatch__:4844 2025-08-14T21:49:53.8625197Z Dynamo produced 1 graphs covering 373 ops with 0 graph breaks (0 unique) 2025-08-14T21:49:59.0992899Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:49:59.0995279Z from pkg_resources import resource_filename 2025-08-14T21:49:59.6751657Z 2025-08-14T21:50:04.9004600Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:50:04.9004951Z loading model: 0it [00:05, ?it/s] 2025-08-14T21:50:04.9059427Z cpu eval MBartForConditionalGeneration 2025-08-14T21:50:08.6317472Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:10.1600764Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:11.6859724Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:29.1973043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.1973511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.1973885Z return mod(**inputs) 2025-08-14T21:50:29.1974360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1436, in forward 2025-08-14T21:50:29.1974889Z decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id) 2025-08-14T21:50:29.1975436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 76, in shift_tokens_right 2025-08-14T21:50:29.1975961Z index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1) 2025-08-14T21:50:29.1976182Z 2025-08-14T21:50:29.1976271Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1976489Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1976691Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1976921Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1977127Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1977328Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1977523Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1978035Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1978253Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1978453Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1978662Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1978874Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1979110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.1979532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.1979895Z return mod(**inputs) 2025-08-14T21:50:29.1980308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.1980732Z outputs = self.model( 2025-08-14T21:50:29.1981139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.1981593Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.1981989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.1982379Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.1982873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.1983254Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.1983677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.1984126Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.1984544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.1985016Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.1985234Z 2025-08-14T21:50:29.1985344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.1985763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.1986109Z return mod(**inputs) 2025-08-14T21:50:29.1986504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.1986922Z outputs = self.model( 2025-08-14T21:50:29.1987319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.1987806Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.1988216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.1988694Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.1989077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.1989465Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.1989889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.1990322Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.1990755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.1991174Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.1991326Z 2025-08-14T21:50:29.1991436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.1991818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.1992162Z return mod(**inputs) 2025-08-14T21:50:29.1992549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.1993020Z outputs = self.model( 2025-08-14T21:50:29.1993443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.1993857Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.1994286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.1994715Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.1995093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.1995495Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.1996289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.1996755Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.1997203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.1997649Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.1997812Z 2025-08-14T21:50:29.1997901Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1998184Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1998420Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1998643Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.1998894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.1999269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.1999614Z return mod(**inputs) 2025-08-14T21:50:29.2000008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2000420Z outputs = self.model( 2025-08-14T21:50:29.2000805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2001233Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2001642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2002063Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2002440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2002836Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2003239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2003635Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2004039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2004462Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2004942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2005448Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2005655Z 2025-08-14T21:50:29.2005804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2006206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2006526Z return mod(**inputs) 2025-08-14T21:50:29.2006891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2007275Z outputs = self.model( 2025-08-14T21:50:29.2007634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2008022Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2008398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2009006Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2009390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2009759Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2010154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2010569Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2010970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2011418Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2011899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2012393Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2012571Z 2025-08-14T21:50:29.2012684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2013071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2013474Z return mod(**inputs) 2025-08-14T21:50:29.2013844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2014223Z outputs = self.model( 2025-08-14T21:50:29.2014606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2015022Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2015421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2015829Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2016223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2016620Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2017029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2017455Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2017891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2018304Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2018459Z 2025-08-14T21:50:29.2018570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2018950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2019296Z return mod(**inputs) 2025-08-14T21:50:29.2019676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2020086Z outputs = self.model( 2025-08-14T21:50:29.2020475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2020886Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2021285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2021698Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2022072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2022447Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2022859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2023317Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2023529Z 2025-08-14T21:50:29.2023649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2024017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2024359Z return mod(**inputs) 2025-08-14T21:50:29.2024741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2025140Z outputs = self.model( 2025-08-14T21:50:29.2025513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2025926Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2026324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2026721Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2027088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2027470Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2027892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2028358Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2028766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2029123Z return self.act(input) 2025-08-14T21:50:29.2029240Z 2025-08-14T21:50:29.2029349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2029733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2030093Z return mod(**inputs) 2025-08-14T21:50:29.2030487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2030907Z outputs = self.model( 2025-08-14T21:50:29.2031304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2031727Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2032132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2032551Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2032926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2033318Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2033734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2034162Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2034310Z 2025-08-14T21:50:29.2034437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2034837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2035185Z return mod(**inputs) 2025-08-14T21:50:29.2035580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2036177Z outputs = self.model( 2025-08-14T21:50:29.2036575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2036999Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2037449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2037864Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2038232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2038665Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2039083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2039519Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2039961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2040467Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2040716Z 2025-08-14T21:50:29.2040838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2041223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2041575Z return mod(**inputs) 2025-08-14T21:50:29.2041967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2042384Z outputs = self.model( 2025-08-14T21:50:29.2042773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2043214Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2043646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2044054Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2044429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2044820Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2045239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2045672Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2046123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2046557Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2046708Z 2025-08-14T21:50:29.2046832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2047276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2047649Z return mod(**inputs) 2025-08-14T21:50:29.2048037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2048438Z outputs = self.model( 2025-08-14T21:50:29.2048823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2049245Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2049647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2050052Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2050420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2050803Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2051205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2051645Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2052072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2052497Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2052649Z 2025-08-14T21:50:29.2052735Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2052963Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2053186Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2053399Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2053673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2054056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2054408Z return mod(**inputs) 2025-08-14T21:50:29.2054821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2055226Z outputs = self.model( 2025-08-14T21:50:29.2055608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2056015Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2056417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2056829Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2057196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2057568Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2057975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2058438Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2058863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2059291Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2059764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2060276Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2060472Z 2025-08-14T21:50:29.2060582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2060989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2061338Z return mod(**inputs) 2025-08-14T21:50:29.2061723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2062121Z outputs = self.model( 2025-08-14T21:50:29.2062506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2062917Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2063319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2063718Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2064087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2064464Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2064871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2065297Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2065720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2066171Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2066628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2067111Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2067280Z 2025-08-14T21:50:29.2067398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2067776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2068115Z return mod(**inputs) 2025-08-14T21:50:29.2068527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2068931Z outputs = self.model( 2025-08-14T21:50:29.2069311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2069718Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2070120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2070541Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2070911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2071301Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2071721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2072156Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2072591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2073041Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2073206Z 2025-08-14T21:50:29.2073326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2073704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2074062Z return mod(**inputs) 2025-08-14T21:50:29.2074454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2074873Z outputs = self.model( 2025-08-14T21:50:29.2075257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2075678Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2076303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2076728Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2077125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2077528Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2077947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2078515Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2078708Z 2025-08-14T21:50:29.2078820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2079204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2079541Z return mod(**inputs) 2025-08-14T21:50:29.2079932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2080341Z outputs = self.model( 2025-08-14T21:50:29.2080733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2081138Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2081546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2081964Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2082335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2082711Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2083126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2083601Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2084031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2084398Z return self.act(input) 2025-08-14T21:50:29.2084525Z 2025-08-14T21:50:29.2084640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2085026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2085370Z return mod(**inputs) 2025-08-14T21:50:29.2085769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2086189Z outputs = self.model( 2025-08-14T21:50:29.2086562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2086968Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2087366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2087769Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2088126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2088545Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2088958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2089401Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2089551Z 2025-08-14T21:50:29.2089665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2090057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2090409Z return mod(**inputs) 2025-08-14T21:50:29.2090797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2091245Z outputs = self.model( 2025-08-14T21:50:29.2091629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2092034Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2092430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2092832Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2093211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2093604Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2094025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:50:29.2094647Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2094792Z 2025-08-14T21:50:29.2094910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2095290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2095632Z return mod(**inputs) 2025-08-14T21:50:29.2096022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2096428Z outputs = self.model( 2025-08-14T21:50:29.2096799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2097216Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2097625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2098043Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2098423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2098849Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2099279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2099720Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2100162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2100669Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2100885Z 2025-08-14T21:50:29.2101004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2101381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2101727Z return mod(**inputs) 2025-08-14T21:50:29.2102124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2102550Z outputs = self.model( 2025-08-14T21:50:29.2102962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2103401Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2103830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2104247Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2104618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2105027Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2105453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2105952Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2106449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2106884Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2107030Z 2025-08-14T21:50:29.2107145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2107536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2107891Z return mod(**inputs) 2025-08-14T21:50:29.2108289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2108831Z outputs = self.model( 2025-08-14T21:50:29.2109236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2109659Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2110072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2110500Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2110887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2111287Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2111702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2112145Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2112591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2113034Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2113201Z 2025-08-14T21:50:29.2113291Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2113527Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2113762Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2113982Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2114328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2114719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2115066Z return mod(**inputs) 2025-08-14T21:50:29.2115461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2115946Z outputs = self.model( 2025-08-14T21:50:29.2116374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2116802Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2117224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2117649Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2118028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2118423Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2118847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2119349Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2119785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2120246Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2120742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2121277Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2121485Z 2025-08-14T21:50:29.2121604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2122042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2122387Z return mod(**inputs) 2025-08-14T21:50:29.2122772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2123186Z outputs = self.model( 2025-08-14T21:50:29.2123575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2123986Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2124382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2124795Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2125146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2125506Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2125902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2126309Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2126717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2127123Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2127567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2128028Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2128196Z 2025-08-14T21:50:29.2128315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2128699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2129049Z return mod(**inputs) 2025-08-14T21:50:29.2129455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2129852Z outputs = self.model( 2025-08-14T21:50:29.2130238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2130619Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2130996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2131371Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2131718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2132089Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2132502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2132932Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2133352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2133848Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2133999Z 2025-08-14T21:50:29.2134104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2134462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2134786Z return mod(**inputs) 2025-08-14T21:50:29.2135146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2135552Z outputs = self.model( 2025-08-14T21:50:29.2135937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2136347Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2137545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2137985Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2138357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2138734Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2139137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2139594Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2139778Z 2025-08-14T21:50:29.2139898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2140278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2140616Z return mod(**inputs) 2025-08-14T21:50:29.2141001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2141403Z outputs = self.model( 2025-08-14T21:50:29.2142526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2142931Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2143327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2143728Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2144085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2144461Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2144867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2145311Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2145747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2146106Z return self.act(input) 2025-08-14T21:50:29.2146222Z 2025-08-14T21:50:29.2146339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2146708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2147050Z return mod(**inputs) 2025-08-14T21:50:29.2147431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2147832Z outputs = self.model( 2025-08-14T21:50:29.2148209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2148614Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2149017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2149416Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2149781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2150209Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2150619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2151031Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2151183Z 2025-08-14T21:50:29.2151292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2151674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2152011Z return mod(**inputs) 2025-08-14T21:50:29.2152395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2152822Z outputs = self.model( 2025-08-14T21:50:29.2153221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2153639Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2154056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2154487Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2154863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2155258Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2155679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2156225Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2156658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2157162Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2157393Z 2025-08-14T21:50:29.2157509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2157898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2158244Z return mod(**inputs) 2025-08-14T21:50:29.2158640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2159060Z outputs = self.model( 2025-08-14T21:50:29.2159446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2159876Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2160287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2160724Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2161090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2161483Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2161901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2162342Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2162767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2163186Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2163332Z 2025-08-14T21:50:29.2163452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2163831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2164187Z return mod(**inputs) 2025-08-14T21:50:29.2164577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2165005Z outputs = self.model( 2025-08-14T21:50:29.2165420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2165827Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2166231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2166608Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2166956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2167316Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2167740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2168164Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2168596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2169020Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2169171Z 2025-08-14T21:50:29.2169264Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2169484Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2169708Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2169933Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2170160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2170521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2170850Z return mod(**inputs) 2025-08-14T21:50:29.2171211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2171606Z outputs = self.model( 2025-08-14T21:50:29.2171972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2172367Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2172742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2173151Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2173527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2173926Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2174331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2174767Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2175213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2175644Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2176115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2176622Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2176816Z 2025-08-14T21:50:29.2176930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2177301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2177645Z return mod(**inputs) 2025-08-14T21:50:29.2178033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2178416Z outputs = self.model( 2025-08-14T21:50:29.2178773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2179159Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2179567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2179948Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2180302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2180668Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2181059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2181461Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2181866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2182303Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2182776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2183277Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2183458Z 2025-08-14T21:50:29.2183572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2183961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2184312Z return mod(**inputs) 2025-08-14T21:50:29.2184686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2185077Z outputs = self.model( 2025-08-14T21:50:29.2185458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2185882Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2186309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2186737Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2187113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2187509Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2187930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2188375Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2188810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2189254Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2189403Z 2025-08-14T21:50:29.2189524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2189922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2190269Z return mod(**inputs) 2025-08-14T21:50:29.2190676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2191095Z outputs = self.model( 2025-08-14T21:50:29.2191488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2191907Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2192311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2192732Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2193092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2193478Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2193899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2194403Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2194599Z 2025-08-14T21:50:29.2194716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2195109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2195464Z return mod(**inputs) 2025-08-14T21:50:29.2195946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2196369Z outputs = self.model( 2025-08-14T21:50:29.2196768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2197181Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2197628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2198039Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2198412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2198789Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2199204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2199672Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2200077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2200413Z return self.act(input) 2025-08-14T21:50:29.2200531Z 2025-08-14T21:50:29.2200636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2200997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2201315Z return mod(**inputs) 2025-08-14T21:50:29.2201683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2202066Z outputs = self.model( 2025-08-14T21:50:29.2202431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2202813Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2203211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2203625Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2203985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2204367Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2204813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2205236Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2205382Z 2025-08-14T21:50:29.2205493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2205869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2206191Z return mod(**inputs) 2025-08-14T21:50:29.2206544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2206932Z outputs = self.model( 2025-08-14T21:50:29.2207317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2207722Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2208113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2208516Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2209011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2209468Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2209847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:50:29.2210239Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2210374Z 2025-08-14T21:50:29.2210487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2210838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2211167Z return mod(**inputs) 2025-08-14T21:50:29.2211538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2211954Z outputs = self.model( 2025-08-14T21:50:29.2212308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2212691Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2213067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2213441Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2213785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2214139Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2214519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2214911Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2215307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2215794Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2216006Z 2025-08-14T21:50:29.2216126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2216499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2216840Z return mod(**inputs) 2025-08-14T21:50:29.2217219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2217614Z outputs = self.model( 2025-08-14T21:50:29.2217975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2218363Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2218760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2219195Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2219565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2219951Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2220352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2220792Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2221212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2221634Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2221779Z 2025-08-14T21:50:29.2221891Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2222271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2222621Z return mod(**inputs) 2025-08-14T21:50:29.2223003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2223422Z outputs = self.model( 2025-08-14T21:50:29.2223827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2224238Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2224633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2225038Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2225405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2225787Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2226211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2226644Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2227070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2227486Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2227644Z 2025-08-14T21:50:29.2227731Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2227961Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2228186Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2228399Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2228646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2229025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2229367Z return mod(**inputs) 2025-08-14T21:50:29.2229760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2230168Z outputs = self.model( 2025-08-14T21:50:29.2230557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2230965Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2231369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2231780Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2232145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2232529Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2232939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2233375Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2233828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2234280Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2234772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2235300Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2235500Z 2025-08-14T21:50:29.2235614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2236079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2236437Z return mod(**inputs) 2025-08-14T21:50:29.2236830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2237248Z outputs = self.model( 2025-08-14T21:50:29.2237656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2238082Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2238534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2238955Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2239329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2239719Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2240140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2240588Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2241024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2241488Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2241966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2242466Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2242642Z 2025-08-14T21:50:29.2242761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2243147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2243500Z return mod(**inputs) 2025-08-14T21:50:29.2243896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2244304Z outputs = self.model( 2025-08-14T21:50:29.2244696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2245122Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2245542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2245962Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2246346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2246739Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2247158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2247597Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2248037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2248478Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2248623Z 2025-08-14T21:50:29.2248737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2249149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2249499Z return mod(**inputs) 2025-08-14T21:50:29.2249860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2250234Z outputs = self.model( 2025-08-14T21:50:29.2250595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2250979Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2251370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2251788Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2252152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2252592Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2252968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2253445Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2253616Z 2025-08-14T21:50:29.2253729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2254096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2254424Z return mod(**inputs) 2025-08-14T21:50:29.2254823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2255236Z outputs = self.model( 2025-08-14T21:50:29.2255615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2256038Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2256466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2256869Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2257225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2257598Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2257998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2258435Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2258832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2259184Z return self.act(input) 2025-08-14T21:50:29.2259300Z 2025-08-14T21:50:29.2259414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2259785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2260124Z return mod(**inputs) 2025-08-14T21:50:29.2260493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2260894Z outputs = self.model( 2025-08-14T21:50:29.2261273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2261681Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2262079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2262462Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2262810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2263186Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2263625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2264031Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2264186Z 2025-08-14T21:50:29.2264297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2264670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2265001Z return mod(**inputs) 2025-08-14T21:50:29.2265358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2265748Z outputs = self.model( 2025-08-14T21:50:29.2266138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2266516Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2266907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2267309Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2267683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2268079Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2268489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2268915Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2269330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2269826Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2270047Z 2025-08-14T21:50:29.2270160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2270556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2270896Z return mod(**inputs) 2025-08-14T21:50:29.2271277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2271686Z outputs = self.model( 2025-08-14T21:50:29.2272068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2272478Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2272885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2273294Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2273662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2274059Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2274489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2274934Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2275368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2275882Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2276031Z 2025-08-14T21:50:29.2276154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2276540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2276908Z return mod(**inputs) 2025-08-14T21:50:29.2277304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2277722Z outputs = self.model( 2025-08-14T21:50:29.2278104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2278547Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2278955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2279361Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2279733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2280123Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2280537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2280969Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2281393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2281829Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2281983Z 2025-08-14T21:50:29.2282081Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2282304Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2282547Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2282782Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2283025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2283406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2283749Z return mod(**inputs) 2025-08-14T21:50:29.2284123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2284527Z outputs = self.model( 2025-08-14T21:50:29.2284908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2285323Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2285734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2286153Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2286521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2286892Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2287364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2287799Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2288225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2288663Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2289134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2289657Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2289846Z 2025-08-14T21:50:29.2289962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2290327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2290660Z return mod(**inputs) 2025-08-14T21:50:29.2291050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2291456Z outputs = self.model( 2025-08-14T21:50:29.2291847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2292255Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2292658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2293093Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2293532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2293926Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2294340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2294769Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2295211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2295660Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2296122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2296610Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2296796Z 2025-08-14T21:50:29.2296909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2297293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2297643Z return mod(**inputs) 2025-08-14T21:50:29.2298026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2298441Z outputs = self.model( 2025-08-14T21:50:29.2298831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2299219Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2299661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2300054Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2300403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2300795Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2301196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2301629Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2302019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2302413Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2302547Z 2025-08-14T21:50:29.2302660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2303009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2303332Z return mod(**inputs) 2025-08-14T21:50:29.2303705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2304108Z outputs = self.model( 2025-08-14T21:50:29.2304478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2304885Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2305285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2305690Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2306049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2306435Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2306839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2307287Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2307478Z 2025-08-14T21:50:29.2307611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2307991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2308337Z return mod(**inputs) 2025-08-14T21:50:29.2308848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2309262Z outputs = self.model( 2025-08-14T21:50:29.2309655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2310061Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2310467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2310877Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2311249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2311632Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2312048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2312571Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2312981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2313331Z return self.act(input) 2025-08-14T21:50:29.2313454Z 2025-08-14T21:50:29.2313563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2313943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2314277Z return mod(**inputs) 2025-08-14T21:50:29.2314662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2315064Z outputs = self.model( 2025-08-14T21:50:29.2315483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2315955Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2316382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2316800Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2317179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2317572Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2317984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2318409Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2318555Z 2025-08-14T21:50:29.2318667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2319049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2319396Z return mod(**inputs) 2025-08-14T21:50:29.2319770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2320177Z outputs = self.model( 2025-08-14T21:50:29.2320565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2320970Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2321363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2321781Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2322146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2322534Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2322976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:50:29.2323392Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2323536Z 2025-08-14T21:50:29.2323655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2324030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2324373Z return mod(**inputs) 2025-08-14T21:50:29.2324746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2325128Z outputs = self.model( 2025-08-14T21:50:29.2325480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2325866Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2326251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2326632Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2327004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2327381Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2327771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2328171Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2328577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2329064Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2329269Z 2025-08-14T21:50:29.2329381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2329752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2330083Z return mod(**inputs) 2025-08-14T21:50:29.2330456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2330844Z outputs = self.model( 2025-08-14T21:50:29.2331219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2331616Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2332059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2332447Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2332802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2333177Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2333604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2334041Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2334458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2334863Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2335001Z 2025-08-14T21:50:29.2335106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2335474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2335810Z return mod(**inputs) 2025-08-14T21:50:29.2336182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2336568Z outputs = self.model( 2025-08-14T21:50:29.2336943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2337352Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2337739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2338129Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2338486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2338860Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2339259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2339661Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2340061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2340450Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2340600Z 2025-08-14T21:50:29.2340686Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2340908Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2341130Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2341338Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2341572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2341926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2342243Z return mod(**inputs) 2025-08-14T21:50:29.2342605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2342985Z outputs = self.model( 2025-08-14T21:50:29.2343347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2343733Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2344140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2344532Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2344881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2345253Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2345632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2346030Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2346415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2346840Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2347321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2347808Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2347991Z 2025-08-14T21:50:29.2348098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2348465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2348817Z return mod(**inputs) 2025-08-14T21:50:29.2349169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2349548Z outputs = self.model( 2025-08-14T21:50:29.2349904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2350284Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2350651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2351092Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2351433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2351790Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2352174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2352583Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2352989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2353420Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2353885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2354377Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2354552Z 2025-08-14T21:50:29.2354673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2355049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2355412Z return mod(**inputs) 2025-08-14T21:50:29.2355908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2356373Z outputs = self.model( 2025-08-14T21:50:29.2356788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2357219Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2357634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2358046Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2358417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2358837Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2359250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2359666Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2360085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2360515Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2360658Z 2025-08-14T21:50:29.2360769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2361149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2361491Z return mod(**inputs) 2025-08-14T21:50:29.2361874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2362276Z outputs = self.model( 2025-08-14T21:50:29.2362658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2363068Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2363461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2363872Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2364209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2364560Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2364934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2365375Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2365542Z 2025-08-14T21:50:29.2365675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2366024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2366338Z return mod(**inputs) 2025-08-14T21:50:29.2366691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2367061Z outputs = self.model( 2025-08-14T21:50:29.2367406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2367789Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2368156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2368526Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2368855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2369208Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2369584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2370070Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2370446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2370776Z return self.act(input) 2025-08-14T21:50:29.2370884Z 2025-08-14T21:50:29.2370990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2371331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2371645Z return mod(**inputs) 2025-08-14T21:50:29.2371995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2372363Z outputs = self.model( 2025-08-14T21:50:29.2372722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2373103Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2373351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2373434Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2373646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2373725Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2373982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2374064Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2374068Z 2025-08-14T21:50:29.2374180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2374381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2374456Z return mod(**inputs) 2025-08-14T21:50:29.2374709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2374775Z outputs = self.model( 2025-08-14T21:50:29.2375023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2375093Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2375335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2375412Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2375622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2375698Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2375962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2376052Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2376300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2376446Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2376451Z 2025-08-14T21:50:29.2376552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2376750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2376813Z return mod(**inputs) 2025-08-14T21:50:29.2377063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2377130Z outputs = self.model( 2025-08-14T21:50:29.2377375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2377454Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2377725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2377798Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2378017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2378092Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2378344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2378433Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2378675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2378778Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2378782Z 2025-08-14T21:50:29.2378884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2379084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2379148Z return mod(**inputs) 2025-08-14T21:50:29.2379391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2379464Z outputs = self.model( 2025-08-14T21:50:29.2379708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2379780Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2380025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2380097Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2380318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2380394Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2380640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2380736Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2380975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2381058Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2381070Z 2025-08-14T21:50:29.2381150Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2381226Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2381308Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2381382Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2381482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2381715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2381778Z return mod(**inputs) 2025-08-14T21:50:29.2382019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2382094Z outputs = self.model( 2025-08-14T21:50:29.2382336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2382412Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2382653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2382721Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2382943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2383023Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2383278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2383400Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2383647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2383754Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2384042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2384182Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2384186Z 2025-08-14T21:50:29.2384292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2384481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2384566Z return mod(**inputs) 2025-08-14T21:50:29.2384818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2384884Z outputs = self.model( 2025-08-14T21:50:29.2385129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2385200Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2385442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2385510Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2385715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2385795Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2386036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2386125Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2386376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2386472Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2386759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2386866Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2386870Z 2025-08-14T21:50:29.2386969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2387168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2387233Z return mod(**inputs) 2025-08-14T21:50:29.2387478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2387570Z outputs = self.model( 2025-08-14T21:50:29.2387814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2387895Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2388136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2388204Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2388424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2388499Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2388748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2388835Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2389082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2389169Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2389189Z 2025-08-14T21:50:29.2389302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2389493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2389565Z return mod(**inputs) 2025-08-14T21:50:29.2389804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2389876Z outputs = self.model( 2025-08-14T21:50:29.2390121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2390194Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2390463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2390535Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2390758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2390840Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2391087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2391213Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2391217Z 2025-08-14T21:50:29.2391316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2391511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2391583Z return mod(**inputs) 2025-08-14T21:50:29.2391830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2391909Z outputs = self.model( 2025-08-14T21:50:29.2392156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2392232Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2392482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2392552Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2392768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2392854Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2393101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2393234Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2393456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2393549Z return self.act(input) 2025-08-14T21:50:29.2393553Z 2025-08-14T21:50:29.2393670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2393882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2393960Z return mod(**inputs) 2025-08-14T21:50:29.2394228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2394300Z outputs = self.model( 2025-08-14T21:50:29.2394579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2394655Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2394921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2395009Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2395243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2395373Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2395634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2395804Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2395811Z 2025-08-14T21:50:29.2395938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2396153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2396231Z return mod(**inputs) 2025-08-14T21:50:29.2396504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2396577Z outputs = self.model( 2025-08-14T21:50:29.2396881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2396963Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2397238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2397327Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2397570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2397663Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2397929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:50:29.2398016Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2398019Z 2025-08-14T21:50:29.2398134Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2398344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2398414Z return mod(**inputs) 2025-08-14T21:50:29.2398685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2398756Z outputs = self.model( 2025-08-14T21:50:29.2399025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2399101Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2399361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2399444Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2399672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2399760Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2400042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2400140Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2400408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2400569Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2400572Z 2025-08-14T21:50:29.2400683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2400899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2400967Z return mod(**inputs) 2025-08-14T21:50:29.2401238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2401308Z outputs = self.model( 2025-08-14T21:50:29.2401575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2401658Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2401961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2402045Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2402277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2402360Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2402634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2402729Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2402992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2403103Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2403107Z 2025-08-14T21:50:29.2403214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2403431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2403499Z return mod(**inputs) 2025-08-14T21:50:29.2403767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2403843Z outputs = self.model( 2025-08-14T21:50:29.2404109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2404180Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2404431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2404502Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2404729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2404817Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2405060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2405158Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2405397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2405487Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2405490Z 2025-08-14T21:50:29.2405568Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2405644Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2405725Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2405800Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2405899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2406129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2406192Z return mod(**inputs) 2025-08-14T21:50:29.2406453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2406521Z outputs = self.model( 2025-08-14T21:50:29.2406771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2406850Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2407098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2407166Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2407392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2407471Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2407725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2407848Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2408097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2408202Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2408489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2408627Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2408631Z 2025-08-14T21:50:29.2408912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2409116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2409235Z return mod(**inputs) 2025-08-14T21:50:29.2409491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2409562Z outputs = self.model( 2025-08-14T21:50:29.2409825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2409899Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2410158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2410229Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2410449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2410538Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2410788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2410879Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2411136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2411238Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2411537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2411650Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2411654Z 2025-08-14T21:50:29.2411757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2411976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2412043Z return mod(**inputs) 2025-08-14T21:50:29.2412306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2412403Z outputs = self.model( 2025-08-14T21:50:29.2412658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2412742Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2412990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2413061Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2413287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2413375Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2413627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2413715Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2413963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2414050Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2414077Z 2025-08-14T21:50:29.2414207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2414407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2414472Z return mod(**inputs) 2025-08-14T21:50:29.2414718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2414790Z outputs = self.model( 2025-08-14T21:50:29.2415034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2415104Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2415369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2415443Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2415662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2415741Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2415980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2416104Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2416108Z 2025-08-14T21:50:29.2416208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2416408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2416475Z return mod(**inputs) 2025-08-14T21:50:29.2416725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2416802Z outputs = self.model( 2025-08-14T21:50:29.2417052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2417128Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2417388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2417461Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2417687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2417766Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2418040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2418172Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2418405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2418501Z return self.act(input) 2025-08-14T21:50:29.2418511Z 2025-08-14T21:50:29.2418613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2418810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2418883Z return mod(**inputs) 2025-08-14T21:50:29.2419134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2419200Z outputs = self.model( 2025-08-14T21:50:29.2438722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2438957Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2439307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2439419Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2439687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2439908Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2440204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2440299Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2440307Z 2025-08-14T21:50:29.2440429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2440667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2440744Z return mod(**inputs) 2025-08-14T21:50:29.2441031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2441112Z outputs = self.model( 2025-08-14T21:50:29.2441436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2441530Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2441801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2441880Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2442126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2442213Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2442482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2442586Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2442850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2443027Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2443033Z 2025-08-14T21:50:29.2443147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2443376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2443451Z return mod(**inputs) 2025-08-14T21:50:29.2443720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2443805Z outputs = self.model( 2025-08-14T21:50:29.2444074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2444157Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2444432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2444540Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2444781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2444867Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2445131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2445238Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2445499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2445593Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2445597Z 2025-08-14T21:50:29.2445710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2445923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2446001Z return mod(**inputs) 2025-08-14T21:50:29.2446271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2446343Z outputs = self.model( 2025-08-14T21:50:29.2446649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2446728Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2446997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2447072Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2447302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2447390Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2447652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2447768Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2448039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2448134Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2448138Z 2025-08-14T21:50:29.2448233Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2448317Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2448398Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2448487Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2448596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2448810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2448887Z return mod(**inputs) 2025-08-14T21:50:29.2449156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2449239Z outputs = self.model( 2025-08-14T21:50:29.2449503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2449587Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2449858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2449935Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2450175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2450258Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2450521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2450621Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2450888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2451014Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2451336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2451482Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2451486Z 2025-08-14T21:50:29.2451603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2451815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2451885Z return mod(**inputs) 2025-08-14T21:50:29.2452162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2452234Z outputs = self.model( 2025-08-14T21:50:29.2452507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2452589Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2452867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2452972Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2453209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2453295Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2453577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2453675Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2453960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2454062Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2454399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2454529Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2454535Z 2025-08-14T21:50:29.2454642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2454858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2454928Z return mod(**inputs) 2025-08-14T21:50:29.2455195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2455269Z outputs = self.model( 2025-08-14T21:50:29.2455517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2455591Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2455848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2455920Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2456148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2456226Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2456473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2456572Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2456818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2456904Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2456915Z 2025-08-14T21:50:29.2457026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2457251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2457331Z return mod(**inputs) 2025-08-14T21:50:29.2457597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2457672Z outputs = self.model( 2025-08-14T21:50:29.2457940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2458016Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2458287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2458362Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2458590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2458681Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2458943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2459074Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2459102Z 2025-08-14T21:50:29.2459256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2459467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2459546Z return mod(**inputs) 2025-08-14T21:50:29.2459806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2459878Z outputs = self.model( 2025-08-14T21:50:29.2460152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2460228Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2460515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2460596Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2460827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2460918Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2461179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2461304Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2461537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2461611Z return self.act(input) 2025-08-14T21:50:29.2461615Z 2025-08-14T21:50:29.2461730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2461934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2462006Z return mod(**inputs) 2025-08-14T21:50:29.2462276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2462353Z outputs = self.model( 2025-08-14T21:50:29.2462615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2462701Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2462961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2463043Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2463270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2463351Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2463622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2463725Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2463729Z 2025-08-14T21:50:29.2463845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2464056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2464125Z return mod(**inputs) 2025-08-14T21:50:29.2464396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2464469Z outputs = self.model( 2025-08-14T21:50:29.2464732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2464815Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2465076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2465162Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2465393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2465507Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2465777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:50:29.2465862Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2465866Z 2025-08-14T21:50:29.2465980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2466190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2466259Z return mod(**inputs) 2025-08-14T21:50:29.2466527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2466601Z outputs = self.model( 2025-08-14T21:50:29.2466882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2466972Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2467240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2467323Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2467555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2467635Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2467905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2468000Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2468264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2468436Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2468440Z 2025-08-14T21:50:29.2468550Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2468770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2468840Z return mod(**inputs) 2025-08-14T21:50:29.2469103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2469183Z outputs = self.model( 2025-08-14T21:50:29.2469446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2469529Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2469794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2469887Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2470129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2470213Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2470477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2470580Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2470844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2470933Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2470937Z 2025-08-14T21:50:29.2471044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2471249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2471326Z return mod(**inputs) 2025-08-14T21:50:29.2471594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2471676Z outputs = self.model( 2025-08-14T21:50:29.2471973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2472050Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2472326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2472402Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2472641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2472732Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2472994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2473112Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2473384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2473480Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2473484Z 2025-08-14T21:50:29.2473582Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2473671Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2473755Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2473846Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2473958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2474185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2474257Z return mod(**inputs) 2025-08-14T21:50:29.2474542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2474629Z outputs = self.model( 2025-08-14T21:50:29.2474916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2474998Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2475279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2475354Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2475602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2475781Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2476061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2476166Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2476439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2476574Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2476898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2477046Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2477050Z 2025-08-14T21:50:29.2477168Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2477384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2477459Z return mod(**inputs) 2025-08-14T21:50:29.2477753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2477825Z outputs = self.model( 2025-08-14T21:50:29.2478115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2478195Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2478474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2478578Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2478830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2478913Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2479179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2479283Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2479549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2479651Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2479997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2480117Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2480122Z 2025-08-14T21:50:29.2480237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2480445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2480514Z return mod(**inputs) 2025-08-14T21:50:29.2480841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2480914Z outputs = self.model( 2025-08-14T21:50:29.2481193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2481268Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2481529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2481611Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2481842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2481925Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2482191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2482284Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2482551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2482636Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2482640Z 2025-08-14T21:50:29.2482747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2482985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2483054Z return mod(**inputs) 2025-08-14T21:50:29.2483327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2483402Z outputs = self.model( 2025-08-14T21:50:29.2483664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2483744Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2484004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2484078Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2484317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2484400Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2484672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2484797Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2484824Z 2025-08-14T21:50:29.2484947Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2485163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2485232Z return mod(**inputs) 2025-08-14T21:50:29.2485497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2485575Z outputs = self.model( 2025-08-14T21:50:29.2485838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2485922Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2486200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2486279Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2486517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2486601Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2486872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2486997Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2487221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2487304Z return self.act(input) 2025-08-14T21:50:29.2487308Z 2025-08-14T21:50:29.2487414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2487624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2487702Z return mod(**inputs) 2025-08-14T21:50:29.2487976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2488053Z outputs = self.model( 2025-08-14T21:50:29.2488303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2488376Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2488636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2488709Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2488929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2489014Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2489280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2489431Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2489435Z 2025-08-14T21:50:29.2489545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2489758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2489835Z return mod(**inputs) 2025-08-14T21:50:29.2490102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2490181Z outputs = self.model( 2025-08-14T21:50:29.2490450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2490526Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2490800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2490878Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2491111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2491235Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2491496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2491599Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2491858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2492017Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2492021Z 2025-08-14T21:50:29.2492136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2492344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2492437Z return mod(**inputs) 2025-08-14T21:50:29.2492701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2492776Z outputs = self.model( 2025-08-14T21:50:29.2493046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2493123Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2493384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2493467Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2493705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2493793Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2494061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2494152Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2494405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2494484Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2494488Z 2025-08-14T21:50:29.2494599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2494806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2494876Z return mod(**inputs) 2025-08-14T21:50:29.2495153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2495224Z outputs = self.model( 2025-08-14T21:50:29.2495499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2495605Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2495864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2495950Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2496179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2496261Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2496539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2496626Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2496872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2496963Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2496967Z 2025-08-14T21:50:29.2497050Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2497134Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2497210Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2497303Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2497430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2497629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2497693Z return mod(**inputs) 2025-08-14T21:50:29.2497956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2498023Z outputs = self.model( 2025-08-14T21:50:29.2498284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2498357Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2498622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2498703Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2498922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2499009Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2499257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2499346Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2499600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2499699Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2499987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2500129Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2500133Z 2025-08-14T21:50:29.2500233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2500439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2500505Z return mod(**inputs) 2025-08-14T21:50:29.2500776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2500855Z outputs = self.model( 2025-08-14T21:50:29.2501131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2501215Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2501477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2501552Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2501808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2501890Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2502157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2502259Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2502521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2502628Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2502939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2503055Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2503059Z 2025-08-14T21:50:29.2503173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2503385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2503463Z return mod(**inputs) 2025-08-14T21:50:29.2503760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2503834Z outputs = self.model( 2025-08-14T21:50:29.2504105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2504182Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2504445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2504529Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2504759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2504883Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2505149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:50:29.2505248Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:50:29.2505517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2505602Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2505606Z 2025-08-14T21:50:29.2505714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2505930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2505999Z return mod(**inputs) 2025-08-14T21:50:29.2506270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2506339Z outputs = self.model( 2025-08-14T21:50:29.2506608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2506693Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2506960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2507044Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2507275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2507356Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2507625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2507750Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2507754Z 2025-08-14T21:50:29.2507861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2508105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2508178Z return mod(**inputs) 2025-08-14T21:50:29.2508457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2508531Z outputs = self.model( 2025-08-14T21:50:29.2508963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2509058Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2509322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2509405Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2509636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2509721Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2509999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:50:29.2510177Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2510424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2510508Z return self.act(input) 2025-08-14T21:50:29.2510512Z 2025-08-14T21:50:29.2510620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2510834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2510905Z return mod(**inputs) 2025-08-14T21:50:29.2511170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2511249Z outputs = self.model( 2025-08-14T21:50:29.2511540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2511621Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2511891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2511969Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2512204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2512288Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2512555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:50:29.2512650Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2512654Z 2025-08-14T21:50:29.2512763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2512984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2513058Z return mod(**inputs) 2025-08-14T21:50:29.2513327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2513410Z outputs = self.model( 2025-08-14T21:50:29.2513682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:50:29.2513760Z encoder_outputs = self.encoder( 2025-08-14T21:50:29.2514032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:50:29.2514109Z layer_outputs = encoder_layer( 2025-08-14T21:50:29.2514351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2514436Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2514709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:50:29.2514832Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2514836Z 2025-08-14T21:50:29.2514948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2515169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2515240Z return mod(**inputs) 2025-08-14T21:50:29.2515523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2515600Z outputs = self.model( 2025-08-14T21:50:29.2516169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2516254Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2516544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2516628Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2516874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2517000Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2517277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2517395Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2517674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2517835Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2517847Z 2025-08-14T21:50:29.2517956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2518162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2518257Z return mod(**inputs) 2025-08-14T21:50:29.2518533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2518607Z outputs = self.model( 2025-08-14T21:50:29.2518895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2518971Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2519304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2519379Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2519616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2519706Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2519967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2520070Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2520328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2520410Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2520413Z 2025-08-14T21:50:29.2520523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2520718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2520784Z return mod(**inputs) 2025-08-14T21:50:29.2521042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2521109Z outputs = self.model( 2025-08-14T21:50:29.2521366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2521458Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2521709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2521795Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2522027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2522110Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2522388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2522486Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2522744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2522828Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2522833Z 2025-08-14T21:50:29.2522914Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2523001Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2523076Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2523171Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2523297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2523494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2523566Z return mod(**inputs) 2025-08-14T21:50:29.2523822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2523886Z outputs = self.model( 2025-08-14T21:50:29.2524135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2524205Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2524472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2524550Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2524760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2524842Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2525082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2525175Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2525423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2525527Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2525842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2525985Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2525989Z 2025-08-14T21:50:29.2526102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2526306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2526373Z return mod(**inputs) 2025-08-14T21:50:29.2526623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2526700Z outputs = self.model( 2025-08-14T21:50:29.2526946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2527025Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2527273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2527356Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2527594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2527669Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2527927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2528022Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2528264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2528364Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2528651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2528758Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2528762Z 2025-08-14T21:50:29.2528873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2529066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2529151Z return mod(**inputs) 2025-08-14T21:50:29.2529416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2529485Z outputs = self.model( 2025-08-14T21:50:29.2529739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2529810Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2530062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2530132Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2530346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2530470Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2530708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2530806Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2531050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2531128Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2531132Z 2025-08-14T21:50:29.2531236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2531425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2531488Z return mod(**inputs) 2025-08-14T21:50:29.2531736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2531804Z outputs = self.model( 2025-08-14T21:50:29.2532045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2532122Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2532364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2532440Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2532649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2532726Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2532974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2533080Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2533330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2533489Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2533494Z 2025-08-14T21:50:29.2533602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2533818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2533890Z return mod(**inputs) 2025-08-14T21:50:29.2534152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2534240Z outputs = self.model( 2025-08-14T21:50:29.2534486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2534566Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2534812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2534888Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2535111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2535616Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2535898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2536013Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2536276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2536367Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2536371Z 2025-08-14T21:50:29.2536474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2536669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2536762Z return mod(**inputs) 2025-08-14T21:50:29.2537011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2537089Z outputs = self.model( 2025-08-14T21:50:29.2537337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2537410Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2537679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2537756Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2537985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2538076Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2538338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2538460Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2538722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2538814Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2538818Z 2025-08-14T21:50:29.2538911Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2538993Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2539083Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2539161Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2539269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2539484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2539550Z return mod(**inputs) 2025-08-14T21:50:29.2539800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2539893Z outputs = self.model( 2025-08-14T21:50:29.2540138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2540223Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2540471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2540541Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2540763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2540840Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2541094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2541199Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2541449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2541554Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2541869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2542002Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2542014Z 2025-08-14T21:50:29.2542116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2542314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2542388Z return mod(**inputs) 2025-08-14T21:50:29.2542637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2542703Z outputs = self.model( 2025-08-14T21:50:29.2542976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2543049Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2543306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2543377Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2543592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2543679Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2543931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2544040Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2544301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2544403Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2544702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2544815Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2544819Z 2025-08-14T21:50:29.2544924Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2545132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2545200Z return mod(**inputs) 2025-08-14T21:50:29.2545465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2545536Z outputs = self.model( 2025-08-14T21:50:29.2545796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2545900Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2546160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2546237Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2546474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2546555Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2546833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2546942Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2547196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2547287Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2547290Z 2025-08-14T21:50:29.2547403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2547617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2547704Z return mod(**inputs) 2025-08-14T21:50:29.2547983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2548064Z outputs = self.model( 2025-08-14T21:50:29.2548330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2548408Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2548678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2548754Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2548989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2549089Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2549361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2549494Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2549498Z 2025-08-14T21:50:29.2549602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2549806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2549880Z return mod(**inputs) 2025-08-14T21:50:29.2550138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2550214Z outputs = self.model( 2025-08-14T21:50:29.2550470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2550546Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2550808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2550882Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2551110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2551191Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2551452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2551582Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2551805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2551878Z return self.act(input) 2025-08-14T21:50:29.2551882Z 2025-08-14T21:50:29.2551995Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2552221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2552296Z return mod(**inputs) 2025-08-14T21:50:29.2552559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2552628Z outputs = self.model( 2025-08-14T21:50:29.2552894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2552970Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2553230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2553311Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2553537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2553624Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2553892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2553995Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2553999Z 2025-08-14T21:50:29.2554139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2554355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2554432Z return mod(**inputs) 2025-08-14T21:50:29.2554701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2554773Z outputs = self.model( 2025-08-14T21:50:29.2555049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2555127Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2555415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2555504Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2555828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2555931Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2556204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2556311Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2556590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2556754Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2556759Z 2025-08-14T21:50:29.2556887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2557088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2557155Z return mod(**inputs) 2025-08-14T21:50:29.2557411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2557482Z outputs = self.model( 2025-08-14T21:50:29.2557730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2557811Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2558059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2558141Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2558356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2558434Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2558710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2558809Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2559067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2559145Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2559149Z 2025-08-14T21:50:29.2559249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2559451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2559516Z return mod(**inputs) 2025-08-14T21:50:29.2559761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2559836Z outputs = self.model( 2025-08-14T21:50:29.2560084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2560164Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2560450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2560522Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2560745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2560822Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2561070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2561174Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2561422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2561534Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2561538Z 2025-08-14T21:50:29.2561620Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2561699Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2561788Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2561865Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2561982Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2562185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2562250Z return mod(**inputs) 2025-08-14T21:50:29.2562520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2562591Z outputs = self.model( 2025-08-14T21:50:29.2562865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2562951Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2563222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2563307Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2563543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2563627Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2563905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2564007Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2564280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2564382Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2564695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2564860Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2564866Z 2025-08-14T21:50:29.2564974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2565180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2565257Z return mod(**inputs) 2025-08-14T21:50:29.2565520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2565598Z outputs = self.model( 2025-08-14T21:50:29.2565863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2565940Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2566207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2566285Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2566517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2566648Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2566913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2567021Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2567283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2567383Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2567694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2567807Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2567829Z 2025-08-14T21:50:29.2567944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2568152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2568227Z return mod(**inputs) 2025-08-14T21:50:29.2568500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2568571Z outputs = self.model( 2025-08-14T21:50:29.2568840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2568914Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2569177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2569259Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2569491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2569574Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2569848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2569949Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2570218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2570304Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2570308Z 2025-08-14T21:50:29.2570415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2570632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2570702Z return mod(**inputs) 2025-08-14T21:50:29.2570974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2571062Z outputs = self.model( 2025-08-14T21:50:29.2571333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2571422Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2571693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2571768Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2572010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2572093Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2572366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-08-14T21:50:29.2572450Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2572455Z 2025-08-14T21:50:29.2572563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2572781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2572867Z return mod(**inputs) 2025-08-14T21:50:29.2573152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2573232Z outputs = self.model( 2025-08-14T21:50:29.2573498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2573580Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2573842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2573916Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2574168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2574252Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2574521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2574638Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2574898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2575061Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2575065Z 2025-08-14T21:50:29.2575172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2575380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2575457Z return mod(**inputs) 2025-08-14T21:50:29.2575720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2575800Z outputs = self.model( 2025-08-14T21:50:29.2576060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2576139Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2576409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2576484Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2576721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2576804Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2577070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2577189Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2577469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2577551Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2577557Z 2025-08-14T21:50:29.2577675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2577878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2577954Z return mod(**inputs) 2025-08-14T21:50:29.2578214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2578283Z outputs = self.model( 2025-08-14T21:50:29.2578553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2578630Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2578893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2578977Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2579206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2579339Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2579607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2579720Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2579997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2580088Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2580091Z 2025-08-14T21:50:29.2580183Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2580267Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2580347Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2580452Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2580561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2580770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2580847Z return mod(**inputs) 2025-08-14T21:50:29.2581112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2581190Z outputs = self.model( 2025-08-14T21:50:29.2581451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2581526Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2581795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2581871Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2582100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2582191Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2582455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2582583Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2582828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2582924Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2583218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2583349Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2583352Z 2025-08-14T21:50:29.2583476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2583671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2583738Z return mod(**inputs) 2025-08-14T21:50:29.2584008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2584080Z outputs = self.model( 2025-08-14T21:50:29.2584339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2584422Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2584683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2584766Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2584991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2585075Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2585341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2585487Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2585756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2585860Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2586144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2586257Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2586261Z 2025-08-14T21:50:29.2586361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2586555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2586671Z return mod(**inputs) 2025-08-14T21:50:29.2586924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2587001Z outputs = self.model( 2025-08-14T21:50:29.2587252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2587323Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2587583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2587656Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2587875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2587961Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2588211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2588327Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2588579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2588662Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2588665Z 2025-08-14T21:50:29.2588776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2588972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2589044Z return mod(**inputs) 2025-08-14T21:50:29.2589295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2589364Z outputs = self.model( 2025-08-14T21:50:29.2589621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2589711Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2589964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2590047Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2590266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2590352Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2590604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2590721Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2590725Z 2025-08-14T21:50:29.2590833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2591031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2591107Z return mod(**inputs) 2025-08-14T21:50:29.2591360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2591441Z outputs = self.model( 2025-08-14T21:50:29.2591712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2591784Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2592031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2592110Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2592330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2592420Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2592696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2592821Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2593053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2593129Z return self.act(input) 2025-08-14T21:50:29.2593133Z 2025-08-14T21:50:29.2593248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2593457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2593525Z return mod(**inputs) 2025-08-14T21:50:29.2593799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2593870Z outputs = self.model( 2025-08-14T21:50:29.2594136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2594224Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2594492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2594576Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2594811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2594892Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2595169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2595253Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2595257Z 2025-08-14T21:50:29.2595364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2595587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2595657Z return mod(**inputs) 2025-08-14T21:50:29.2596032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2596112Z outputs = self.model( 2025-08-14T21:50:29.2596404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2596491Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2596763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2596855Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2597096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2597177Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2597439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2597547Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2597802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2597998Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2598002Z 2025-08-14T21:50:29.2598109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2598322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2598390Z return mod(**inputs) 2025-08-14T21:50:29.2598668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2598747Z outputs = self.model( 2025-08-14T21:50:29.2599021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2599102Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2599383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2599462Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2599706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2599788Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2600050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2600171Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2600433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2600521Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2600524Z 2025-08-14T21:50:29.2600629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2600833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2600906Z return mod(**inputs) 2025-08-14T21:50:29.2601164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2601234Z outputs = self.model( 2025-08-14T21:50:29.2601513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2601585Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2601885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2601959Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2602198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2602316Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2602578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2602690Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2602950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2603040Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2603043Z 2025-08-14T21:50:29.2603135Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2603219Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2603299Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2603389Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2603496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2603711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2603783Z return mod(**inputs) 2025-08-14T21:50:29.2604066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2604163Z outputs = self.model( 2025-08-14T21:50:29.2604452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2604530Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2604811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2604887Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2605128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2605211Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2605492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2605609Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2605878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2605983Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2606297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2606438Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2606442Z 2025-08-14T21:50:29.2606557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2606773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2606842Z return mod(**inputs) 2025-08-14T21:50:29.2607116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2607189Z outputs = self.model( 2025-08-14T21:50:29.2607463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2607544Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2607813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2607896Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2608127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2608211Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2608485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2608589Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2609059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2609167Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2609476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2609600Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2609604Z 2025-08-14T21:50:29.2609711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2609926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2609995Z return mod(**inputs) 2025-08-14T21:50:29.2610255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2610336Z outputs = self.model( 2025-08-14T21:50:29.2610602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2610679Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2611019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2611096Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2611331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2611414Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2611676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2611786Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2612047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2612172Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2612176Z 2025-08-14T21:50:29.2612286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2612499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2612580Z return mod(**inputs) 2025-08-14T21:50:29.2612849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2612922Z outputs = self.model( 2025-08-14T21:50:29.2613195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2613271Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2613542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2613616Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2613848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2613939Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2614208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2614330Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2614590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2614747Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2614751Z 2025-08-14T21:50:29.2614865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2615074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2615141Z return mod(**inputs) 2025-08-14T21:50:29.2615438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2615508Z outputs = self.model( 2025-08-14T21:50:29.2615781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2615856Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2616114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2616196Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2616420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2616501Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2616768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2616882Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2617148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2617264Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2617268Z 2025-08-14T21:50:29.2617375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2617587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2617656Z return mod(**inputs) 2025-08-14T21:50:29.2617931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2618001Z outputs = self.model( 2025-08-14T21:50:29.2618265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2618348Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2618631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2618709Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2618956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2619034Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2619288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2619394Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2619640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2619733Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2619737Z 2025-08-14T21:50:29.2619816Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2619904Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2619981Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2620057Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2620170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2620372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2620444Z return mod(**inputs) 2025-08-14T21:50:29.2620718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2620787Z outputs = self.model( 2025-08-14T21:50:29.2621058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2621135Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2621400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2621503Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2621733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2621818Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2622089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2622202Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2622468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2622571Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2622874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2623023Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2623030Z 2025-08-14T21:50:29.2623131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2623335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2623433Z return mod(**inputs) 2025-08-14T21:50:29.2623691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2623777Z outputs = self.model( 2025-08-14T21:50:29.2624041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2624117Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2624387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2624461Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2624729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2624814Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2625080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2625201Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2625465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2625569Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2625879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2625990Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2625994Z 2025-08-14T21:50:29.2626108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2626318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2626389Z return mod(**inputs) 2025-08-14T21:50:29.2626666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2626738Z outputs = self.model( 2025-08-14T21:50:29.2627008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2627086Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2627349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2627430Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2627661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2627745Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2628034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2628147Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2628420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2628505Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2628509Z 2025-08-14T21:50:29.2628616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2628831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2628901Z return mod(**inputs) 2025-08-14T21:50:29.2629170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2629242Z outputs = self.model( 2025-08-14T21:50:29.2629507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2629594Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2629891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2629968Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2630204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2630285Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2630557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-08-14T21:50:29.2630644Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2630648Z 2025-08-14T21:50:29.2630757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2630996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2631069Z return mod(**inputs) 2025-08-14T21:50:29.2631352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2631428Z outputs = self.model( 2025-08-14T21:50:29.2631702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2631786Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2632082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2632157Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2632402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2632485Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2632769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2632896Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2632902Z 2025-08-14T21:50:29.2633012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2633235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2633304Z return mod(**inputs) 2025-08-14T21:50:29.2633579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2633659Z outputs = self.model( 2025-08-14T21:50:29.2633933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2634021Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2634320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2634414Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2634663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2634751Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2635034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2635163Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2635398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2635482Z return self.act(input) 2025-08-14T21:50:29.2635486Z 2025-08-14T21:50:29.2635597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2635883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2635970Z return mod(**inputs) 2025-08-14T21:50:29.2636244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2636346Z outputs = self.model( 2025-08-14T21:50:29.2636634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2636716Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2637004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2637082Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2637334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2637428Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2637722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2637825Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2637829Z 2025-08-14T21:50:29.2637944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2638159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2638240Z return mod(**inputs) 2025-08-14T21:50:29.2638521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2638603Z outputs = self.model( 2025-08-14T21:50:29.2638886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2638963Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2639256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2639334Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2639577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2639671Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2639950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2640066Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2640342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2640506Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2640510Z 2025-08-14T21:50:29.2640629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2640849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2640947Z return mod(**inputs) 2025-08-14T21:50:29.2641233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2641308Z outputs = self.model( 2025-08-14T21:50:29.2641593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2641671Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2641952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2642036Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2642278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2642369Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2642642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2642751Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2643043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2643146Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2643150Z 2025-08-14T21:50:29.2643267Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2643480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2643552Z return mod(**inputs) 2025-08-14T21:50:29.2643834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2643908Z outputs = self.model( 2025-08-14T21:50:29.2644189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2644293Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2644593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2644680Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2644907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2644989Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2645259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2645361Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2645621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2645716Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2645720Z 2025-08-14T21:50:29.2645806Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2645897Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2645978Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2646059Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2646174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2646378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2646446Z return mod(**inputs) 2025-08-14T21:50:29.2646723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2646792Z outputs = self.model( 2025-08-14T21:50:29.2647068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2647144Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2647418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2647518Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2647747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2647838Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2648097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2648200Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2648473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2648575Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2648879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2649031Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2649035Z 2025-08-14T21:50:29.2649141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2649396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2649468Z return mod(**inputs) 2025-08-14T21:50:29.2649730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2649809Z outputs = self.model( 2025-08-14T21:50:29.2650071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2650153Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2650417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2650491Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2650742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2650825Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2651091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2651199Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2651463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2651571Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2651874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2651988Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2651991Z 2025-08-14T21:50:29.2652104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2652316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2652393Z return mod(**inputs) 2025-08-14T21:50:29.2652661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2652732Z outputs = self.model( 2025-08-14T21:50:29.2653001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2653080Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2653339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2653426Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2653654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2653763Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2654026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2654132Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2654401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2654485Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2654489Z 2025-08-14T21:50:29.2654600Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2654808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2654877Z return mod(**inputs) 2025-08-14T21:50:29.2655150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2655223Z outputs = self.model( 2025-08-14T21:50:29.2655490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2655591Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2655868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2655953Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2656182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2656263Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2656530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2656642Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2656904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2657086Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2657090Z 2025-08-14T21:50:29.2657199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2657415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2657485Z return mod(**inputs) 2025-08-14T21:50:29.2657745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2657824Z outputs = self.model( 2025-08-14T21:50:29.2658085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2658170Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2658432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2658510Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2658750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2658838Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2659101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2659222Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2659483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2659574Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2659577Z 2025-08-14T21:50:29.2659687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2659895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2659989Z return mod(**inputs) 2025-08-14T21:50:29.2660253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2660333Z outputs = self.model( 2025-08-14T21:50:29.2660597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2660675Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2660946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2661022Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2661250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2661339Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2661600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2661721Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2661981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2662103Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2662107Z 2025-08-14T21:50:29.2662199Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2662280Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2662362Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2662446Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2662552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2662769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2662837Z return mod(**inputs) 2025-08-14T21:50:29.2663123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2663203Z outputs = self.model( 2025-08-14T21:50:29.2663466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2663546Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2663817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2663892Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2664130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2664213Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2664477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2664596Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2664861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2664974Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2665280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2665419Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2665423Z 2025-08-14T21:50:29.2665537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2665742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2665811Z return mod(**inputs) 2025-08-14T21:50:29.2666081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2666154Z outputs = self.model( 2025-08-14T21:50:29.2666448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2666528Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2666794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2666879Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2667106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2667196Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2667456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2667566Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2667835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2667942Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2668244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2668398Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2668402Z 2025-08-14T21:50:29.2668510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2668725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2668795Z return mod(**inputs) 2025-08-14T21:50:29.2669058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2669138Z outputs = self.model( 2025-08-14T21:50:29.2669400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2669485Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2669763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2669842Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2670079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2670162Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2670425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2670544Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2670804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2670896Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2670899Z 2025-08-14T21:50:29.2671006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2671216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2671294Z return mod(**inputs) 2025-08-14T21:50:29.2671562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2671632Z outputs = self.model( 2025-08-14T21:50:29.2671902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2671977Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2672246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2672320Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2672546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2672658Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2672923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2673059Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2673063Z 2025-08-14T21:50:29.2673174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2673385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2673465Z return mod(**inputs) 2025-08-14T21:50:29.2673730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2673804Z outputs = self.model( 2025-08-14T21:50:29.2674077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2674154Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2674433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2674538Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2674786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2674879Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2675145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2675279Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2675511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2675588Z return self.act(input) 2025-08-14T21:50:29.2675592Z 2025-08-14T21:50:29.2675800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2676049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2676125Z return mod(**inputs) 2025-08-14T21:50:29.2676411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2676488Z outputs = self.model( 2025-08-14T21:50:29.2676770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2676849Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2677150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2677234Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2677465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2677548Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2677832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2677922Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2677928Z 2025-08-14T21:50:29.2678050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2678267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2678340Z return mod(**inputs) 2025-08-14T21:50:29.2678625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2678697Z outputs = self.model( 2025-08-14T21:50:29.2678981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2679061Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2679335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2679437Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2679687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2679774Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2680059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:50:29.2680144Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2680148Z 2025-08-14T21:50:29.2680266Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2680482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2680554Z return mod(**inputs) 2025-08-14T21:50:29.2680837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2680912Z outputs = self.model( 2025-08-14T21:50:29.2681209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2681329Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2681602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2681684Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2681928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2682012Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2682289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2682397Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2682690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2682856Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2682862Z 2025-08-14T21:50:29.2682977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2683200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2683271Z return mod(**inputs) 2025-08-14T21:50:29.2683562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2683634Z outputs = self.model( 2025-08-14T21:50:29.2683918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2684003Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2684291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2684370Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2684664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2684755Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2685034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2685142Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2685404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2685492Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2685496Z 2025-08-14T21:50:29.2685598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2685795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2685887Z return mod(**inputs) 2025-08-14T21:50:29.2686167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2686247Z outputs = self.model( 2025-08-14T21:50:29.2686530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2686605Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2686903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2686978Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2687221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2687302Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2687567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2687678Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2688002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2688088Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2688100Z 2025-08-14T21:50:29.2688180Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2688258Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2688341Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2688416Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2688517Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2688721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2688786Z return mod(**inputs) 2025-08-14T21:50:29.2689050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2689132Z outputs = self.model( 2025-08-14T21:50:29.2689384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2689463Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2689733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2689808Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2690050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2690132Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2690396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2690508Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2690774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2690884Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2691195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2691334Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2691337Z 2025-08-14T21:50:29.2691453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2691671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2691743Z return mod(**inputs) 2025-08-14T21:50:29.2691992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2692058Z outputs = self.model( 2025-08-14T21:50:29.2692334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2692409Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2692657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2692736Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2692954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2693044Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2693306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2693408Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2693675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2693781Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2694110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2694257Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2694261Z 2025-08-14T21:50:29.2694367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2694586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2694655Z return mod(**inputs) 2025-08-14T21:50:29.2694927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2695003Z outputs = self.model( 2025-08-14T21:50:29.2695253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2695347Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2695597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2695672Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2695897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2695973Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2696230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2696329Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2696576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2696665Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2696670Z 2025-08-14T21:50:29.2696772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2696967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2697041Z return mod(**inputs) 2025-08-14T21:50:29.2697299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2697378Z outputs = self.model( 2025-08-14T21:50:29.2697641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2697717Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2697990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2698066Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2698293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2698412Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2698677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2698801Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2699077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2699223Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2699227Z 2025-08-14T21:50:29.2699336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2699534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2699608Z return mod(**inputs) 2025-08-14T21:50:29.2699861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2699932Z outputs = self.model( 2025-08-14T21:50:29.2700189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2700294Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2700543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2700623Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2700837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2700921Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2701169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2701274Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2701549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2701634Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2701640Z 2025-08-14T21:50:29.2701755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2701960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2702028Z return mod(**inputs) 2025-08-14T21:50:29.2702298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2702368Z outputs = self.model( 2025-08-14T21:50:29.2702632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2702717Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2702976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2703058Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2703278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2703357Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2703613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2703717Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2703982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2704072Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2704076Z 2025-08-14T21:50:29.2704161Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2704252Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2704333Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2704431Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2704547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2704762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2704838Z return mod(**inputs) 2025-08-14T21:50:29.2705106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2705176Z outputs = self.model( 2025-08-14T21:50:29.2705452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2705528Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2705793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2705874Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2706112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2706200Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2706498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2706611Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2706880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2706983Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2707288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2707434Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2707437Z 2025-08-14T21:50:29.2707544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2707779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2707849Z return mod(**inputs) 2025-08-14T21:50:29.2708118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2708198Z outputs = self.model( 2025-08-14T21:50:29.2708460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2708542Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2709069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2709152Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2709393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2709480Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2709746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2709870Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2710133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2710243Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2710549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2710662Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2710666Z 2025-08-14T21:50:29.2710782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2710991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2711113Z return mod(**inputs) 2025-08-14T21:50:29.2711377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2711450Z outputs = self.model( 2025-08-14T21:50:29.2711720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2711798Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2712063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2712147Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2712375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2712464Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2712726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2712840Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2713135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2713247Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2713251Z 2025-08-14T21:50:29.2713367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2713578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2713648Z return mod(**inputs) 2025-08-14T21:50:29.2713920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2713991Z outputs = self.model( 2025-08-14T21:50:29.2714254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2714366Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2714640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2714728Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2714969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2715052Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2715333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2715461Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2715465Z 2025-08-14T21:50:29.2715582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2715856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2715938Z return mod(**inputs) 2025-08-14T21:50:29.2716215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2716291Z outputs = self.model( 2025-08-14T21:50:29.2716560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2716647Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2716916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2717011Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2717238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2717321Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2717585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2717732Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2717952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2718038Z return self.act(input) 2025-08-14T21:50:29.2718042Z 2025-08-14T21:50:29.2718149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2718366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2718436Z return mod(**inputs) 2025-08-14T21:50:29.2718699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2718778Z outputs = self.model( 2025-08-14T21:50:29.2719040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2719125Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2719391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2719489Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2719752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2719831Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2720077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2720167Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2720171Z 2025-08-14T21:50:29.2720274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2720478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2720543Z return mod(**inputs) 2025-08-14T21:50:29.2720821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2720900Z outputs = self.model( 2025-08-14T21:50:29.2721151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2721223Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2721519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2721594Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2721833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2721915Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2722176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2722289Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2722553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2722722Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2722726Z 2025-08-14T21:50:29.2722835Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2723045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2723120Z return mod(**inputs) 2025-08-14T21:50:29.2723396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2723466Z outputs = self.model( 2025-08-14T21:50:29.2723747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2723821Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2724115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2724191Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2724421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2724511Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2724773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2724883Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2725142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2725226Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2725229Z 2025-08-14T21:50:29.2725343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2725551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2725620Z return mod(**inputs) 2025-08-14T21:50:29.2725939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2726013Z outputs = self.model( 2025-08-14T21:50:29.2726294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2726368Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2726641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2726721Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2726941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2727027Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2727296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2727397Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2727654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2727739Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2727742Z 2025-08-14T21:50:29.2727821Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2727908Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2727984Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2728065Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2728167Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2728364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2728438Z return mod(**inputs) 2025-08-14T21:50:29.2728690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2728758Z outputs = self.model( 2025-08-14T21:50:29.2729017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2729090Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2729346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2729419Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2729636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2729728Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2729989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2730110Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2730382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2730490Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2730802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2730950Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2730954Z 2025-08-14T21:50:29.2731054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2731259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2731325Z return mod(**inputs) 2025-08-14T21:50:29.2731585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2731655Z outputs = self.model( 2025-08-14T21:50:29.2731905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2732022Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2732273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2732346Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2732570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2732648Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2732921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2733025Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2733308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2733429Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2733720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2733838Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2733841Z 2025-08-14T21:50:29.2733963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2734176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2734254Z return mod(**inputs) 2025-08-14T21:50:29.2734519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2734593Z outputs = self.model( 2025-08-14T21:50:29.2734869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2734949Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2735227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2735304Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2735539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2735633Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2735897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2736011Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2736275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2736379Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2736382Z 2025-08-14T21:50:29.2736498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2736712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2736783Z return mod(**inputs) 2025-08-14T21:50:29.2737053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2737124Z outputs = self.model( 2025-08-14T21:50:29.2737390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2737466Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2737729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2737811Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2738047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2738130Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2738435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-08-14T21:50:29.2738520Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2738524Z 2025-08-14T21:50:29.2738639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2738849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2738916Z return mod(**inputs) 2025-08-14T21:50:29.2739190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2739260Z outputs = self.model( 2025-08-14T21:50:29.2739548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2739627Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2739893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2739980Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2740210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2740291Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2740563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2740678Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2740949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2741106Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2741113Z 2025-08-14T21:50:29.2741219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2741437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2741507Z return mod(**inputs) 2025-08-14T21:50:29.2741782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2741854Z outputs = self.model( 2025-08-14T21:50:29.2742120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2742203Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2742471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2742547Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2742804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2742885Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2743161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2743275Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2743539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2743630Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2743634Z 2025-08-14T21:50:29.2743740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2743958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2744027Z return mod(**inputs) 2025-08-14T21:50:29.2744302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2744382Z outputs = self.model( 2025-08-14T21:50:29.2744670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2744778Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2745068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2745144Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2745393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2745474Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2745745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2745863Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2746149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2746243Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2746255Z 2025-08-14T21:50:29.2746341Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2746421Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2746509Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2746588Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2746696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2746909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2746978Z return mod(**inputs) 2025-08-14T21:50:29.2747254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2747332Z outputs = self.model( 2025-08-14T21:50:29.2747620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2747704Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2747992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2748067Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2748313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2748396Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2748668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2748780Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2749044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2749174Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2749488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2749630Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2749642Z 2025-08-14T21:50:29.2749750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2749957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2750036Z return mod(**inputs) 2025-08-14T21:50:29.2750314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2750386Z outputs = self.model( 2025-08-14T21:50:29.2750662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2750742Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2751033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2751140Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2751377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2751469Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2751731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2751842Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2752112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2752216Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2752546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2752660Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2752666Z 2025-08-14T21:50:29.2752776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2752990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2753059Z return mod(**inputs) 2025-08-14T21:50:29.2753330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2753400Z outputs = self.model( 2025-08-14T21:50:29.2753661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2753746Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2754009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2754084Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2754324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2754407Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2754674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2754785Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2755047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2755139Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2755143Z 2025-08-14T21:50:29.2755249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2755473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2755562Z return mod(**inputs) 2025-08-14T21:50:29.2755919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2756011Z outputs = self.model( 2025-08-14T21:50:29.2756281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2756359Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2756639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2756716Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2756958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2757041Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2757309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2757456Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2757482Z 2025-08-14T21:50:29.2757608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2757817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2757897Z return mod(**inputs) 2025-08-14T21:50:29.2758166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2758248Z outputs = self.model( 2025-08-14T21:50:29.2758518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2758596Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2758898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2758979Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2759223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2759310Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2759583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2759717Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2759952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2760027Z return self.act(input) 2025-08-14T21:50:29.2760031Z 2025-08-14T21:50:29.2760147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2760361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2760445Z return mod(**inputs) 2025-08-14T21:50:29.2760715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2760790Z outputs = self.model( 2025-08-14T21:50:29.2761072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2761153Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2761427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2761514Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2761747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2761839Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2762111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2762221Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2762225Z 2025-08-14T21:50:29.2762345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2762560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2762640Z return mod(**inputs) 2025-08-14T21:50:29.2762910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2762983Z outputs = self.model( 2025-08-14T21:50:29.2763258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2763337Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2763610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2763697Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2763932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2764045Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2764338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2764447Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2764726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2764888Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2764892Z 2025-08-14T21:50:29.2765008Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2765221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2765294Z return mod(**inputs) 2025-08-14T21:50:29.2765592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2765669Z outputs = self.model( 2025-08-14T21:50:29.2765938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2766025Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2766295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2766380Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2766616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2766701Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2766991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2767093Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2767340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2767433Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2767436Z 2025-08-14T21:50:29.2767537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2767740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2767804Z return mod(**inputs) 2025-08-14T21:50:29.2768053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2768128Z outputs = self.model( 2025-08-14T21:50:29.2768376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2768476Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2768723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2768796Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2769019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2769097Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2769351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2769456Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2769705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2769797Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2769801Z 2025-08-14T21:50:29.2769880Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2769963Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2770049Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2770124Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2770267Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2770473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2770537Z return mod(**inputs) 2025-08-14T21:50:29.2770799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2770865Z outputs = self.model( 2025-08-14T21:50:29.2771118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2771198Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2771461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2771541Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2771758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2771839Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2772093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2772190Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2772435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2772536Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2772822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2772959Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2772964Z 2025-08-14T21:50:29.2773063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2773264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2773336Z return mod(**inputs) 2025-08-14T21:50:29.2773587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2773662Z outputs = self.model( 2025-08-14T21:50:29.2773923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2774000Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2774267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2774342Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2774591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2774682Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2774949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2775059Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2775321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2775424Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2775745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2775853Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2775857Z 2025-08-14T21:50:29.2775969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2776168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2776237Z return mod(**inputs) 2025-08-14T21:50:29.2776526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2776595Z outputs = self.model( 2025-08-14T21:50:29.2776843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2776924Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2777176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2777255Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2777469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2777550Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2777817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2777916Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2778163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2778252Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2778255Z 2025-08-14T21:50:29.2778356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2778558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2778624Z return mod(**inputs) 2025-08-14T21:50:29.2778873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2778947Z outputs = self.model( 2025-08-14T21:50:29.2779202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2779283Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2779533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2779605Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2779827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2779906Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2780154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2780271Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2780524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2780705Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2780709Z 2025-08-14T21:50:29.2780816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2781023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2781102Z return mod(**inputs) 2025-08-14T21:50:29.2781363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2781442Z outputs = self.model( 2025-08-14T21:50:29.2781705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2781781Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2782054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2782133Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2782359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2782482Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2782743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2782859Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2783105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2783185Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2783189Z 2025-08-14T21:50:29.2783299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2783495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2783570Z return mod(**inputs) 2025-08-14T21:50:29.2783841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2783912Z outputs = self.model( 2025-08-14T21:50:29.2784171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2784248Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2784509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2784591Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2784820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2784909Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2785173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2785288Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2785560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2785648Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2785652Z 2025-08-14T21:50:29.2785730Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2785815Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2785891Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2785975Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2786080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2786290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2786366Z return mod(**inputs) 2025-08-14T21:50:29.2786642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2786741Z outputs = self.model( 2025-08-14T21:50:29.2787026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2787105Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2787389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2787463Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2787697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2787788Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2788047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2788166Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2788429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2788533Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2788924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2789067Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2789071Z 2025-08-14T21:50:29.2789177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2789392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2789463Z return mod(**inputs) 2025-08-14T21:50:29.2789747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2789818Z outputs = self.model( 2025-08-14T21:50:29.2790114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2790203Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2790471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2790554Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2790783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2790866Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2791168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2791280Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2791547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2791659Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2791981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2792107Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2792111Z 2025-08-14T21:50:29.2792220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2792437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2792520Z return mod(**inputs) 2025-08-14T21:50:29.2792810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2792892Z outputs = self.model( 2025-08-14T21:50:29.2793178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2793258Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2793571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2793652Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2793901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2793994Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2794267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2794397Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2794658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2794743Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2794747Z 2025-08-14T21:50:29.2794862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2795076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2795145Z return mod(**inputs) 2025-08-14T21:50:29.2795460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2795532Z outputs = self.model( 2025-08-14T21:50:29.2795889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2795975Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2796256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2796344Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2796580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2796678Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2796983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-08-14T21:50:29.2797072Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2797077Z 2025-08-14T21:50:29.2797193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2797404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2797474Z return mod(**inputs) 2025-08-14T21:50:29.2797745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2797817Z outputs = self.model( 2025-08-14T21:50:29.2798086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2798162Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2798428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2798511Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2798742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2798823Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2799090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2799215Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2799219Z 2025-08-14T21:50:29.2799333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2799542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2799611Z return mod(**inputs) 2025-08-14T21:50:29.2799881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2799969Z outputs = self.model( 2025-08-14T21:50:29.2800249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2800328Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2800597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2800680Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2800917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2801000Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2801276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2801401Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2801638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2801712Z return self.act(input) 2025-08-14T21:50:29.2801731Z 2025-08-14T21:50:29.2801855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2802070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2802136Z return mod(**inputs) 2025-08-14T21:50:29.2802406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2802478Z outputs = self.model( 2025-08-14T21:50:29.2802738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2802819Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2803104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2803177Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2803402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2803484Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2803744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2803823Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2803827Z 2025-08-14T21:50:29.2803929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2804133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2804200Z return mod(**inputs) 2025-08-14T21:50:29.2804451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2804527Z outputs = self.model( 2025-08-14T21:50:29.2804778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2804859Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2805111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2805179Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2805405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2805483Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2805740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2805842Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2806096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2806276Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2806281Z 2025-08-14T21:50:29.2806388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2806585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2806660Z return mod(**inputs) 2025-08-14T21:50:29.2806911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2806989Z outputs = self.model( 2025-08-14T21:50:29.2807245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2807320Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2807580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2807654Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2807882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2807993Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2808243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2808349Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2808596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2808790Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2808804Z 2025-08-14T21:50:29.2808914Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2809110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2809231Z return mod(**inputs) 2025-08-14T21:50:29.2809485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2809557Z outputs = self.model( 2025-08-14T21:50:29.2809817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2809890Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2810156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2810227Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2810445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2810534Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2810786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2810885Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2811146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2811234Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2811237Z 2025-08-14T21:50:29.2811327Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2811405Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2811482Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2811565Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2811667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2811862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2811935Z return mod(**inputs) 2025-08-14T21:50:29.2812188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2812287Z outputs = self.model( 2025-08-14T21:50:29.2812544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2812619Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2812873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2812943Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2813179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2813269Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2813534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2813650Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2813905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2814027Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2814372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2814511Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2814514Z 2025-08-14T21:50:29.2814626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2814830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2814897Z return mod(**inputs) 2025-08-14T21:50:29.2815170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2815240Z outputs = self.model( 2025-08-14T21:50:29.2815526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2815611Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2815897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2815979Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2816215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2816297Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2816565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2816666Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2816936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2817042Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2817344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2817478Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2817482Z 2025-08-14T21:50:29.2817589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2817798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2817864Z return mod(**inputs) 2025-08-14T21:50:29.2818122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2818198Z outputs = self.model( 2025-08-14T21:50:29.2818451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2818545Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2818814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2818889Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2819133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2819212Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2819464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2819566Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2819828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2819913Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2819924Z 2025-08-14T21:50:29.2820038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2820250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2820344Z return mod(**inputs) 2025-08-14T21:50:29.2820627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2820701Z outputs = self.model( 2025-08-14T21:50:29.2820979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2821052Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2821310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2821381Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2821600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2821708Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2821969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2822087Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2822356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2822512Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2822516Z 2025-08-14T21:50:29.2822631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2822838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2822906Z return mod(**inputs) 2025-08-14T21:50:29.2823178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2823253Z outputs = self.model( 2025-08-14T21:50:29.2823528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2823604Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2823851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2823930Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2824151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2824229Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2824481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2824588Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2824853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2824957Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2824962Z 2025-08-14T21:50:29.2825072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2825285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2825354Z return mod(**inputs) 2025-08-14T21:50:29.2825621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2825698Z outputs = self.model( 2025-08-14T21:50:29.2825961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2826046Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2826312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2826389Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2826637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2826747Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2827006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2827113Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2827360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2827450Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2827454Z 2025-08-14T21:50:29.2827532Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2827608Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2827693Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2827784Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2827893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2828090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2828160Z return mod(**inputs) 2025-08-14T21:50:29.2828418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2828489Z outputs = self.model( 2025-08-14T21:50:29.2828762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2828847Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2829119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2829200Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2829437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2829521Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2829793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2829905Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2830165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2830276Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2830587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2830730Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2830734Z 2025-08-14T21:50:29.2830842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2831066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2831142Z return mod(**inputs) 2025-08-14T21:50:29.2831427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2831503Z outputs = self.model( 2025-08-14T21:50:29.2831774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2831850Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2832125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2832201Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2832433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2832520Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2832783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2832918Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2833194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2833297Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2833615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2833727Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2833731Z 2025-08-14T21:50:29.2833842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2834047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2834117Z return mod(**inputs) 2025-08-14T21:50:29.2834412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2834485Z outputs = self.model( 2025-08-14T21:50:29.2834761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2834846Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2835120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2835201Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2835437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2835518Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2835887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2836012Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2836296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2836387Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2836391Z 2025-08-14T21:50:29.2836504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2836728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2836801Z return mod(**inputs) 2025-08-14T21:50:29.2837088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2837170Z outputs = self.model( 2025-08-14T21:50:29.2837449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2837561Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2837846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2837924Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2838164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2838247Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2838518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2838646Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2838650Z 2025-08-14T21:50:29.2838757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2838974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2839042Z return mod(**inputs) 2025-08-14T21:50:29.2839307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2839405Z outputs = self.model( 2025-08-14T21:50:29.2839694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2839778Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2840038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2840112Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2840346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2840428Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2840696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2840834Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2841040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2841119Z return self.act(input) 2025-08-14T21:50:29.2841123Z 2025-08-14T21:50:29.2841222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2841415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2841488Z return mod(**inputs) 2025-08-14T21:50:29.2841735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2841806Z outputs = self.model( 2025-08-14T21:50:29.2842064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2842140Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2842414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2842488Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2842719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2842808Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2843070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2843162Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2843167Z 2025-08-14T21:50:29.2843275Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2843483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2843558Z return mod(**inputs) 2025-08-14T21:50:29.2843826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2843923Z outputs = self.model( 2025-08-14T21:50:29.2844188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2844267Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2844535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2844609Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2844837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2844927Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2845189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:50:29.2845280Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2845286Z 2025-08-14T21:50:29.2845394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2845601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2845715Z return mod(**inputs) 2025-08-14T21:50:29.2845983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2846054Z outputs = self.model( 2025-08-14T21:50:29.2846328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2846406Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2846678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2846749Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2846986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2847074Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2847323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2847433Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2847678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2847826Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2847830Z 2025-08-14T21:50:29.2847936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2848129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2848194Z return mod(**inputs) 2025-08-14T21:50:29.2848448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2848516Z outputs = self.model( 2025-08-14T21:50:29.2848771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2848846Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2849093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2849169Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2849385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2849471Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2849717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2849815Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2850091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2850171Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2850176Z 2025-08-14T21:50:29.2850280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2850484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2850548Z return mod(**inputs) 2025-08-14T21:50:29.2850805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2850873Z outputs = self.model( 2025-08-14T21:50:29.2851123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2851203Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2851455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2851529Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2851769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2851863Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2852124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2852226Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2852486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2852587Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2852590Z 2025-08-14T21:50:29.2852676Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2852765Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2852867Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2852950Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2853066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2853284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2853354Z return mod(**inputs) 2025-08-14T21:50:29.2853625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2853697Z outputs = self.model( 2025-08-14T21:50:29.2853975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2854050Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2854300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2854377Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2854604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2854681Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2854934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2855029Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2855281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2855375Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2855655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2855792Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2855795Z 2025-08-14T21:50:29.2855916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2856117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2856185Z return mod(**inputs) 2025-08-14T21:50:29.2856434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2856509Z outputs = self.model( 2025-08-14T21:50:29.2856759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2856831Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2857088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2857158Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2857382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2857462Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2857709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2857860Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2858102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2858195Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2858487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2858591Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2858595Z 2025-08-14T21:50:29.2858699Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2858891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2858978Z return mod(**inputs) 2025-08-14T21:50:29.2859238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2859310Z outputs = self.model( 2025-08-14T21:50:29.2859567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2859639Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2859886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2859964Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2860179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2860256Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2860511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2860609Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2860865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2860947Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2860950Z 2025-08-14T21:50:29.2861052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2861256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2861319Z return mod(**inputs) 2025-08-14T21:50:29.2861571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2861638Z outputs = self.model( 2025-08-14T21:50:29.2861884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2861984Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2862232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2862306Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2862527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2862604Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2862855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2862962Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2863208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2863364Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2863370Z 2025-08-14T21:50:29.2863472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2863669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2863774Z return mod(**inputs) 2025-08-14T21:50:29.2864023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2864096Z outputs = self.model( 2025-08-14T21:50:29.2864341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2864413Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2864666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2864737Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2864973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2865053Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2865299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2865413Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2865663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2865749Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2865753Z 2025-08-14T21:50:29.2865856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2866052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2866125Z return mod(**inputs) 2025-08-14T21:50:29.2866373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2866441Z outputs = self.model( 2025-08-14T21:50:29.2866696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2866772Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2867027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2867098Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2867314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2867400Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2867646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2867751Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2868029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2868116Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2868121Z 2025-08-14T21:50:29.2868209Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2868288Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2868365Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2868449Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2868549Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2868746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2868821Z return mod(**inputs) 2025-08-14T21:50:29.2869069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2869145Z outputs = self.model( 2025-08-14T21:50:29.2869398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2869471Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2869758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2869831Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2870063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2870142Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2870390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2870505Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2870766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2870887Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2871200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2871341Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2871344Z 2025-08-14T21:50:29.2871458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2871665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2871734Z return mod(**inputs) 2025-08-14T21:50:29.2872016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2872088Z outputs = self.model( 2025-08-14T21:50:29.2872368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2872450Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2872731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2872816Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2873100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2873185Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2873455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2873567Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2873839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2873941Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2874252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2874392Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2874398Z 2025-08-14T21:50:29.2874511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2874732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2874802Z return mod(**inputs) 2025-08-14T21:50:29.2875090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2875172Z outputs = self.model( 2025-08-14T21:50:29.2875458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2875536Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2875915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2876005Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2876252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2876380Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2876651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2876775Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2877044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2877140Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2877144Z 2025-08-14T21:50:29.2877256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2877469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2877566Z return mod(**inputs) 2025-08-14T21:50:29.2877852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2877928Z outputs = self.model( 2025-08-14T21:50:29.2878218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2878297Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2878585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2878661Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2878906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2878997Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2879269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2879398Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2879411Z 2025-08-14T21:50:29.2879523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2879735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2879812Z return mod(**inputs) 2025-08-14T21:50:29.2880093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2880164Z outputs = self.model( 2025-08-14T21:50:29.2880444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2880522Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2880799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2880896Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2881132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2881227Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2881494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2881623Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2881857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2881932Z return self.act(input) 2025-08-14T21:50:29.2881936Z 2025-08-14T21:50:29.2882054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2882265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2882338Z return mod(**inputs) 2025-08-14T21:50:29.2882618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2882710Z outputs = self.model( 2025-08-14T21:50:29.2883007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2883087Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2883359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2883445Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2883679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2883763Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2884044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2884152Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2884156Z 2025-08-14T21:50:29.2884277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2884495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2884567Z return mod(**inputs) 2025-08-14T21:50:29.2884855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2884924Z outputs = self.model( 2025-08-14T21:50:29.2885185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2885268Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2885531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2885611Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2885841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2885922Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2886203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2886303Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2886558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2886710Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2886714Z 2025-08-14T21:50:29.2886816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2887018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2887083Z return mod(**inputs) 2025-08-14T21:50:29.2887385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2887465Z outputs = self.model( 2025-08-14T21:50:29.2887733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2887817Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2888081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2888158Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2888396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2888479Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2888746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2888854Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2889120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2889247Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2889251Z 2025-08-14T21:50:29.2889368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2889563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2889636Z return mod(**inputs) 2025-08-14T21:50:29.2889884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2889958Z outputs = self.model( 2025-08-14T21:50:29.2890204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2890279Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2890550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2890625Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2890844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2890929Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2891176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2891281Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2891527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2891611Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2891614Z 2025-08-14T21:50:29.2891701Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2891781Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2891865Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2891939Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2892043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2892247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2892311Z return mod(**inputs) 2025-08-14T21:50:29.2892577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2892654Z outputs = self.model( 2025-08-14T21:50:29.2892918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2893009Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2893257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2893346Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2893570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2893651Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2893905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2894011Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2894262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2894367Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2894660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2894790Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2894797Z 2025-08-14T21:50:29.2894909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2895107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2895216Z return mod(**inputs) 2025-08-14T21:50:29.2895470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2895538Z outputs = self.model( 2025-08-14T21:50:29.2895795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2895868Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2896116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2896194Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2896425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2896516Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2896769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2896867Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2897123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2897220Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2897517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2897626Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2897630Z 2025-08-14T21:50:29.2897732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2897938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2898005Z return mod(**inputs) 2025-08-14T21:50:29.2898260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2898337Z outputs = self.model( 2025-08-14T21:50:29.2898587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2898667Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2898917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2898989Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2899213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2899291Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2899556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2899660Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2899909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2899997Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2900001Z 2025-08-14T21:50:29.2900103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2900302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2900376Z return mod(**inputs) 2025-08-14T21:50:29.2900623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2900697Z outputs = self.model( 2025-08-14T21:50:29.2900949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2901021Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2901311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2901383Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2901600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2901686Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2901934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-08-14T21:50:29.2902019Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.2902023Z 2025-08-14T21:50:29.2902123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2902334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2902411Z return mod(**inputs) 2025-08-14T21:50:29.2902665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2902739Z outputs = self.model( 2025-08-14T21:50:29.2902990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2903061Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2903320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2903391Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2903609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2903695Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2903952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2904068Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2904323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2904471Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2904475Z 2025-08-14T21:50:29.2904587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2904782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2904855Z return mod(**inputs) 2025-08-14T21:50:29.2905109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2905176Z outputs = self.model( 2025-08-14T21:50:29.2905444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2905533Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2905776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2905853Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2906061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2906141Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2906382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2906484Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2906729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2906808Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2906812Z 2025-08-14T21:50:29.2906916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2907137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2907205Z return mod(**inputs) 2025-08-14T21:50:29.2907458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2907524Z outputs = self.model( 2025-08-14T21:50:29.2907772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2907850Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2908097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2908173Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2908425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2908505Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2908931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2909046Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2909306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2909403Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2909407Z 2025-08-14T21:50:29.2909491Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2909581Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2909663Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2909743Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2909859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2910069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2910142Z return mod(**inputs) 2025-08-14T21:50:29.2910413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2910484Z outputs = self.model( 2025-08-14T21:50:29.2910757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2910835Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2911096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2911180Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2911409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2911533Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2911800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2911916Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2912185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2912288Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2912591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2912735Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2912739Z 2025-08-14T21:50:29.2912849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2913069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2913143Z return mod(**inputs) 2025-08-14T21:50:29.2913425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2913554Z outputs = self.model( 2025-08-14T21:50:29.2913820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2913898Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2914168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2914244Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2914479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2914560Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2914841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2914964Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2915231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2915344Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2915647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2915811Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2915818Z 2025-08-14T21:50:29.2915935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2916147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2916219Z return mod(**inputs) 2025-08-14T21:50:29.2916502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2916575Z outputs = self.model( 2025-08-14T21:50:29.2916859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2916938Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2917201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2917294Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2917505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2917590Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2917834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2917937Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2918218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2918669Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2918828Z 2025-08-14T21:50:29.2918936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2919426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2919813Z return mod(**inputs) 2025-08-14T21:50:29.2920209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2920632Z outputs = self.model( 2025-08-14T21:50:29.2921020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2921470Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2921859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2922271Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2922671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2923035Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2923418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2923854Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2924037Z 2025-08-14T21:50:29.2924144Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2924522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2924862Z return mod(**inputs) 2025-08-14T21:50:29.2925263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2925673Z outputs = self.model( 2025-08-14T21:50:29.2926064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2926479Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2926884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2927294Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2927659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2928079Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2928504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.2928937Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.2929327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.2929672Z return self.act(input) 2025-08-14T21:50:29.2929785Z 2025-08-14T21:50:29.2929899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2930260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2930591Z return mod(**inputs) 2025-08-14T21:50:29.2930957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2931347Z outputs = self.model( 2025-08-14T21:50:29.2931709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2932100Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2932504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2932944Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2933313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2933698Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2934110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.2934514Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.2934658Z 2025-08-14T21:50:29.2934762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2935123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2935442Z return mod(**inputs) 2025-08-14T21:50:29.2935825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2936231Z outputs = self.model( 2025-08-14T21:50:29.2936621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2937043Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2937470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2937870Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2938225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2938594Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2939001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2939437Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2939888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2940376Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2940601Z 2025-08-14T21:50:29.2940716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2941203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2941539Z return mod(**inputs) 2025-08-14T21:50:29.2941921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2942327Z outputs = self.model( 2025-08-14T21:50:29.2942705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2943114Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2943516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2943926Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2944288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2944675Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2945085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2945523Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2945949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2946364Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2946506Z 2025-08-14T21:50:29.2946625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2946998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2947371Z return mod(**inputs) 2025-08-14T21:50:29.2947753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2948161Z outputs = self.model( 2025-08-14T21:50:29.2948538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2948947Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2949347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2949746Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2950112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2950525Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2950940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2951364Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2951811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2952261Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2952412Z 2025-08-14T21:50:29.2952506Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2952739Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2952970Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2953199Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2953446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2953840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2954194Z return mod(**inputs) 2025-08-14T21:50:29.2954595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2955016Z outputs = self.model( 2025-08-14T21:50:29.2955415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2955913Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2956341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2956774Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2957283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2957688Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2958107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2958568Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2959023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2959467Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2959968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2960501Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2960707Z 2025-08-14T21:50:29.2960827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2961216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2961582Z return mod(**inputs) 2025-08-14T21:50:29.2961983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2962399Z outputs = self.model( 2025-08-14T21:50:29.2962812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2963230Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2963648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2964067Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2964441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2964800Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2965192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2965584Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2965980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2966390Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2966826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.2967317Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.2967487Z 2025-08-14T21:50:29.2967592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2967956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2968288Z return mod(**inputs) 2025-08-14T21:50:29.2968647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2969035Z outputs = self.model( 2025-08-14T21:50:29.2969397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2969782Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2970204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2970606Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2970940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2971291Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2971670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.2972074Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.2972471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.2972897Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.2973041Z 2025-08-14T21:50:29.2973159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2973553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2973890Z return mod(**inputs) 2025-08-14T21:50:29.2974282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2974690Z outputs = self.model( 2025-08-14T21:50:29.2975075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2975464Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2975866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2976289Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2976647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2977036Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2977421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2977842Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2978258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.2978724Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.2978933Z 2025-08-14T21:50:29.2979043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2979401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2979724Z return mod(**inputs) 2025-08-14T21:50:29.2980088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2980475Z outputs = self.model( 2025-08-14T21:50:29.2980832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2981243Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2981675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2982089Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2982462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2982844Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2983251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2983698Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2984153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.2984589Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.2984731Z 2025-08-14T21:50:29.2984855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2985236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2985559Z return mod(**inputs) 2025-08-14T21:50:29.2985915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2986291Z outputs = self.model( 2025-08-14T21:50:29.2986651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2987037Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2987413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2987794Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2988140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2988503Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2988885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2989307Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2989725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.2990126Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.2990271Z 2025-08-14T21:50:29.2990354Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2990572Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2990784Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2991017Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.2991254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2991635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2991986Z return mod(**inputs) 2025-08-14T21:50:29.2992363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2992775Z outputs = self.model( 2025-08-14T21:50:29.2993157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.2993560Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.2993966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.2994381Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.2994757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.2995137Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.2995579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.2996147Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.2996622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.2997083Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.2997533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.2998020Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.2998216Z 2025-08-14T21:50:29.2998332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.2998766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.2999124Z return mod(**inputs) 2025-08-14T21:50:29.2999527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.2999942Z outputs = self.model( 2025-08-14T21:50:29.3000341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3000770Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3001181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3001599Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3001984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3002382Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3002809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.3003270Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.3003722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.3004173Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.3004645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.3005139Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.3005317Z 2025-08-14T21:50:29.3005442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3005847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3006236Z return mod(**inputs) 2025-08-14T21:50:29.3006633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3007064Z outputs = self.model( 2025-08-14T21:50:29.3007442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3007886Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3008286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3008872Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3009255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3009643Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3010059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.3010505Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.3010944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.3011454Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.3011603Z 2025-08-14T21:50:29.3011725Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3012105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3012454Z return mod(**inputs) 2025-08-14T21:50:29.3012837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3013238Z outputs = self.model( 2025-08-14T21:50:29.3013627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3014026Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3014428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3014797Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3015136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3015486Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3015867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-08-14T21:50:29.3016252Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.3016394Z 2025-08-14T21:50:29.3016500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3016861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3017179Z return mod(**inputs) 2025-08-14T21:50:29.3017540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3017920Z outputs = self.model( 2025-08-14T21:50:29.3018269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3018631Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3018996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3019365Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3019690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3020034Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3020409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.3020867Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.3021035Z 2025-08-14T21:50:29.3021137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3021503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3021828Z return mod(**inputs) 2025-08-14T21:50:29.3022184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3022555Z outputs = self.model( 2025-08-14T21:50:29.3022908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3023314Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3023771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3024162Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3024512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3024868Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3025288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.3025706Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.3026084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.3026407Z return self.act(input) 2025-08-14T21:50:29.3026526Z 2025-08-14T21:50:29.3026629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3026980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3027298Z return mod(**inputs) 2025-08-14T21:50:29.3027661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3028032Z outputs = self.model( 2025-08-14T21:50:29.3028392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3028765Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3029135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3029517Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3029852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3030193Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3030573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.3030971Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.3031108Z 2025-08-14T21:50:29.3031219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3031571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3031907Z return mod(**inputs) 2025-08-14T21:50:29.3032285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3032677Z outputs = self.model( 2025-08-14T21:50:29.3033058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3033478Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3033877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3034286Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3034653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3035059Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3035475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.3036029Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.3036494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.3037013Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.3037237Z 2025-08-14T21:50:29.3037341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3037702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3038026Z return mod(**inputs) 2025-08-14T21:50:29.3038392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3038784Z outputs = self.model( 2025-08-14T21:50:29.3039162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3039605Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3039984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3040371Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3040718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3041096Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3041496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.3041934Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.3042381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.3042801Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.3042946Z 2025-08-14T21:50:29.3043058Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3043442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3043786Z return mod(**inputs) 2025-08-14T21:50:29.3044162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3044567Z outputs = self.model( 2025-08-14T21:50:29.3044953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3045364Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3045762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3046174Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3046550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3046929Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3047345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.3047781Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.3048209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.3048623Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.3048779Z 2025-08-14T21:50:29.3048866Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.3049096Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.3049341Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.3049564Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.3049812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3050200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3050539Z return mod(**inputs) 2025-08-14T21:50:29.3050921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3051340Z outputs = self.model( 2025-08-14T21:50:29.3051717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3052141Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3052548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3052970Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3053336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3053749Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3054200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.3054634Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.3055063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.3055519Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.3056000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.3056504Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.3056707Z 2025-08-14T21:50:29.3056819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3057229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3057577Z return mod(**inputs) 2025-08-14T21:50:29.3057952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3058358Z outputs = self.model( 2025-08-14T21:50:29.3058741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3059156Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3059553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3059961Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3060330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3060711Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3061130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.3061546Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.3061955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.3062365Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.3062812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.3063291Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.3063460Z 2025-08-14T21:50:29.3063576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3063955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3064332Z return mod(**inputs) 2025-08-14T21:50:29.3064719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3065130Z outputs = self.model( 2025-08-14T21:50:29.3065636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3066064Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3066467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3066926Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3067319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3067699Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3068107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:50:29.3068533Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:50:29.3069008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.3069452Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.3069596Z 2025-08-14T21:50:29.3069715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3070091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3070436Z return mod(**inputs) 2025-08-14T21:50:29.3070817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3071227Z outputs = self.model( 2025-08-14T21:50:29.3071627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3072106Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3072521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3072933Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3073311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3073712Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3074126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.3074609Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.3075071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:50:29.3075622Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:50:29.3075935Z 2025-08-14T21:50:29.3076058Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3076456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3076812Z return mod(**inputs) 2025-08-14T21:50:29.3077208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3077628Z outputs = self.model( 2025-08-14T21:50:29.3078023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3078445Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3078847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3079277Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3079654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3080081Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3080497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.3080968Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.3081417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:50:29.3081859Z key_states = self.k_proj(current_states) 2025-08-14T21:50:29.3082007Z 2025-08-14T21:50:29.3082122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3082518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3082872Z return mod(**inputs) 2025-08-14T21:50:29.3083268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3083697Z outputs = self.model( 2025-08-14T21:50:29.3084102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3084568Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3084989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3085421Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3085799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3086192Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3086612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.3087069Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.3087543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:50:29.3087979Z value_states = self.v_proj(current_states) 2025-08-14T21:50:29.3088143Z 2025-08-14T21:50:29.3088233Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.3088470Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.3088700Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.3088920Z cudagraph partition due to non gpu ops 2025-08-14T21:50:29.3089175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3089574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3089907Z return mod(**inputs) 2025-08-14T21:50:29.3090298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3090768Z outputs = self.model( 2025-08-14T21:50:29.3091144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3091547Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3091950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3092349Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3092708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3093098Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3093505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.3093953Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.3094378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.3094831Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.3095298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:50:29.3095797Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:50:29.3095999Z 2025-08-14T21:50:29.3096110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3096491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3096835Z return mod(**inputs) 2025-08-14T21:50:29.3097212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3097627Z outputs = self.model( 2025-08-14T21:50:29.3098011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3098421Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3098816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3099247Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3101966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3102373Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3102792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.3103244Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.3103681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:50:29.3104121Z attn_output, attn_weights = attention_interface( 2025-08-14T21:50:29.3104634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:50:29.3105119Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:50:29.3105289Z 2025-08-14T21:50:29.3105400Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3105813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3106153Z return mod(**inputs) 2025-08-14T21:50:29.3106526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3106928Z outputs = self.model( 2025-08-14T21:50:29.3107311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3107712Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3108112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3108517Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3109063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3109448Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3109861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:50:29.3110306Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:50:29.3110750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:50:29.3111159Z attn_output = self.out_proj(attn_output) 2025-08-14T21:50:29.3111317Z 2025-08-14T21:50:29.3111428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3111814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3112191Z return mod(**inputs) 2025-08-14T21:50:29.3112573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3112973Z outputs = self.model( 2025-08-14T21:50:29.3113361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3113784Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3114199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3114631Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3115003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3115399Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3115894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.3116380Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.3116386Z 2025-08-14T21:50:29.3116537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3116757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3116931Z return mod(**inputs) 2025-08-14T21:50:29.3117211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3117293Z outputs = self.model( 2025-08-14T21:50:29.3117568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3117647Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3117928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3118037Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3118278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3118373Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3118647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:50:29.3118784Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:50:29.3119017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:29.3119094Z return self.act(input) 2025-08-14T21:50:29.3119098Z 2025-08-14T21:50:29.3119220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3119438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3119519Z return mod(**inputs) 2025-08-14T21:50:29.3119799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3119873Z outputs = self.model( 2025-08-14T21:50:29.3120160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3120242Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3120515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3120603Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3120842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3120933Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3121206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:50:29.3121315Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:50:29.3121320Z 2025-08-14T21:50:29.3121440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3121656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3121728Z return mod(**inputs) 2025-08-14T21:50:29.3122022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:50:29.3122095Z outputs = self.model( 2025-08-14T21:50:29.3122385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:50:29.3122464Z decoder_outputs = self.decoder( 2025-08-14T21:50:29.3122793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:50:29.3122877Z layer_outputs = decoder_layer( 2025-08-14T21:50:29.3123122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:29.3123213Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:29.3123503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:50:29.3123625Z hidden_states = residual + hidden_states 2025-08-14T21:50:29.3123629Z 2025-08-14T21:50:29.3123751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3123965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3124036Z return mod(**inputs) 2025-08-14T21:50:29.3124325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1456, in forward 2025-08-14T21:50:29.3124465Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-08-14T21:50:29.3124470Z 2025-08-14T21:50:29.3124592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:29.3124788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:29.3124855Z return mod(**inputs) 2025-08-14T21:50:29.3125117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1461, in forward 2025-08-14T21:50:29.3125285Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:50:29.3125290Z 2025-08-14T21:50:42.6051165Z Compilation time (from dynamo_timed): 28.332362983 2025-08-14T21:50:42.6284770Z pass 2025-08-14T21:50:42.6286253Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:42.6287240Z TIMING: _recursive_pre_grad_passes:0.01475 _recursive_joint_graph_passes:1.16054 _recursive_post_grad_passes:0.18954 async_compile.wait:0.8151 code_gen:11.76265 inductor_compile:14.92361 backend_compile:22.40423 gc:0.00018 entire_frame_compile:28.33236 total_wall_time:28.33236 2025-08-14T21:50:42.6288267Z STATS: call_* op count: 986 | FakeTensorMode.__torch_dispatch__:33703 | FakeTensor.__torch_dispatch__:12062 | ProxyTorchDispatchMode.__torch_dispatch__:12456 2025-08-14T21:50:42.6288847Z Dynamo produced 1 graphs covering 986 ops with 0 graph breaks (0 unique) 2025-08-14T21:50:48.6647216Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:50:48.6648273Z from pkg_resources import resource_filename 2025-08-14T21:50:49.2705626Z 2025-08-14T21:50:51.9232439Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:50:51.9232977Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:50:51.9247652Z cpu eval MT5ForConditionalGeneration 2025-08-14T21:50:52.5519923Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:52.8225427Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:53.0877339Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:05.9845893Z cudagraph partition due to non gpu ops 2025-08-14T21:51:05.9846240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9846662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9847037Z return mod(**inputs) 2025-08-14T21:51:05.9847458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9847923Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9848371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9848852Z layer_outputs = layer_module( 2025-08-14T21:51:05.9849229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9849998Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9850527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9850965Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9851378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9851851Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9852283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 421, in forward 2025-08-14T21:51:05.9852697Z position_bias = position_bias + causal_mask 2025-08-14T21:51:05.9852859Z 2025-08-14T21:51:05.9853039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9853435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9853797Z return mod(**inputs) 2025-08-14T21:51:05.9854171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9854644Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9855084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9855504Z layer_outputs = layer_module( 2025-08-14T21:51:05.9855879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9856272Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9856677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9857086Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9857490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:05.9857929Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:05.9858367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:05.9858779Z return self.weight * hidden_states 2025-08-14T21:51:05.9858924Z 2025-08-14T21:51:05.9859045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9859430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9859780Z return mod(**inputs) 2025-08-14T21:51:05.9860155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9860605Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9861001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9861407Z layer_outputs = layer_module( 2025-08-14T21:51:05.9861783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9862185Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9862658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9863075Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9863481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9863887Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9864298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:05.9864773Z query_states = self.q(hidden_states) 2025-08-14T21:51:05.9864944Z 2025-08-14T21:51:05.9865055Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9865463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9865817Z return mod(**inputs) 2025-08-14T21:51:05.9866190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9866581Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9866975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9867377Z layer_outputs = layer_module( 2025-08-14T21:51:05.9867752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9868177Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9868588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9869002Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9869392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9869869Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9870266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:05.9870677Z key_states = self.k(current_states) 2025-08-14T21:51:05.9870821Z 2025-08-14T21:51:05.9870933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9871358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9871720Z return mod(**inputs) 2025-08-14T21:51:05.9872096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9872515Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9872922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9873334Z layer_outputs = layer_module( 2025-08-14T21:51:05.9873709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9874110Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9874515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9874934Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9875340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9875855Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9876275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:05.9876740Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:05.9876945Z 2025-08-14T21:51:05.9877062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9877449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9877801Z return mod(**inputs) 2025-08-14T21:51:05.9878186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9878588Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9878985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9879372Z layer_outputs = layer_module( 2025-08-14T21:51:05.9879741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9880145Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9880556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9880951Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9881346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9881758Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9882161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:05.9882638Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:05.9882867Z 2025-08-14T21:51:05.9882999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9883383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9883718Z return mod(**inputs) 2025-08-14T21:51:05.9884093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9884490Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9884876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9885260Z layer_outputs = layer_module( 2025-08-14T21:51:05.9885622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9886001Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9886387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9886803Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9887212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9887627Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9888029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:05.9888438Z value_states = self.v(current_states) 2025-08-14T21:51:05.9888592Z 2025-08-14T21:51:05.9888706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9889093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9889436Z return mod(**inputs) 2025-08-14T21:51:05.9889819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9890231Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9890652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9891064Z layer_outputs = layer_module( 2025-08-14T21:51:05.9891438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9891830Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9892235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9892635Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9893030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9893436Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9893836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:05.9894288Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:05.9894465Z 2025-08-14T21:51:05.9894588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9895024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9895393Z return mod(**inputs) 2025-08-14T21:51:05.9895772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9896184Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9896561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9896950Z layer_outputs = layer_module( 2025-08-14T21:51:05.9897312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9897684Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9898099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9898542Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9898937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9899347Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9899736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:05.9900177Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:05.9900353Z 2025-08-14T21:51:05.9900483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9900852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9901189Z return mod(**inputs) 2025-08-14T21:51:05.9901562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9901969Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9902360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9902768Z layer_outputs = layer_module( 2025-08-14T21:51:05.9903131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9903510Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9903906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9904318Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9904726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9905268Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9905681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:05.9906129Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:05.9906309Z 2025-08-14T21:51:05.9906434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9906820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9907177Z return mod(**inputs) 2025-08-14T21:51:05.9907557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9907964Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9908366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9909130Z layer_outputs = layer_module( 2025-08-14T21:51:05.9909531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9909915Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9910384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9910850Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9911261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9911675Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9912083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:05.9912499Z attn_output = self.o(attn_output) 2025-08-14T21:51:05.9912639Z 2025-08-14T21:51:05.9912753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9913169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9913523Z return mod(**inputs) 2025-08-14T21:51:05.9913900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:05.9914303Z decoder_outputs = self.decoder( 2025-08-14T21:51:05.9914710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9915119Z layer_outputs = layer_module( 2025-08-14T21:51:05.9915484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9916114Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9916533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:05.9916951Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:05.9917364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:05.9917787Z attention_output = self.EncDecAttention( 2025-08-14T21:51:05.9918203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:05.9918626Z query_states = self.q(hidden_states) 2025-08-14T21:51:05.9918770Z 2025-08-14T21:51:05.9918881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9919266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9919617Z return mod(**inputs) 2025-08-14T21:51:05.9919984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:05.9920387Z encoder_outputs = self.encoder( 2025-08-14T21:51:05.9920784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9921209Z layer_outputs = layer_module( 2025-08-14T21:51:05.9921563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9921955Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9922352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9922753Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9923163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9923570Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9923978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:05.9924375Z query_states = self.q(hidden_states) 2025-08-14T21:51:05.9924527Z 2025-08-14T21:51:05.9924640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9925021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9925404Z return mod(**inputs) 2025-08-14T21:51:05.9925794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:05.9926204Z encoder_outputs = self.encoder( 2025-08-14T21:51:05.9926594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9926978Z layer_outputs = layer_module( 2025-08-14T21:51:05.9927348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9927733Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9928153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9928560Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9928967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9929379Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9929778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:05.9930207Z key_states = self.k(current_states) 2025-08-14T21:51:05.9930351Z 2025-08-14T21:51:05.9930460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9931386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9931723Z return mod(**inputs) 2025-08-14T21:51:05.9932097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:05.9932499Z encoder_outputs = self.encoder( 2025-08-14T21:51:05.9932892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9933282Z layer_outputs = layer_module( 2025-08-14T21:51:05.9933649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9934039Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9934426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9934843Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9935259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9935660Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9936061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:05.9936549Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:05.9936747Z 2025-08-14T21:51:05.9936866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9937255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9937597Z return mod(**inputs) 2025-08-14T21:51:05.9937985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:05.9938392Z encoder_outputs = self.encoder( 2025-08-14T21:51:05.9938785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9939190Z layer_outputs = layer_module( 2025-08-14T21:51:05.9939561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9939956Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9940353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9940786Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9941214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9941618Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9942030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:05.9942523Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:05.9942749Z 2025-08-14T21:51:05.9942870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9943255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9943627Z return mod(**inputs) 2025-08-14T21:51:05.9944009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:05.9944413Z encoder_outputs = self.encoder( 2025-08-14T21:51:05.9944803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9945205Z layer_outputs = layer_module( 2025-08-14T21:51:05.9947797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9948682Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9949349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9949991Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9950638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9951192Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9951819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:05.9952583Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:05.9952827Z 2025-08-14T21:51:05.9952977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9953570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9954080Z return mod(**inputs) 2025-08-14T21:51:05.9954622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:05.9955265Z encoder_outputs = self.encoder( 2025-08-14T21:51:05.9955821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9956723Z layer_outputs = layer_module( 2025-08-14T21:51:05.9957284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9957852Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9958486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9959082Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9959702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9960298Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9960946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:05.9961564Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:05.9961807Z 2025-08-14T21:51:05.9961931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9962519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9963099Z return mod(**inputs) 2025-08-14T21:51:05.9963789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:05.9964373Z encoder_outputs = self.encoder( 2025-08-14T21:51:05.9964980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9965467Z layer_outputs = layer_module( 2025-08-14T21:51:05.9965998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9966598Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9967171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9967795Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9968286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9968823Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9969254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:05.9969663Z value_states = self.v(current_states) 2025-08-14T21:51:05.9969824Z 2025-08-14T21:51:05.9969989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9970377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9970756Z return mod(**inputs) 2025-08-14T21:51:05.9971185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:05.9971614Z encoder_outputs = self.encoder( 2025-08-14T21:51:05.9972062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:05.9972477Z layer_outputs = layer_module( 2025-08-14T21:51:05.9972905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:05.9973291Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:05.9973685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:05.9974084Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:05.9974496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:05.9974906Z attention_output = self.SelfAttention( 2025-08-14T21:51:05.9975352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:05.9975813Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:05.9975996Z 2025-08-14T21:51:05.9976106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:05.9976489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:05.9976850Z return mod(**inputs) 2025-08-14T21:51:06.0000026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0000503Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0000947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0001379Z layer_outputs = layer_module( 2025-08-14T21:51:06.0001775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0002202Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0002627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0004040Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0004511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0004946Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0005372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0005828Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0006012Z 2025-08-14T21:51:06.0006136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0006546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0006951Z return mod(**inputs) 2025-08-14T21:51:06.0007342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0007763Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0008170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0008576Z layer_outputs = layer_module( 2025-08-14T21:51:06.0009151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0009556Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0009959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0010364Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0010761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0011168Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0011570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0012000Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0012184Z 2025-08-14T21:51:06.0012300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0012690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0013040Z return mod(**inputs) 2025-08-14T21:51:06.0013411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0013813Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0014204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0014676Z layer_outputs = layer_module( 2025-08-14T21:51:06.0015054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0015447Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0015852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0016260Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0016675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0017089Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0017503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0017907Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0018053Z 2025-08-14T21:51:06.0018143Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0018410Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0018796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0019186Z return mod(**inputs) 2025-08-14T21:51:06.0019594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0020001Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0020385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0020917Z layer_outputs = layer_module( 2025-08-14T21:51:06.0021290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0021671Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0022121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0022551Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0022971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0023395Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0023821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0024228Z return self.weight * hidden_states 2025-08-14T21:51:06.0024372Z 2025-08-14T21:51:06.0024492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0024876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0025229Z return mod(**inputs) 2025-08-14T21:51:06.0025615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0026018Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0026417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0026820Z layer_outputs = layer_module( 2025-08-14T21:51:06.0027198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0027579Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0027982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0028407Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0028816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0029275Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0029715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0030165Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0030327Z 2025-08-14T21:51:06.0030437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0030817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0031173Z return mod(**inputs) 2025-08-14T21:51:06.0031538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0031937Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0032338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0032743Z layer_outputs = layer_module( 2025-08-14T21:51:06.0033119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0033527Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0033948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0034390Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0034832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0035284Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0035835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0036297Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0036454Z 2025-08-14T21:51:06.0036570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0036962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0037363Z return mod(**inputs) 2025-08-14T21:51:06.0037747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0038162Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0038572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0038980Z layer_outputs = layer_module( 2025-08-14T21:51:06.0039364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0039764Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0040180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0040602Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0041031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0041483Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0041934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0042363Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0042530Z 2025-08-14T21:51:06.0042647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0043048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0043403Z return mod(**inputs) 2025-08-14T21:51:06.0043794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0044215Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0044626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0045047Z layer_outputs = layer_module( 2025-08-14T21:51:06.0045424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0045818Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0046228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0046644Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0047064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0047514Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0047912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0048292Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0048435Z 2025-08-14T21:51:06.0048520Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0048762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0049125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0049462Z return mod(**inputs) 2025-08-14T21:51:06.0049831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0050216Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0050590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0050961Z layer_outputs = layer_module( 2025-08-14T21:51:06.0051309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0051699Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0052091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0052472Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0052849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0053252Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0053639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0054012Z return self.weight * hidden_states 2025-08-14T21:51:06.0054142Z 2025-08-14T21:51:06.0054253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0054612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0054940Z return mod(**inputs) 2025-08-14T21:51:06.0055315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0055709Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0056095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0056489Z layer_outputs = layer_module( 2025-08-14T21:51:06.0056854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0057233Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0057618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0057995Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0058369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0058749Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0059176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0059563Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0059700Z 2025-08-14T21:51:06.0059816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0060170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0060495Z return mod(**inputs) 2025-08-14T21:51:06.0060844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0061209Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0061568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0061950Z layer_outputs = layer_module( 2025-08-14T21:51:06.0062311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0062683Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0063079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0063506Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0063922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0064321Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0064720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0065120Z key_states = self.k(current_states) 2025-08-14T21:51:06.0065260Z 2025-08-14T21:51:06.0065370Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0065751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0066119Z return mod(**inputs) 2025-08-14T21:51:06.0066493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0066887Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0067281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0067683Z layer_outputs = layer_module( 2025-08-14T21:51:06.0068048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0068434Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0068835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0069243Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0069639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0070046Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0070446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0070904Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0071098Z 2025-08-14T21:51:06.0071208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0071592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0071936Z return mod(**inputs) 2025-08-14T21:51:06.0072300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0072701Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0073099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0073528Z layer_outputs = layer_module( 2025-08-14T21:51:06.0073900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0074297Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0074704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0075104Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0075500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0076001Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0076426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0076926Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0077167Z 2025-08-14T21:51:06.0077279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0077672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0078079Z return mod(**inputs) 2025-08-14T21:51:06.0078453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0078835Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0079205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0079580Z layer_outputs = layer_module( 2025-08-14T21:51:06.0079917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0080277Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0080675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0081044Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0081414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0081795Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0082196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0082665Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0082882Z 2025-08-14T21:51:06.0082987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0083349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0083676Z return mod(**inputs) 2025-08-14T21:51:06.0084021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0084400Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0084764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0085132Z layer_outputs = layer_module( 2025-08-14T21:51:06.0085479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0085838Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0086205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0086573Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0086951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0087338Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0087711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0088176Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0088390Z 2025-08-14T21:51:06.0088493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0088863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0089178Z return mod(**inputs) 2025-08-14T21:51:06.0089530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0089904Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0090272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0090641Z layer_outputs = layer_module( 2025-08-14T21:51:06.0090988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0091348Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0091711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0092108Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0092531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0092910Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0093273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0093645Z value_states = self.v(current_states) 2025-08-14T21:51:06.0093780Z 2025-08-14T21:51:06.0093892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0094242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0094587Z return mod(**inputs) 2025-08-14T21:51:06.0094941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0095316Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0095679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0096051Z layer_outputs = layer_module( 2025-08-14T21:51:06.0096396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0096750Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0097127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0097510Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0097891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0098272Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0098669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0099099Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0099270Z 2025-08-14T21:51:06.0099386Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0099759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0100100Z return mod(**inputs) 2025-08-14T21:51:06.0100443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0100825Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0101217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0101641Z layer_outputs = layer_module( 2025-08-14T21:51:06.0102008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0102378Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0102777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0103180Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0103569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0103971Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0104361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0104785Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0104952Z 2025-08-14T21:51:06.0105067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0105449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0105825Z return mod(**inputs) 2025-08-14T21:51:06.0106211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0106607Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0106998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0107396Z layer_outputs = layer_module( 2025-08-14T21:51:06.0107754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0108139Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0108540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0109371Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0109766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0110169Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0110570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0111008Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0111178Z 2025-08-14T21:51:06.0111290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0111674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0112011Z return mod(**inputs) 2025-08-14T21:51:06.0112371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0112768Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0113155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0113551Z layer_outputs = layer_module( 2025-08-14T21:51:06.0113907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0114288Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0114703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0115100Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0115503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0115977Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0116413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0116884Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0117037Z 2025-08-14T21:51:06.0117149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0117548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0117875Z return mod(**inputs) 2025-08-14T21:51:06.0118228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0118603Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0118975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0119339Z layer_outputs = layer_module( 2025-08-14T21:51:06.0119686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0120054Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0120439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0120852Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0121266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0121662Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0122048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0122450Z return self.weight * hidden_states 2025-08-14T21:51:06.0122596Z 2025-08-14T21:51:06.0122707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0123088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0123440Z return mod(**inputs) 2025-08-14T21:51:06.0123837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0124234Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0124593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0124965Z layer_outputs = layer_module( 2025-08-14T21:51:06.0125310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0125668Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0126034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0126428Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0126813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0127227Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0127627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0128024Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0128176Z 2025-08-14T21:51:06.0128289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0128633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0128950Z return mod(**inputs) 2025-08-14T21:51:06.0129295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0129665Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0130019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0130387Z layer_outputs = layer_module( 2025-08-14T21:51:06.0130769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0131127Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0131501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0131896Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0132304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0132742Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0133175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0133575Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0133718Z 2025-08-14T21:51:06.0133834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0134202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0134515Z return mod(**inputs) 2025-08-14T21:51:06.0134892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0135265Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0135634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0136006Z layer_outputs = layer_module( 2025-08-14T21:51:06.0136347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0136701Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0137075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0137498Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0137906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0138396Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0138812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0139197Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0139341Z 2025-08-14T21:51:06.0139446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0139799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0140130Z return mod(**inputs) 2025-08-14T21:51:06.0140474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0140853Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0141251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0141639Z layer_outputs = layer_module( 2025-08-14T21:51:06.0141996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0142391Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0142789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0143194Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0143589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0144032Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0144473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0144899Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0145033Z 2025-08-14T21:51:06.0145121Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0145362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0145730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0146048Z return mod(**inputs) 2025-08-14T21:51:06.0146408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0146807Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0147198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0147587Z layer_outputs = layer_module( 2025-08-14T21:51:06.0147948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0148317Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0148707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0149144Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0149554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0149985Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0150400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0150798Z return self.weight * hidden_states 2025-08-14T21:51:06.0150937Z 2025-08-14T21:51:06.0151054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0151429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0151803Z return mod(**inputs) 2025-08-14T21:51:06.0152174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0152580Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0152972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0153377Z layer_outputs = layer_module( 2025-08-14T21:51:06.0153747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0154137Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0154534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0154942Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0155355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0155838Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0156247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0156646Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0156788Z 2025-08-14T21:51:06.0156910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0157287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0157634Z return mod(**inputs) 2025-08-14T21:51:06.0158005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0158400Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0158791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0159227Z layer_outputs = layer_module( 2025-08-14T21:51:06.0159595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0159978Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0160383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0160782Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0161189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0161597Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0162007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0162498Z key_states = self.k(current_states) 2025-08-14T21:51:06.0162638Z 2025-08-14T21:51:06.0162750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0163138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0163493Z return mod(**inputs) 2025-08-14T21:51:06.0163901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0164316Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0164708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0165101Z layer_outputs = layer_module( 2025-08-14T21:51:06.0165463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0165826Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0166207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0166598Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0166971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0167371Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0167750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0168177Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0168362Z 2025-08-14T21:51:06.0168474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0168857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0169203Z return mod(**inputs) 2025-08-14T21:51:06.0169570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0169978Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0170380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0170771Z layer_outputs = layer_module( 2025-08-14T21:51:06.0171119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0171491Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0171873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0172249Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0172638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0173021Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0173399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0173871Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0174097Z 2025-08-14T21:51:06.0174209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0174590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0174938Z return mod(**inputs) 2025-08-14T21:51:06.0175302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0175698Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0176087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0176479Z layer_outputs = layer_module( 2025-08-14T21:51:06.0176849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0177238Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0177632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0178054Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0178465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0178876Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0179261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0179729Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0179954Z 2025-08-14T21:51:06.0180065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0180452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0180788Z return mod(**inputs) 2025-08-14T21:51:06.0181180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0181583Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0181979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0182370Z layer_outputs = layer_module( 2025-08-14T21:51:06.0182736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0183119Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0183510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0183916Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0184320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0184729Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0185124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0185626Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0185844Z 2025-08-14T21:51:06.0185963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0186346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0186684Z return mod(**inputs) 2025-08-14T21:51:06.0187052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0187455Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0187839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0188306Z layer_outputs = layer_module( 2025-08-14T21:51:06.0188673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0189064Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0189457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0189864Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0190267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0190668Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0191070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0191478Z value_states = self.v(current_states) 2025-08-14T21:51:06.0191621Z 2025-08-14T21:51:06.0191736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0192118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0192462Z return mod(**inputs) 2025-08-14T21:51:06.0192856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0193274Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0193664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0194052Z layer_outputs = layer_module( 2025-08-14T21:51:06.0194425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0194808Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0195214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0195647Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0196153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0196584Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0197017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0197462Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0197627Z 2025-08-14T21:51:06.0197732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0198096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0198429Z return mod(**inputs) 2025-08-14T21:51:06.0198787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0199161Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0199539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0199906Z layer_outputs = layer_module( 2025-08-14T21:51:06.0200249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0200617Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0201023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0201428Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0201827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0202232Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0202638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0203117Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0203285Z 2025-08-14T21:51:06.0203393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0203778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0204138Z return mod(**inputs) 2025-08-14T21:51:06.0204504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0204907Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0205298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0205696Z layer_outputs = layer_module( 2025-08-14T21:51:06.0206055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0206442Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0206852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0207287Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0207697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0208122Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0208519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0209145Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0209328Z 2025-08-14T21:51:06.0209439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0209825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0210169Z return mod(**inputs) 2025-08-14T21:51:06.0210600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0211001Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0211394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0211785Z layer_outputs = layer_module( 2025-08-14T21:51:06.0212153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0212542Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0212943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0213348Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0213744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0214158Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0214555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0214954Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0215098Z 2025-08-14T21:51:06.0215185Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0215443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0215820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0216166Z return mod(**inputs) 2025-08-14T21:51:06.0216540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0216931Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0217322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0217768Z layer_outputs = layer_module( 2025-08-14T21:51:06.0218130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0218502Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0218912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0219325Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0219738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0220152Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0220580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0220988Z return self.weight * hidden_states 2025-08-14T21:51:06.0221126Z 2025-08-14T21:51:06.0221235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0221618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0221974Z return mod(**inputs) 2025-08-14T21:51:06.0222379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0222799Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0223170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0223543Z layer_outputs = layer_module( 2025-08-14T21:51:06.0223886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0224250Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0224625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0225042Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0225453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0225895Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0226339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0226766Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0226928Z 2025-08-14T21:51:06.0227038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0227411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0227743Z return mod(**inputs) 2025-08-14T21:51:06.0228086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0228476Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0228872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0229271Z layer_outputs = layer_module( 2025-08-14T21:51:06.0229627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0230016Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0230414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0230828Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0231231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0231677Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0232114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0232532Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0232685Z 2025-08-14T21:51:06.0232796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0233182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0233530Z return mod(**inputs) 2025-08-14T21:51:06.0233894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0234300Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0234691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0235077Z layer_outputs = layer_module( 2025-08-14T21:51:06.0235441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0235892Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0236302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0236751Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0237192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0237638Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0238049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0238429Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0238584Z 2025-08-14T21:51:06.0238700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0239097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0239441Z return mod(**inputs) 2025-08-14T21:51:06.0239857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0240269Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0240676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0241084Z layer_outputs = layer_module( 2025-08-14T21:51:06.0241471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0241871Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0242278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0242708Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0243134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0243591Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0244035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0244455Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0244605Z 2025-08-14T21:51:06.0244708Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0244977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0245361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0245693Z return mod(**inputs) 2025-08-14T21:51:06.0246046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0246420Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0246795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0247191Z layer_outputs = layer_module( 2025-08-14T21:51:06.0247534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0247894Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0248269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0248650Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0249021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0249424Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0249824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0250196Z return self.weight * hidden_states 2025-08-14T21:51:06.0250333Z 2025-08-14T21:51:06.0250441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0250805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0251151Z return mod(**inputs) 2025-08-14T21:51:06.0251510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0251887Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0252265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0252658Z layer_outputs = layer_module( 2025-08-14T21:51:06.0253008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0253374Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0253748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0254146Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0254516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0254899Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0255278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0255645Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0255786Z 2025-08-14T21:51:06.0255893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0256249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0256573Z return mod(**inputs) 2025-08-14T21:51:06.0256914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0257290Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0257656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0258018Z layer_outputs = layer_module( 2025-08-14T21:51:06.0258359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0258719Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0259087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0259457Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0259835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0260215Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0260590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0260982Z key_states = self.k(current_states) 2025-08-14T21:51:06.0261121Z 2025-08-14T21:51:06.0261226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0261608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0261966Z return mod(**inputs) 2025-08-14T21:51:06.0262339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0262736Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0263128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0263519Z layer_outputs = layer_module( 2025-08-14T21:51:06.0263864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0264226Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0264603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0265036Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0265447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0265829Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0266199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0266625Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0266806Z 2025-08-14T21:51:06.0266919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0267286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0267597Z return mod(**inputs) 2025-08-14T21:51:06.0267958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0268325Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0268688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0269063Z layer_outputs = layer_module( 2025-08-14T21:51:06.0269409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0269775Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0270132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0270499Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0270870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0271245Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0271644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0272122Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0272342Z 2025-08-14T21:51:06.0272460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0272838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0273184Z return mod(**inputs) 2025-08-14T21:51:06.0273567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0273984Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0274364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0274791Z layer_outputs = layer_module( 2025-08-14T21:51:06.0275171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0275559Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0276051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0276468Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0276878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0277282Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0277691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0277858Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0277863Z 2025-08-14T21:51:06.0277984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0278208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0278317Z return mod(**inputs) 2025-08-14T21:51:06.0278579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0278676Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0278945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0279021Z layer_outputs = layer_module( 2025-08-14T21:51:06.0279272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0279368Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0279628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0279744Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0280002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0280093Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0280359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0280522Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0280526Z 2025-08-14T21:51:06.0280647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0280866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0280937Z return mod(**inputs) 2025-08-14T21:51:06.0281204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0281284Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0281544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0281629Z layer_outputs = layer_module( 2025-08-14T21:51:06.0281869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0281961Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0282219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0282308Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0282574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0282662Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0282927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0283080Z value_states = self.v(current_states) 2025-08-14T21:51:06.0283084Z 2025-08-14T21:51:06.0283197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0283424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0283496Z return mod(**inputs) 2025-08-14T21:51:06.0283757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0283844Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0284104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0284187Z layer_outputs = layer_module( 2025-08-14T21:51:06.0284434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0284520Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0284778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0284884Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0285163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0285259Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0285505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0285629Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0285633Z 2025-08-14T21:51:06.0285740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0285950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0286026Z return mod(**inputs) 2025-08-14T21:51:06.0286297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0286382Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0286634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0286711Z layer_outputs = layer_module( 2025-08-14T21:51:06.0286948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0287031Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0287280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0287379Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0287630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0287728Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0287979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0288094Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0288097Z 2025-08-14T21:51:06.0288215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0288424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0288499Z return mod(**inputs) 2025-08-14T21:51:06.0288751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0288830Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0289090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0289164Z layer_outputs = layer_module( 2025-08-14T21:51:06.0289419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0289511Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0289761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0289853Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0290104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0290189Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0290447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0290563Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0290566Z 2025-08-14T21:51:06.0290682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0290894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0290962Z return mod(**inputs) 2025-08-14T21:51:06.0291240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0291317Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0291584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0291668Z layer_outputs = layer_module( 2025-08-14T21:51:06.0291892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0291980Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0292224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0292307Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0292580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0292672Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0292925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0293010Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0293014Z 2025-08-14T21:51:06.0293115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0293316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0293381Z return mod(**inputs) 2025-08-14T21:51:06.0293614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0293694Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0293931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0294011Z layer_outputs = layer_module( 2025-08-14T21:51:06.0294225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0294302Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0294542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0294622Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0294854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:51:06.0294995Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:51:06.0294999Z 2025-08-14T21:51:06.0295079Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0295188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0295415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0295481Z return mod(**inputs) 2025-08-14T21:51:06.0295727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0295800Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0296043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0296116Z layer_outputs = layer_module( 2025-08-14T21:51:06.0296332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0296416Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0296650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0296746Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0296987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0297102Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0297366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0297445Z return self.weight * hidden_states 2025-08-14T21:51:06.0297449Z 2025-08-14T21:51:06.0297551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0297761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0297827Z return mod(**inputs) 2025-08-14T21:51:06.0298069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0298149Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0298410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0298489Z layer_outputs = layer_module( 2025-08-14T21:51:06.0298712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0298792Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0299029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0299119Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0299354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0299469Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0299696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0299805Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0299809Z 2025-08-14T21:51:06.0299910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0300109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0300182Z return mod(**inputs) 2025-08-14T21:51:06.0300429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0300510Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0300740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0300809Z layer_outputs = layer_module( 2025-08-14T21:51:06.0301029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0301108Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0301367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0301458Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0301691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0301811Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0302043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0302122Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0302125Z 2025-08-14T21:51:06.0302234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0302435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0302506Z return mod(**inputs) 2025-08-14T21:51:06.0302747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0302822Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0303100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0303189Z layer_outputs = layer_module( 2025-08-14T21:51:06.0303420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0303517Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0303749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0303846Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0304083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0304216Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0304458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0304546Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0304550Z 2025-08-14T21:51:06.0304663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0304871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0304940Z return mod(**inputs) 2025-08-14T21:51:06.0305199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0305283Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0305519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0305598Z layer_outputs = layer_module( 2025-08-14T21:51:06.0305815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0305900Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0306148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0306241Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0306495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0306614Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0306880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0306963Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0306967Z 2025-08-14T21:51:06.0307051Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0307191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0307400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0307467Z return mod(**inputs) 2025-08-14T21:51:06.0307715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0307786Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0308026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0308095Z layer_outputs = layer_module( 2025-08-14T21:51:06.0308311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0308395Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0308808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0308908Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0309162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0309332Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0309620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0309703Z return self.weight * hidden_states 2025-08-14T21:51:06.0309707Z 2025-08-14T21:51:06.0309815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0310051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0310119Z return mod(**inputs) 2025-08-14T21:51:06.0310379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0310483Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0310738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0310823Z layer_outputs = layer_module( 2025-08-14T21:51:06.0311064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0311148Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0311407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0311492Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0311755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0311843Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0312102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0312192Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0312195Z 2025-08-14T21:51:06.0312303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0312520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0312590Z return mod(**inputs) 2025-08-14T21:51:06.0312842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0312923Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0313173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0313248Z layer_outputs = layer_module( 2025-08-14T21:51:06.0313484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0313597Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0313854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0313940Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0314187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0314282Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0314527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0314607Z key_states = self.k(current_states) 2025-08-14T21:51:06.0314618Z 2025-08-14T21:51:06.0314726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0314931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0315015Z return mod(**inputs) 2025-08-14T21:51:06.0315274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0315353Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0315639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0315787Z layer_outputs = layer_module( 2025-08-14T21:51:06.0316044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0316131Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0316390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0316483Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0316740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0316846Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0317117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0317257Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0317261Z 2025-08-14T21:51:06.0317377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0317585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0317655Z return mod(**inputs) 2025-08-14T21:51:06.0317914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0317990Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0318250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0318327Z layer_outputs = layer_module( 2025-08-14T21:51:06.0318559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0318649Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0318906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0318991Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0319230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0319312Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0319549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0319703Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0319706Z 2025-08-14T21:51:06.0319806Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0320031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0320097Z return mod(**inputs) 2025-08-14T21:51:06.0320335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0320415Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0320651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0320728Z layer_outputs = layer_module( 2025-08-14T21:51:06.0320950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0321025Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0321259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0321339Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0321574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0321668Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0321910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0322069Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0322073Z 2025-08-14T21:51:06.0322171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0322363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0322433Z return mod(**inputs) 2025-08-14T21:51:06.0322670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0322748Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0323000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0323075Z layer_outputs = layer_module( 2025-08-14T21:51:06.0323309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0323392Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0323645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0323728Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0323972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0324064Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0324307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0324469Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0324480Z 2025-08-14T21:51:06.0324579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0324774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0324847Z return mod(**inputs) 2025-08-14T21:51:06.0325082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0325153Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0325396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0325466Z layer_outputs = layer_module( 2025-08-14T21:51:06.0325688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0325783Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0326016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0326102Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0326335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0326417Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0326656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0326733Z value_states = self.v(current_states) 2025-08-14T21:51:06.0326737Z 2025-08-14T21:51:06.0326846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0327044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0327109Z return mod(**inputs) 2025-08-14T21:51:06.0327358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0327430Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0327682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0327778Z layer_outputs = layer_module( 2025-08-14T21:51:06.0327995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0328087Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0328314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0328392Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0328629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0328725Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0328961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0329069Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0329072Z 2025-08-14T21:51:06.0329172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0329373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0329438Z return mod(**inputs) 2025-08-14T21:51:06.0329672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0329750Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0329980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0330055Z layer_outputs = layer_module( 2025-08-14T21:51:06.0330266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0330343Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0330581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0330659Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0330892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0330970Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0331194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0331305Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0331308Z 2025-08-14T21:51:06.0331408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0331626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0331700Z return mod(**inputs) 2025-08-14T21:51:06.0331935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0332011Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0332247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0332317Z layer_outputs = layer_module( 2025-08-14T21:51:06.0332535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0332612Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0332844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0332932Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0333166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0333275Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0333514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0333642Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0333647Z 2025-08-14T21:51:06.0333762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0333968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0334044Z return mod(**inputs) 2025-08-14T21:51:06.0334295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0334371Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0334680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0334756Z layer_outputs = layer_module( 2025-08-14T21:51:06.0334988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0335075Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0335311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0335399Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0335644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0335730Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0335986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0336069Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0336074Z 2025-08-14T21:51:06.0336166Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0336273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0336486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0336564Z return mod(**inputs) 2025-08-14T21:51:06.0336813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0336890Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0337150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0337223Z layer_outputs = layer_module( 2025-08-14T21:51:06.0337457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0337539Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0337806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0337909Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0338156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0338255Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0338511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0338592Z return self.weight * hidden_states 2025-08-14T21:51:06.0338596Z 2025-08-14T21:51:06.0338707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0338915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0338982Z return mod(**inputs) 2025-08-14T21:51:06.0339242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0339318Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0339599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0339699Z layer_outputs = layer_module( 2025-08-14T21:51:06.0339930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0340020Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0340268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0340360Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0340614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0340753Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0341006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0341111Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0341114Z 2025-08-14T21:51:06.0341222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0341436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0341502Z return mod(**inputs) 2025-08-14T21:51:06.0341758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0341834Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0342082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0342163Z layer_outputs = layer_module( 2025-08-14T21:51:06.0342394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0342476Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0342731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0342826Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0343080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0343202Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0343447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0343535Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0343539Z 2025-08-14T21:51:06.0343643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0343882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0343951Z return mod(**inputs) 2025-08-14T21:51:06.0344209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0344294Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0344550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0344628Z layer_outputs = layer_module( 2025-08-14T21:51:06.0344867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0344949Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0345208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0345305Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0345555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0345698Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0345960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0346054Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0346068Z 2025-08-14T21:51:06.0346174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0346382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0346464Z return mod(**inputs) 2025-08-14T21:51:06.0346716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0346790Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0347069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0347156Z layer_outputs = layer_module( 2025-08-14T21:51:06.0347380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0347458Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0347692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0347784Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0348016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0348128Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0348369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0348451Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0348454Z 2025-08-14T21:51:06.0348541Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0348644Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0348842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0348916Z return mod(**inputs) 2025-08-14T21:51:06.0349153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0349226Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0349472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0349540Z layer_outputs = layer_module( 2025-08-14T21:51:06.0349760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0349859Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0350096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0350186Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0350437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0350554Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0350804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0350883Z return self.weight * hidden_states 2025-08-14T21:51:06.0350887Z 2025-08-14T21:51:06.0350999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0351209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0351278Z return mod(**inputs) 2025-08-14T21:51:06.0351542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0351637Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0351915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0351990Z layer_outputs = layer_module( 2025-08-14T21:51:06.0352218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0352308Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0352558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0352642Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0352900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0353014Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0353270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0353353Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0353356Z 2025-08-14T21:51:06.0353465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0353679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0353746Z return mod(**inputs) 2025-08-14T21:51:06.0354009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0354082Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0354336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0354418Z layer_outputs = layer_module( 2025-08-14T21:51:06.0354648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0354731Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0354992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0355075Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0355333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0355419Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0355669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0355838Z key_states = self.k(current_states) 2025-08-14T21:51:06.0355845Z 2025-08-14T21:51:06.0355953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0356200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0356272Z return mod(**inputs) 2025-08-14T21:51:06.0356527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0356615Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0356880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0356958Z layer_outputs = layer_module( 2025-08-14T21:51:06.0357206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0357290Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0357546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0357630Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0357878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0357993Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0358257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0358397Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0358408Z 2025-08-14T21:51:06.0358513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0358719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0358788Z return mod(**inputs) 2025-08-14T21:51:06.0359018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0359087Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0359342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0359413Z layer_outputs = layer_module( 2025-08-14T21:51:06.0359632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0359708Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0359938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0360024Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0360254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0360333Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0360575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0360731Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0360736Z 2025-08-14T21:51:06.0360845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0361044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0361110Z return mod(**inputs) 2025-08-14T21:51:06.0361360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0361431Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0361676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0361747Z layer_outputs = layer_module( 2025-08-14T21:51:06.0361962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0362046Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0362303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0362384Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0362625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0362704Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0362943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0363091Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0363095Z 2025-08-14T21:51:06.0363194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0363399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0363464Z return mod(**inputs) 2025-08-14T21:51:06.0363710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0363781Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0364037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0364134Z layer_outputs = layer_module( 2025-08-14T21:51:06.0364363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0364444Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0364701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0364784Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0365039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0365136Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0365370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0365527Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0365531Z 2025-08-14T21:51:06.0365633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0365835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0365901Z return mod(**inputs) 2025-08-14T21:51:06.0366133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0366212Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0366447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0366519Z layer_outputs = layer_module( 2025-08-14T21:51:06.0366742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0366822Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0367061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0367139Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0367372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0367460Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0367692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0367769Z value_states = self.v(current_states) 2025-08-14T21:51:06.0367779Z 2025-08-14T21:51:06.0367877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0368093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0368166Z return mod(**inputs) 2025-08-14T21:51:06.0368403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0368476Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0368717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0368786Z layer_outputs = layer_module( 2025-08-14T21:51:06.0369007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0369082Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0369314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0369401Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0369634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0369733Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0369987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0370096Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0370100Z 2025-08-14T21:51:06.0370208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0370404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0370469Z return mod(**inputs) 2025-08-14T21:51:06.0370717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0370789Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0371053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0371125Z layer_outputs = layer_module( 2025-08-14T21:51:06.0371341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0371426Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0371665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0371744Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0371986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0372066Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0372306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0372415Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0372420Z 2025-08-14T21:51:06.0372521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0372727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0372792Z return mod(**inputs) 2025-08-14T21:51:06.0373028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0373109Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0373343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0373421Z layer_outputs = layer_module( 2025-08-14T21:51:06.0373638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0373716Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0373976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0374055Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0374295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0374377Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0374610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0374722Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0374725Z 2025-08-14T21:51:06.0374827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0375023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0375096Z return mod(**inputs) 2025-08-14T21:51:06.0375330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0375410Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0375665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0375736Z layer_outputs = layer_module( 2025-08-14T21:51:06.0375980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0376058Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0376290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0376375Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0376608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0376697Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0376950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0377032Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0377036Z 2025-08-14T21:51:06.0377144Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0377340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0377411Z return mod(**inputs) 2025-08-14T21:51:06.0377648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0377719Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0377964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0378036Z layer_outputs = layer_module( 2025-08-14T21:51:06.0378254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0378340Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0378573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0378661Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0378895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:51:06.0379028Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:51:06.0379033Z 2025-08-14T21:51:06.0379120Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0379221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0379426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0379501Z return mod(**inputs) 2025-08-14T21:51:06.0379750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0379827Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0380058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0380127Z layer_outputs = layer_module( 2025-08-14T21:51:06.0380343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0380418Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0380651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0380737Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0380963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0381064Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0381289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0381382Z return self.weight * hidden_states 2025-08-14T21:51:06.0381393Z 2025-08-14T21:51:06.0381488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0381695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0381767Z return mod(**inputs) 2025-08-14T21:51:06.0381999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0382068Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0382306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0382372Z layer_outputs = layer_module( 2025-08-14T21:51:06.0382606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0382687Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0382923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0383022Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0383253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0383367Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0383606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0383706Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0383711Z 2025-08-14T21:51:06.0383817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0384015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0384081Z return mod(**inputs) 2025-08-14T21:51:06.0384328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0384400Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0384651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0384721Z layer_outputs = layer_module( 2025-08-14T21:51:06.0384929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0385012Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0385238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0385323Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0385583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0385694Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0385928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0386005Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0386008Z 2025-08-14T21:51:06.0386107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0386307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0386372Z return mod(**inputs) 2025-08-14T21:51:06.0386608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0386678Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0386909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0386986Z layer_outputs = layer_module( 2025-08-14T21:51:06.0387213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0387307Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0387548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0387633Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0387868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0387978Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0388207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0388318Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0388322Z 2025-08-14T21:51:06.0388421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0388622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0388687Z return mod(**inputs) 2025-08-14T21:51:06.0388919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0388996Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0389226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0389292Z layer_outputs = layer_module( 2025-08-14T21:51:06.0389509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0389585Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0389823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0389909Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0390138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0390257Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0390483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0390560Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0390572Z 2025-08-14T21:51:06.0390651Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0390752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0390952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0391033Z return mod(**inputs) 2025-08-14T21:51:06.0391271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0391351Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0391590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0391660Z layer_outputs = layer_module( 2025-08-14T21:51:06.0391881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0391956Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0392198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0392277Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0392504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0392619Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0392867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0392977Z return self.weight * hidden_states 2025-08-14T21:51:06.0392980Z 2025-08-14T21:51:06.0393109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0393320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0393396Z return mod(**inputs) 2025-08-14T21:51:06.0393648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0393723Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0393983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0394060Z layer_outputs = layer_module( 2025-08-14T21:51:06.0394309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0394394Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0394652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0394738Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0394974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0395062Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0395301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0395381Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0395384Z 2025-08-14T21:51:06.0395499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0395781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0395859Z return mod(**inputs) 2025-08-14T21:51:06.0396129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0396208Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0396470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0396546Z layer_outputs = layer_module( 2025-08-14T21:51:06.0396784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0396876Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0397132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0397259Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0397527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0397616Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0397873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0397955Z key_states = self.k(current_states) 2025-08-14T21:51:06.0397959Z 2025-08-14T21:51:06.0398067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0398285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0398353Z return mod(**inputs) 2025-08-14T21:51:06.0398610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0398685Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0398939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0399021Z layer_outputs = layer_module( 2025-08-14T21:51:06.0399278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0399374Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0399632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0399716Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0399970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0400056Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0400304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0400487Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0400491Z 2025-08-14T21:51:06.0400600Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0400814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0400882Z return mod(**inputs) 2025-08-14T21:51:06.0401133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0401216Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0401465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0401538Z layer_outputs = layer_module( 2025-08-14T21:51:06.0401773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0401855Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0402113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0402199Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0402448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0402542Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0402790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0402950Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0402961Z 2025-08-14T21:51:06.0403066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0403275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0403349Z return mod(**inputs) 2025-08-14T21:51:06.0403624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0403701Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0403963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0404039Z layer_outputs = layer_module( 2025-08-14T21:51:06.0404277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0404357Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0404605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0404700Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0404946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0405034Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0405297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0405463Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0405467Z 2025-08-14T21:51:06.0405593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0405788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0405853Z return mod(**inputs) 2025-08-14T21:51:06.0406101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0406172Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0406411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0406478Z layer_outputs = layer_module( 2025-08-14T21:51:06.0406706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0406790Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0407021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0407100Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0407338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0407415Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0407654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0407799Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0407803Z 2025-08-14T21:51:06.0407904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0408112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0408177Z return mod(**inputs) 2025-08-14T21:51:06.0408426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0408501Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0408895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0408980Z layer_outputs = layer_module( 2025-08-14T21:51:06.0409196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0409275Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0409526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0409651Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0409884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0409964Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0410192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0410273Z value_states = self.v(current_states) 2025-08-14T21:51:06.0410277Z 2025-08-14T21:51:06.0410373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0410572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0410636Z return mod(**inputs) 2025-08-14T21:51:06.0410864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0410939Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0411170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0411239Z layer_outputs = layer_module( 2025-08-14T21:51:06.0411481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0411581Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0411815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0411893Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0412125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0412212Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0412445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0412580Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0412593Z 2025-08-14T21:51:06.0412696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0412900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0412974Z return mod(**inputs) 2025-08-14T21:51:06.0413211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0413281Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0413522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0413593Z layer_outputs = layer_module( 2025-08-14T21:51:06.0413815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0413905Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0414136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0414220Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0414450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0414531Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0414768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0414871Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0414875Z 2025-08-14T21:51:06.0414980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0415170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0415234Z return mod(**inputs) 2025-08-14T21:51:06.0415470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0415558Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0415790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0415866Z layer_outputs = layer_module( 2025-08-14T21:51:06.0416076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0416159Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0416387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0416465Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0416699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0416776Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0417013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0417133Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0417137Z 2025-08-14T21:51:06.0417234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0417450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0417515Z return mod(**inputs) 2025-08-14T21:51:06.0417748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0417825Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0418060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0418136Z layer_outputs = layer_module( 2025-08-14T21:51:06.0418373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0418453Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0418697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0418777Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0419016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0419097Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0419326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0419409Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0419412Z 2025-08-14T21:51:06.0419492Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0419592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0419799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0419865Z return mod(**inputs) 2025-08-14T21:51:06.0420119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0420189Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0420423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0420498Z layer_outputs = layer_module( 2025-08-14T21:51:06.0420707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0420783Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0421015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0421101Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0421372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0421469Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0421703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0421791Z return self.weight * hidden_states 2025-08-14T21:51:06.0421794Z 2025-08-14T21:51:06.0421894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0422099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0422165Z return mod(**inputs) 2025-08-14T21:51:06.0422405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0422488Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0422740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0422814Z layer_outputs = layer_module( 2025-08-14T21:51:06.0423080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0423179Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0423446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0423539Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0423794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0423915Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0424153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0424276Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0424280Z 2025-08-14T21:51:06.0424380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0424582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0424654Z return mod(**inputs) 2025-08-14T21:51:06.0424893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0424964Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0425206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0425275Z layer_outputs = layer_module( 2025-08-14T21:51:06.0425502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0425578Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0425817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0425911Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0426149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0426269Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0426531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0426614Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0426618Z 2025-08-14T21:51:06.0426729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0426948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0427016Z return mod(**inputs) 2025-08-14T21:51:06.0427298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0427374Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0427639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0427710Z layer_outputs = layer_module( 2025-08-14T21:51:06.0427926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0428010Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0428249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0428337Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0428585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0428702Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0428944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0429051Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0429054Z 2025-08-14T21:51:06.0429198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0429416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0429485Z return mod(**inputs) 2025-08-14T21:51:06.0429746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0429823Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0430074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0430156Z layer_outputs = layer_module( 2025-08-14T21:51:06.0430408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0430492Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0430750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0430842Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0431097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0431216Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0431472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0431566Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0431570Z 2025-08-14T21:51:06.0431655Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0431774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0431983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0432053Z return mod(**inputs) 2025-08-14T21:51:06.0432316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0432396Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0432647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0432730Z layer_outputs = layer_module( 2025-08-14T21:51:06.0432968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0433057Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0433322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0433426Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0433677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0433789Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0434044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0434132Z return self.weight * hidden_states 2025-08-14T21:51:06.0434136Z 2025-08-14T21:51:06.0434241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0434469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0434539Z return mod(**inputs) 2025-08-14T21:51:06.0434789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0434876Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0435126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0435225Z layer_outputs = layer_module( 2025-08-14T21:51:06.0435465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0435569Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0435894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0435984Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0436241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0436337Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0436595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0436707Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0436712Z 2025-08-14T21:51:06.0436822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0437032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0437109Z return mod(**inputs) 2025-08-14T21:51:06.0437345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0437418Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0437662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0437733Z layer_outputs = layer_module( 2025-08-14T21:51:06.0437957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0438034Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0438270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0438361Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0438595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0438685Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0438917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0438991Z key_states = self.k(current_states) 2025-08-14T21:51:06.0438995Z 2025-08-14T21:51:06.0439102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0439295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0439360Z return mod(**inputs) 2025-08-14T21:51:06.0439641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0439718Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0439980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0440065Z layer_outputs = layer_module( 2025-08-14T21:51:06.0440287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0440370Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0440609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0440685Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0440927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0441008Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0441254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0441401Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0441405Z 2025-08-14T21:51:06.0441507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0441725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0441790Z return mod(**inputs) 2025-08-14T21:51:06.0442030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0442101Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0442336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0442413Z layer_outputs = layer_module( 2025-08-14T21:51:06.0442656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0442740Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0442997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0443080Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0443333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0443418Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0443663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0443829Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0443833Z 2025-08-14T21:51:06.0443939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0444154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0444222Z return mod(**inputs) 2025-08-14T21:51:06.0444481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0444559Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0444795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0444866Z layer_outputs = layer_module( 2025-08-14T21:51:06.0445088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0445164Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0445403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0445482Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0445749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0445844Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0446096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0446262Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0446266Z 2025-08-14T21:51:06.0446373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0446580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0446657Z return mod(**inputs) 2025-08-14T21:51:06.0446911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0446985Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0447248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0447324Z layer_outputs = layer_module( 2025-08-14T21:51:06.0447575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0447673Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0447920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0448011Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0448256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0448340Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0448591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0448765Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0448769Z 2025-08-14T21:51:06.0448886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0449096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0449166Z return mod(**inputs) 2025-08-14T21:51:06.0449424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0449498Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0449756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0449830Z layer_outputs = layer_module( 2025-08-14T21:51:06.0450060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0450149Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0450401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0450484Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0450740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0450824Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0451080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0451160Z value_states = self.v(current_states) 2025-08-14T21:51:06.0451164Z 2025-08-14T21:51:06.0451271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0451490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0451559Z return mod(**inputs) 2025-08-14T21:51:06.0451840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0451917Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0452165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0452245Z layer_outputs = layer_module( 2025-08-14T21:51:06.0452471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0452552Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0452807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0452898Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0453137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0453219Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0453453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0453586Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0453590Z 2025-08-14T21:51:06.0453689Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0453910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0453975Z return mod(**inputs) 2025-08-14T21:51:06.0454224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0454302Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0454532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0454601Z layer_outputs = layer_module( 2025-08-14T21:51:06.0454835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0454914Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0455154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0455233Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0455476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0455561Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0455786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0455889Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0455900Z 2025-08-14T21:51:06.0455998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0456188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0456257Z return mod(**inputs) 2025-08-14T21:51:06.0456483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0456553Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0456789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0456859Z layer_outputs = layer_module( 2025-08-14T21:51:06.0457073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0457148Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0457374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0457458Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0457708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0457787Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0458021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0458126Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0458130Z 2025-08-14T21:51:06.0458235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0458426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0458487Z return mod(**inputs) 2025-08-14T21:51:06.0458732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0458802Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0459047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0459121Z layer_outputs = layer_module( 2025-08-14T21:51:06.0459366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0459451Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0459696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0459775Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0460010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0460090Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0460322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0460397Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0460417Z 2025-08-14T21:51:06.0460518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0460716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0460782Z return mod(**inputs) 2025-08-14T21:51:06.0461011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0461090Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0461320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0461395Z layer_outputs = layer_module( 2025-08-14T21:51:06.0461611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0461687Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0461930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0462012Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0462255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:51:06.0462390Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:51:06.0462393Z 2025-08-14T21:51:06.0462473Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0462582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0462779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0462843Z return mod(**inputs) 2025-08-14T21:51:06.0463094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0463166Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0463428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0463496Z layer_outputs = layer_module( 2025-08-14T21:51:06.0463712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0463799Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0464034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0464124Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0464366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0464462Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0464701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0464780Z return self.weight * hidden_states 2025-08-14T21:51:06.0464783Z 2025-08-14T21:51:06.0464883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0465108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0465173Z return mod(**inputs) 2025-08-14T21:51:06.0465443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0465518Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0465759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0465836Z layer_outputs = layer_module( 2025-08-14T21:51:06.0466052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0466129Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0466389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0466480Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0466728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0466845Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0467080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0467184Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0467187Z 2025-08-14T21:51:06.0467287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0467491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0467557Z return mod(**inputs) 2025-08-14T21:51:06.0467794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0467877Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0468116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0468186Z layer_outputs = layer_module( 2025-08-14T21:51:06.0468413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0468490Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0468734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0468821Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0469056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0469176Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0469427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0469511Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0469514Z 2025-08-14T21:51:06.0469616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0469812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0469883Z return mod(**inputs) 2025-08-14T21:51:06.0470117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0470188Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0470441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0470515Z layer_outputs = layer_module( 2025-08-14T21:51:06.0470748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0470830Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0471095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0471207Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0471456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0471575Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0471829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0471922Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0471925Z 2025-08-14T21:51:06.0472040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0472260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0472330Z return mod(**inputs) 2025-08-14T21:51:06.0472587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0472665Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0472922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0472996Z layer_outputs = layer_module( 2025-08-14T21:51:06.0473228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0473318Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0473571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0473666Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0473931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0474063Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0474317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0474401Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0474405Z 2025-08-14T21:51:06.0474489Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0474603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0474812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0474891Z return mod(**inputs) 2025-08-14T21:51:06.0475145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:51:06.0475243Z encoder_outputs = self.encoder( 2025-08-14T21:51:06.0475511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1115, in forward 2025-08-14T21:51:06.0475628Z hidden_states = self.final_layer_norm(hidden_states) 2025-08-14T21:51:06.0475967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0476063Z return self.weight * hidden_states 2025-08-14T21:51:06.0476067Z 2025-08-14T21:51:06.0476181Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0476406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0476479Z return mod(**inputs) 2025-08-14T21:51:06.0476745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0476833Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0477103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0477180Z layer_outputs = layer_module( 2025-08-14T21:51:06.0477443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0477538Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0477783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0477866Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0478104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0478198Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0478434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0478534Z key_states = self.k(current_states) 2025-08-14T21:51:06.0478539Z 2025-08-14T21:51:06.0478642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0478842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0478915Z return mod(**inputs) 2025-08-14T21:51:06.0479152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0479225Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0479465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0479535Z layer_outputs = layer_module( 2025-08-14T21:51:06.0479759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0479837Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0480074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0480163Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0480401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0480485Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0480726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0480856Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0480859Z 2025-08-14T21:51:06.0480967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0481165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0481230Z return mod(**inputs) 2025-08-14T21:51:06.0481476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0481569Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0481817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0481890Z layer_outputs = layer_module( 2025-08-14T21:51:06.0482119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0482209Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0482461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0482544Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0482803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0482890Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0483152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0483313Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0483334Z 2025-08-14T21:51:06.0483440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0483671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0483741Z return mod(**inputs) 2025-08-14T21:51:06.0484000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0484075Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0484324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0484403Z layer_outputs = layer_module( 2025-08-14T21:51:06.0484649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0484731Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0484988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0485073Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0485327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0485415Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0485664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0485752Z value_states = self.v(current_states) 2025-08-14T21:51:06.0485756Z 2025-08-14T21:51:06.0485866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0486069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0486136Z return mod(**inputs) 2025-08-14T21:51:06.0486376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0486455Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0486694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0486761Z layer_outputs = layer_module( 2025-08-14T21:51:06.0486993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0487073Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0487328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0487412Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0487678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0487774Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0488022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0488137Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0488148Z 2025-08-14T21:51:06.0488253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0488460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0488534Z return mod(**inputs) 2025-08-14T21:51:06.0488784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0488858Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0489119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0489194Z layer_outputs = layer_module( 2025-08-14T21:51:06.0489432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0489540Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0489855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0489952Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0490236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0490323Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0490600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0490713Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0490736Z 2025-08-14T21:51:06.0490854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0491080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0491150Z return mod(**inputs) 2025-08-14T21:51:06.0491409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0491484Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0491744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0491819Z layer_outputs = layer_module( 2025-08-14T21:51:06.0492056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0492141Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0492390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0492474Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0492729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0492816Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0493082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0493195Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0493199Z 2025-08-14T21:51:06.0493307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0493525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0493594Z return mod(**inputs) 2025-08-14T21:51:06.0493846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0493949Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0494202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0494286Z layer_outputs = layer_module( 2025-08-14T21:51:06.0494530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0494612Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0494872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0494956Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0495223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0495308Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0495563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0495652Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0495672Z 2025-08-14T21:51:06.0495758Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0495863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0496113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0496184Z return mod(**inputs) 2025-08-14T21:51:06.0496443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0496525Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0496777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0496859Z layer_outputs = layer_module( 2025-08-14T21:51:06.0497112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0497204Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0497449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0497540Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0497785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0497887Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0498145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0498232Z return self.weight * hidden_states 2025-08-14T21:51:06.0498236Z 2025-08-14T21:51:06.0498343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0498566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0498632Z return mod(**inputs) 2025-08-14T21:51:06.0498879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0498960Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0499205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0499274Z layer_outputs = layer_module( 2025-08-14T21:51:06.0499592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0499700Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0500059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0500177Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0500559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0500684Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0500925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0501034Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0501038Z 2025-08-14T21:51:06.0501143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0501351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0501425Z return mod(**inputs) 2025-08-14T21:51:06.0501673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0501749Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0502007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0502083Z layer_outputs = layer_module( 2025-08-14T21:51:06.0502341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0502422Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0502689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0502790Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0503041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0503172Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0503426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0503529Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0503534Z 2025-08-14T21:51:06.0503654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0503870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0503941Z return mod(**inputs) 2025-08-14T21:51:06.0504207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0504286Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0504552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0504629Z layer_outputs = layer_module( 2025-08-14T21:51:06.0504864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0504970Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0505241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0505374Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0505774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0505930Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0506189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0506281Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0506285Z 2025-08-14T21:51:06.0506393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0506608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0506678Z return mod(**inputs) 2025-08-14T21:51:06.0506947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0507049Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0507317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0507404Z layer_outputs = layer_module( 2025-08-14T21:51:06.0507647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0507733Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0508003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0508096Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0508365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0508488Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0508940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0509095Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0509100Z 2025-08-14T21:51:06.0509214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0509466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0509540Z return mod(**inputs) 2025-08-14T21:51:06.0509800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0509884Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0510140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0510217Z layer_outputs = layer_module( 2025-08-14T21:51:06.0510492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0510580Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0510844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0510933Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0511188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0511309Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0511564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0511645Z return self.weight * hidden_states 2025-08-14T21:51:06.0511657Z 2025-08-14T21:51:06.0511767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0511984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0512062Z return mod(**inputs) 2025-08-14T21:51:06.0512320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0512401Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0512668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0512747Z layer_outputs = layer_module( 2025-08-14T21:51:06.0512988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0513072Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0513326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0513421Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0513715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0513806Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0514068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0514152Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0514156Z 2025-08-14T21:51:06.0514273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0514482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0514552Z return mod(**inputs) 2025-08-14T21:51:06.0514817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0514893Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0515156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0515234Z layer_outputs = layer_module( 2025-08-14T21:51:06.0515466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0515574Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0516256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0516357Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0516619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0516709Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0516971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0517052Z key_states = self.k(current_states) 2025-08-14T21:51:06.0517058Z 2025-08-14T21:51:06.0517187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0517413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0517486Z return mod(**inputs) 2025-08-14T21:51:06.0517752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0517838Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0518100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0518182Z layer_outputs = layer_module( 2025-08-14T21:51:06.0518420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0518504Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0518777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0518866Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0519134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0519224Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0519483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0519631Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0519634Z 2025-08-14T21:51:06.0519744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0519959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0520036Z return mod(**inputs) 2025-08-14T21:51:06.0520297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0520409Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0520664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0520742Z layer_outputs = layer_module( 2025-08-14T21:51:06.0520987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0521071Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0521330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0521417Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0521673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0521766Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0522022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0522187Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0522210Z 2025-08-14T21:51:06.0522329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0522558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0522637Z return mod(**inputs) 2025-08-14T21:51:06.0522898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0522977Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0523251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0523321Z layer_outputs = layer_module( 2025-08-14T21:51:06.0523539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0523644Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0523878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0523965Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0524200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0524281Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0524520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0524596Z value_states = self.v(current_states) 2025-08-14T21:51:06.0524600Z 2025-08-14T21:51:06.0524706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0524912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0524979Z return mod(**inputs) 2025-08-14T21:51:06.0525222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0525296Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0525532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0525607Z layer_outputs = layer_module( 2025-08-14T21:51:06.0525821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0525904Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0526137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0526214Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0526455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0526555Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0526793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0526900Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0526904Z 2025-08-14T21:51:06.0527006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0527205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0527270Z return mod(**inputs) 2025-08-14T21:51:06.0527504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0527582Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0527815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0527896Z layer_outputs = layer_module( 2025-08-14T21:51:06.0528112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0528231Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0528491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0528573Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0528807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0528894Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0529136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0529248Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0529252Z 2025-08-14T21:51:06.0529369Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0529567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0529642Z return mod(**inputs) 2025-08-14T21:51:06.0529881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0529960Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0530198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0530268Z layer_outputs = layer_module( 2025-08-14T21:51:06.0530490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0530566Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0530801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0530889Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0531125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0531214Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0531449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0531554Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0531557Z 2025-08-14T21:51:06.0531665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0531858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0531929Z return mod(**inputs) 2025-08-14T21:51:06.0532170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0532260Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0532508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0532583Z layer_outputs = layer_module( 2025-08-14T21:51:06.0532819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0532909Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0533155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0533245Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0533498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0533581Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0533834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0533919Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0533923Z 2025-08-14T21:51:06.0534014Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0534145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0534369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0534454Z return mod(**inputs) 2025-08-14T21:51:06.0534688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0534760Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0535005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0535076Z layer_outputs = layer_module( 2025-08-14T21:51:06.0535300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0535406Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0535645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0535735Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0535970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:51:06.0536076Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0536319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0536394Z return self.weight * hidden_states 2025-08-14T21:51:06.0536397Z 2025-08-14T21:51:06.0536506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0536702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0536767Z return mod(**inputs) 2025-08-14T21:51:06.0537013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0537086Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0537332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0537403Z layer_outputs = layer_module( 2025-08-14T21:51:06.0537620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0537704Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0537938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0538017Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0538260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0538362Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0538603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0538682Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0538686Z 2025-08-14T21:51:06.0538789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0538991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0539055Z return mod(**inputs) 2025-08-14T21:51:06.0539289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0539368Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0539600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0539678Z layer_outputs = layer_module( 2025-08-14T21:51:06.0539895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0539990Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0540245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0540327Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0540568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0540652Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0540886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0540969Z key_states = self.k(current_states) 2025-08-14T21:51:06.0540973Z 2025-08-14T21:51:06.0541073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0541287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0541359Z return mod(**inputs) 2025-08-14T21:51:06.0541595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0541675Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0541911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0541981Z layer_outputs = layer_module( 2025-08-14T21:51:06.0542205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0542286Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0542540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0542634Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0542885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0542983Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0543239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0543374Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0543377Z 2025-08-14T21:51:06.0543494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0543708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0543784Z return mod(**inputs) 2025-08-14T21:51:06.0544041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0544117Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0544399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0544473Z layer_outputs = layer_module( 2025-08-14T21:51:06.0544700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0544789Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0545032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0545116Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0545343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0545420Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0545653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0545804Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0545807Z 2025-08-14T21:51:06.0545912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0546121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0546201Z return mod(**inputs) 2025-08-14T21:51:06.0546440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0546510Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0546741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0546817Z layer_outputs = layer_module( 2025-08-14T21:51:06.0547028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0547110Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0547359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0547442Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0547684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0547766Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0548004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0548080Z value_states = self.v(current_states) 2025-08-14T21:51:06.0548084Z 2025-08-14T21:51:06.0548184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0548386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0548451Z return mod(**inputs) 2025-08-14T21:51:06.0548688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0548768Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0549004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0549083Z layer_outputs = layer_module( 2025-08-14T21:51:06.0549296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0549373Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0549613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0549692Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0549924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0550033Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0550266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0550380Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0550384Z 2025-08-14T21:51:06.0550486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0550681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0550756Z return mod(**inputs) 2025-08-14T21:51:06.0550999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0551082Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0551332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0551404Z layer_outputs = layer_module( 2025-08-14T21:51:06.0551646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0551723Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0551975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0552077Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0552329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0552422Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0552670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0552782Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0552785Z 2025-08-14T21:51:06.0552898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0553125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0553203Z return mod(**inputs) 2025-08-14T21:51:06.0553454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0553530Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0553794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0553867Z layer_outputs = layer_module( 2025-08-14T21:51:06.0554095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0554185Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0554434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0554522Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0554772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0554859Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0555116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0555228Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0555231Z 2025-08-14T21:51:06.0555345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0555555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0555625Z return mod(**inputs) 2025-08-14T21:51:06.0556007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0556092Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0556367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0556452Z layer_outputs = layer_module( 2025-08-14T21:51:06.0556681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0556771Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0557019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0557102Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0557358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0557440Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0557665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0557752Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0557756Z 2025-08-14T21:51:06.0557837Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0557945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0558154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0558241Z return mod(**inputs) 2025-08-14T21:51:06.0558482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0558551Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0558784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0558861Z layer_outputs = layer_module( 2025-08-14T21:51:06.0559069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0559150Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0559396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0559487Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0559724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0559819Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0560056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0560128Z return self.weight * hidden_states 2025-08-14T21:51:06.0560132Z 2025-08-14T21:51:06.0560230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0560430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0560492Z return mod(**inputs) 2025-08-14T21:51:06.0560729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0560806Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0561041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0561114Z layer_outputs = layer_module( 2025-08-14T21:51:06.0561322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0561395Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0561629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0561713Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0561947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0562080Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0562312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0562413Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0562417Z 2025-08-14T21:51:06.0562514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0562703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0562774Z return mod(**inputs) 2025-08-14T21:51:06.0563011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0563089Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0563324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0563394Z layer_outputs = layer_module( 2025-08-14T21:51:06.0563619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0563697Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0563949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0564069Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0564301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0564418Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0564648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0564724Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0564727Z 2025-08-14T21:51:06.0564833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0565043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0565119Z return mod(**inputs) 2025-08-14T21:51:06.0565361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0565434Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0565680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0565750Z layer_outputs = layer_module( 2025-08-14T21:51:06.0565969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0566053Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0566290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0566388Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0566630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0566746Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0566994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0567081Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0567085Z 2025-08-14T21:51:06.0567192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0567392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0567456Z return mod(**inputs) 2025-08-14T21:51:06.0567700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0567772Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0568028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0568108Z layer_outputs = layer_module( 2025-08-14T21:51:06.0568322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0568407Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0568642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0568729Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0568969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0569081Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0569318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0569401Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0569405Z 2025-08-14T21:51:06.0569484Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0569611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0569822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0569891Z return mod(**inputs) 2025-08-14T21:51:06.0570133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0570206Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0570449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0570519Z layer_outputs = layer_module( 2025-08-14T21:51:06.0570733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0570834Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0571068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0571150Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0571392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0571496Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0571736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0571812Z return self.weight * hidden_states 2025-08-14T21:51:06.0571815Z 2025-08-14T21:51:06.0571915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0572117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0572184Z return mod(**inputs) 2025-08-14T21:51:06.0572429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0572503Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0572738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0572815Z layer_outputs = layer_module( 2025-08-14T21:51:06.0573026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0573102Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0573344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0573423Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0573663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0573766Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0573996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0574081Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0574085Z 2025-08-14T21:51:06.0574186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0574387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0574450Z return mod(**inputs) 2025-08-14T21:51:06.0574684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0574763Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0574998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0575068Z layer_outputs = layer_module( 2025-08-14T21:51:06.0575292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0575388Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0575643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0575725Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0575961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0576052Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0576285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0576361Z key_states = self.k(current_states) 2025-08-14T21:51:06.0576372Z 2025-08-14T21:51:06.0576474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0576686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0576761Z return mod(**inputs) 2025-08-14T21:51:06.0576999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0577070Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0577321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0577390Z layer_outputs = layer_module( 2025-08-14T21:51:06.0577606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0577681Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0577907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0577994Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0578219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0578299Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0578537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0578667Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0578670Z 2025-08-14T21:51:06.0578777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0578973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0579038Z return mod(**inputs) 2025-08-14T21:51:06.0579282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0579353Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0579614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0579703Z layer_outputs = layer_module( 2025-08-14T21:51:06.0579914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0579996Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0580230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0580308Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0580542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0580621Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0580855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0581004Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0581008Z 2025-08-14T21:51:06.0581107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0581348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0581428Z return mod(**inputs) 2025-08-14T21:51:06.0581660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0581738Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0581967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0582042Z layer_outputs = layer_module( 2025-08-14T21:51:06.0582262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0582338Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0582598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0582683Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0582934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0583016Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0583251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0583335Z value_states = self.v(current_states) 2025-08-14T21:51:06.0583338Z 2025-08-14T21:51:06.0583438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0583644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0583720Z return mod(**inputs) 2025-08-14T21:51:06.0583974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0584055Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0584315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0584385Z layer_outputs = layer_module( 2025-08-14T21:51:06.0584610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0584685Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0584920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0585006Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0585241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0585354Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0585588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0585697Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0585701Z 2025-08-14T21:51:06.0585810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0586008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0586080Z return mod(**inputs) 2025-08-14T21:51:06.0586316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0586385Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0586633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0586702Z layer_outputs = layer_module( 2025-08-14T21:51:06.0586924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0587009Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0587258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0587370Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0587604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0587684Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0587925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0588031Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0588034Z 2025-08-14T21:51:06.0588142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0588352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0588418Z return mod(**inputs) 2025-08-14T21:51:06.0588665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0588737Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0588977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0589054Z layer_outputs = layer_module( 2025-08-14T21:51:06.0589271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0589354Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0589590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0589668Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0589912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0589991Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0590230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0590348Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0590352Z 2025-08-14T21:51:06.0590459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0590676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0590743Z return mod(**inputs) 2025-08-14T21:51:06.0590999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0591079Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0591319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0591414Z layer_outputs = layer_module( 2025-08-14T21:51:06.0591636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0591711Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0591959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0592039Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0592292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0592385Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0592640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0592726Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0592733Z 2025-08-14T21:51:06.0592840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0593069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0593165Z return mod(**inputs) 2025-08-14T21:51:06.0593436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0593520Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0593770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0593845Z layer_outputs = layer_module( 2025-08-14T21:51:06.0594098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0594179Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0594469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0594564Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0594814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:51:06.0594961Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:51:06.0594965Z 2025-08-14T21:51:06.0595049Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0595155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0595370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0595438Z return mod(**inputs) 2025-08-14T21:51:06.0595802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0595929Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0596198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0596281Z layer_outputs = layer_module( 2025-08-14T21:51:06.0596512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0596595Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0596853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0596940Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0597229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:51:06.0597343Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0597628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0597750Z return self.weight * hidden_states 2025-08-14T21:51:06.0597755Z 2025-08-14T21:51:06.0597865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0598080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0598159Z return mod(**inputs) 2025-08-14T21:51:06.0598433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0598518Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0598776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0598852Z layer_outputs = layer_module( 2025-08-14T21:51:06.0599095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0599177Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0599465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0599551Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0599856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0599968Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0600254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0600336Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0600340Z 2025-08-14T21:51:06.0600454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0600664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0600741Z return mod(**inputs) 2025-08-14T21:51:06.0601012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0601090Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0601348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0601423Z layer_outputs = layer_module( 2025-08-14T21:51:06.0601670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0601760Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0602026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0602117Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0602382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0602468Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0602724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0602803Z key_states = self.k(current_states) 2025-08-14T21:51:06.0602808Z 2025-08-14T21:51:06.0602919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0603127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0603194Z return mod(**inputs) 2025-08-14T21:51:06.0603448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0603521Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0603768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0603849Z layer_outputs = layer_module( 2025-08-14T21:51:06.0604075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0604192Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0604437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0604521Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0604775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0604861Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0605106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0605249Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0605253Z 2025-08-14T21:51:06.0605357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0605571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0605641Z return mod(**inputs) 2025-08-14T21:51:06.0605889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0605991Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0606256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0606338Z layer_outputs = layer_module( 2025-08-14T21:51:06.0606566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0606646Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0606900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0606983Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0607245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0607341Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0607594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0607766Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0607770Z 2025-08-14T21:51:06.0607880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0608091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0608170Z return mod(**inputs) 2025-08-14T21:51:06.0608425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0608512Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0609082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0609179Z layer_outputs = layer_module( 2025-08-14T21:51:06.0609455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0609539Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0609792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0609885Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0610134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0610231Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0610478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0610560Z value_states = self.v(current_states) 2025-08-14T21:51:06.0610615Z 2025-08-14T21:51:06.0610732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0610939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0611018Z return mod(**inputs) 2025-08-14T21:51:06.0611269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0611346Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0611609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0611684Z layer_outputs = layer_module( 2025-08-14T21:51:06.0611913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0612002Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0612254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0612348Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0612595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0612711Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0612995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0613110Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0613113Z 2025-08-14T21:51:06.0613225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0613431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0613499Z return mod(**inputs) 2025-08-14T21:51:06.0613758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0613859Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0614115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0614199Z layer_outputs = layer_module( 2025-08-14T21:51:06.0614428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0614511Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0614744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0614821Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0615062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0615142Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0615377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0615492Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0615497Z 2025-08-14T21:51:06.0615597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0615801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0615865Z return mod(**inputs) 2025-08-14T21:51:06.0616109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0616193Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0616446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0616526Z layer_outputs = layer_module( 2025-08-14T21:51:06.0616755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0616856Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0617108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0617192Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0617437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0617532Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0617776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0617893Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0617896Z 2025-08-14T21:51:06.0618001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0618207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0618288Z return mod(**inputs) 2025-08-14T21:51:06.0618536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0618632Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0618908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0618978Z layer_outputs = layer_module( 2025-08-14T21:51:06.0619199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0619274Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0619508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0619594Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0619840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0619929Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0620165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0620242Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0620245Z 2025-08-14T21:51:06.0620334Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0620435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0620633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0620705Z return mod(**inputs) 2025-08-14T21:51:06.0620940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0621018Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0621254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0621324Z layer_outputs = layer_module( 2025-08-14T21:51:06.0621547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0621623Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0621860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0621957Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0622190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0622296Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0622544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0622622Z return self.weight * hidden_states 2025-08-14T21:51:06.0622666Z 2025-08-14T21:51:06.0622781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0622989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0623066Z return mod(**inputs) 2025-08-14T21:51:06.0623318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0623393Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0623649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0623723Z layer_outputs = layer_module( 2025-08-14T21:51:06.0623948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0624036Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0624285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0624383Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0624615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0624748Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0625003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0625103Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0625106Z 2025-08-14T21:51:06.0625214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0625407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0625471Z return mod(**inputs) 2025-08-14T21:51:06.0625736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0625812Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0626052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0626136Z layer_outputs = layer_module( 2025-08-14T21:51:06.0626365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0626455Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0626705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0626797Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0627055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0627176Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0627434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0627517Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0627522Z 2025-08-14T21:51:06.0627630Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0627845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0627913Z return mod(**inputs) 2025-08-14T21:51:06.0628165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0628247Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0628499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0628581Z layer_outputs = layer_module( 2025-08-14T21:51:06.0628810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0628911Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0629172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0629266Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0629514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0629642Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0629889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0629988Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0629992Z 2025-08-14T21:51:06.0630099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0630308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0630388Z return mod(**inputs) 2025-08-14T21:51:06.0630641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0630749Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0631023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0631100Z layer_outputs = layer_module( 2025-08-14T21:51:06.0631336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0631418Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0631662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0631763Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0632930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0633085Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0633334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0633420Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0633424Z 2025-08-14T21:51:06.0633516Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0633623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0633836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0633904Z return mod(**inputs) 2025-08-14T21:51:06.0634156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0634238Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0634496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0634570Z layer_outputs = layer_module( 2025-08-14T21:51:06.0634810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0634893Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0635155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0635241Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0635504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0635622Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0635963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0636083Z return self.weight * hidden_states 2025-08-14T21:51:06.0636096Z 2025-08-14T21:51:06.0636206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0636419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0636496Z return mod(**inputs) 2025-08-14T21:51:06.0636755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0636833Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0637105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0637179Z layer_outputs = layer_module( 2025-08-14T21:51:06.0637438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0637519Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0637767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0637860Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0638163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0638267Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0638551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0638633Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0638641Z 2025-08-14T21:51:06.0638756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0638961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0639029Z return mod(**inputs) 2025-08-14T21:51:06.0639318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0639399Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0639651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0639736Z layer_outputs = layer_module( 2025-08-14T21:51:06.0640016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0640105Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0640366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0640449Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0640715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0640802Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0641062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0641143Z key_states = self.k(current_states) 2025-08-14T21:51:06.0641148Z 2025-08-14T21:51:06.0641256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0641487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0641556Z return mod(**inputs) 2025-08-14T21:51:06.0641810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0641895Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0642148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0642230Z layer_outputs = layer_module( 2025-08-14T21:51:06.0642472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0642578Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0642835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0642920Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0643182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0643266Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0643532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0643674Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0643678Z 2025-08-14T21:51:06.0643794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0644002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0644083Z return mod(**inputs) 2025-08-14T21:51:06.0644333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0644436Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0644706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0644781Z layer_outputs = layer_module( 2025-08-14T21:51:06.0645015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0645096Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0645344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0645434Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0645699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0645794Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0646046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0646212Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0646217Z 2025-08-14T21:51:06.0646333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0646544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0646620Z return mod(**inputs) 2025-08-14T21:51:06.0646872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0646948Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0647209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0647284Z layer_outputs = layer_module( 2025-08-14T21:51:06.0647517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0647609Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0647861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0647955Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0648208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0648293Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0648551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0648632Z value_states = self.v(current_states) 2025-08-14T21:51:06.0648657Z 2025-08-14T21:51:06.0648773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0648981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0649050Z return mod(**inputs) 2025-08-14T21:51:06.0649312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0649385Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0649639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0649715Z layer_outputs = layer_module( 2025-08-14T21:51:06.0649930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0650011Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0650251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0650329Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0650573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0650673Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0650928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0651045Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0651049Z 2025-08-14T21:51:06.0651148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0651360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0651429Z return mod(**inputs) 2025-08-14T21:51:06.0651683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0651786Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0652037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0652117Z layer_outputs = layer_module( 2025-08-14T21:51:06.0652343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0652423Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0652677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0652761Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0653006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0653110Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0653344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0653457Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0653462Z 2025-08-14T21:51:06.0653562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0653759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0653837Z return mod(**inputs) 2025-08-14T21:51:06.0654087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0654170Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0654420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0654505Z layer_outputs = layer_module( 2025-08-14T21:51:06.0654726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0654828Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0655061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0655149Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0655382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0655469Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0655700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0655805Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0655808Z 2025-08-14T21:51:06.0655915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0656110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0656186Z return mod(**inputs) 2025-08-14T21:51:06.0656422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0656512Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0656771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0656845Z layer_outputs = layer_module( 2025-08-14T21:51:06.0657059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0657145Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0657377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0657467Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0657732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0657821Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0658078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0658159Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0658163Z 2025-08-14T21:51:06.0658249Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0658366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0658578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0658653Z return mod(**inputs) 2025-08-14T21:51:06.0658905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0658981Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0659241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0659318Z layer_outputs = layer_module( 2025-08-14T21:51:06.0659554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0659638Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0659886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0659987Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0660220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:51:06.0660325Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0660567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0660643Z return self.weight * hidden_states 2025-08-14T21:51:06.0660671Z 2025-08-14T21:51:06.0660781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0660980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0661047Z return mod(**inputs) 2025-08-14T21:51:06.0661295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0661369Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0661605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0661682Z layer_outputs = layer_module( 2025-08-14T21:51:06.0661898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0661981Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0662221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0662302Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0662543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0662644Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0662899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0662979Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0662983Z 2025-08-14T21:51:06.0663086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0663301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0663368Z return mod(**inputs) 2025-08-14T21:51:06.0663616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0663719Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0663973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0664054Z layer_outputs = layer_module( 2025-08-14T21:51:06.0664284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0664363Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0664619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0664703Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0664949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0665045Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0665291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0665378Z key_states = self.k(current_states) 2025-08-14T21:51:06.0665382Z 2025-08-14T21:51:06.0665489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0665696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0665768Z return mod(**inputs) 2025-08-14T21:51:06.0666001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0666080Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0666330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0666403Z layer_outputs = layer_module( 2025-08-14T21:51:06.0666636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0666737Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0666983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0667075Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0667322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0667417Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0667662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0667798Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0667801Z 2025-08-14T21:51:06.0667916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0668124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0668203Z return mod(**inputs) 2025-08-14T21:51:06.0668456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0668550Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0668807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0668896Z layer_outputs = layer_module( 2025-08-14T21:51:06.0669128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0669218Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0669463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0669553Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0669800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0669905Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0670163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0670326Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0670330Z 2025-08-14T21:51:06.0670797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0671005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0671074Z return mod(**inputs) 2025-08-14T21:51:06.0671333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0671410Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0671661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0671748Z layer_outputs = layer_module( 2025-08-14T21:51:06.0671977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0672067Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0672318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0672403Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0672660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0672749Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0673005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0673087Z value_states = self.v(current_states) 2025-08-14T21:51:06.0673091Z 2025-08-14T21:51:06.0673196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0673451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0673520Z return mod(**inputs) 2025-08-14T21:51:06.0673774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0673860Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0674109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0674190Z layer_outputs = layer_module( 2025-08-14T21:51:06.0674419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0674500Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0674753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0674838Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0675083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0675207Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0675474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0675602Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0675606Z 2025-08-14T21:51:06.0675803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0676025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0676106Z return mod(**inputs) 2025-08-14T21:51:06.0676364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0676450Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0676733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0676813Z layer_outputs = layer_module( 2025-08-14T21:51:06.0677064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0677147Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0677394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0677486Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0677731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0677826Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0678072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0678190Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0678194Z 2025-08-14T21:51:06.0678310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0678520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0678598Z return mod(**inputs) 2025-08-14T21:51:06.0678849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0678924Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0679179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0679264Z layer_outputs = layer_module( 2025-08-14T21:51:06.0679584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0679701Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0680097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0680191Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0680437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0680525Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0680776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0680887Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0680891Z 2025-08-14T21:51:06.0681004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0681212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0681280Z return mod(**inputs) 2025-08-14T21:51:06.0681540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0681616Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0681893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0681995Z layer_outputs = layer_module( 2025-08-14T21:51:06.0682225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0682316Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0682565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0682649Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0682907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0683013Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0683261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0683349Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0683354Z 2025-08-14T21:51:06.0683459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0683676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0683743Z return mod(**inputs) 2025-08-14T21:51:06.0683996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0684079Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0684330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0684411Z layer_outputs = layer_module( 2025-08-14T21:51:06.0684644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0684726Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0684980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0685065Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0685314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 524, in forward 2025-08-14T21:51:06.0685457Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:51:06.0685461Z 2025-08-14T21:51:06.0685544Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0685656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0685862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0685930Z return mod(**inputs) 2025-08-14T21:51:06.0686218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0686294Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0686549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0686634Z layer_outputs = layer_module( 2025-08-14T21:51:06.0686863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0686950Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0687198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0687293Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0687548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0687650Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0687904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0688006Z return self.weight * hidden_states 2025-08-14T21:51:06.0688010Z 2025-08-14T21:51:06.0688142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0688359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0688427Z return mod(**inputs) 2025-08-14T21:51:06.0688676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0688760Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0689013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0689094Z layer_outputs = layer_module( 2025-08-14T21:51:06.0689344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0689429Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0689685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0689782Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0690037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0690161Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0690409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0690523Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0690526Z 2025-08-14T21:51:06.0690633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0690841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0690919Z return mod(**inputs) 2025-08-14T21:51:06.0691171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0691256Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0691508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0691583Z layer_outputs = layer_module( 2025-08-14T21:51:06.0691826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0691903Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0692138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0692254Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0692488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0692610Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0692843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0692922Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0692925Z 2025-08-14T21:51:06.0693033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0693228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0693299Z return mod(**inputs) 2025-08-14T21:51:06.0693534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0693606Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0693851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0693938Z layer_outputs = layer_module( 2025-08-14T21:51:06.0694154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0694258Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0694510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0694609Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0694855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0694973Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0695245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0695340Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0695343Z 2025-08-14T21:51:06.0695453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0695661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0695731Z return mod(**inputs) 2025-08-14T21:51:06.0695987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0696060Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0696311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0696391Z layer_outputs = layer_module( 2025-08-14T21:51:06.0696616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0696706Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0696957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0697046Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0697286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0697396Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0697636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0697715Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0697718Z 2025-08-14T21:51:06.0697797Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0697903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0698098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0698196Z return mod(**inputs) 2025-08-14T21:51:06.0698444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0698518Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0698765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0698836Z layer_outputs = layer_module( 2025-08-14T21:51:06.0699052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0699139Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0699375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0699457Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0699706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0699811Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0700054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0700151Z return self.weight * hidden_states 2025-08-14T21:51:06.0700171Z 2025-08-14T21:51:06.0700272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0700480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0700545Z return mod(**inputs) 2025-08-14T21:51:06.0700791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0700862Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0701098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0701195Z layer_outputs = layer_module( 2025-08-14T21:51:06.0701413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0701492Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0701758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0701842Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0702151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0702249Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0702487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0702573Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0702576Z 2025-08-14T21:51:06.0702679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0702882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0702950Z return mod(**inputs) 2025-08-14T21:51:06.0703202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0703286Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0703564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0703633Z layer_outputs = layer_module( 2025-08-14T21:51:06.0703858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0703934Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0704183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0704287Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0704519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0704612Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0704851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0704927Z key_states = self.k(current_states) 2025-08-14T21:51:06.0704938Z 2025-08-14T21:51:06.0705040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0705234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0705309Z return mod(**inputs) 2025-08-14T21:51:06.0705559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0705636Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0705898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0705991Z layer_outputs = layer_module( 2025-08-14T21:51:06.0706235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0706332Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0706591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0706681Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0706938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0707025Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0707279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0707437Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0707442Z 2025-08-14T21:51:06.0707550Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0707745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0707811Z return mod(**inputs) 2025-08-14T21:51:06.0708063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0708138Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0708388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0708469Z layer_outputs = layer_module( 2025-08-14T21:51:06.0708881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0709018Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0709395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0709484Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0709745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0709831Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0710089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0710252Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0710256Z 2025-08-14T21:51:06.0710365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0710579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0710647Z return mod(**inputs) 2025-08-14T21:51:06.0710961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0711042Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0711294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0711377Z layer_outputs = layer_module( 2025-08-14T21:51:06.0711609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0711689Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0711944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0712027Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0712280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0712369Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0712614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0712734Z value_states = self.v(current_states) 2025-08-14T21:51:06.0712737Z 2025-08-14T21:51:06.0712874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0713089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0713164Z return mod(**inputs) 2025-08-14T21:51:06.0713422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0713506Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0713770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0713843Z layer_outputs = layer_module( 2025-08-14T21:51:06.0714104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0714187Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0714434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0714526Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0714771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0714863Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0715110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0715224Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0715228Z 2025-08-14T21:51:06.0715341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0715548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0715623Z return mod(**inputs) 2025-08-14T21:51:06.0715927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0716005Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0716269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0716342Z layer_outputs = layer_module( 2025-08-14T21:51:06.0716571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0716661Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0716905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0716995Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0717264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0717352Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0717608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0717720Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0717724Z 2025-08-14T21:51:06.0717839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0718043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0718111Z return mod(**inputs) 2025-08-14T21:51:06.0718366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0718442Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0718694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0718779Z layer_outputs = layer_module( 2025-08-14T21:51:06.0719030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0719135Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0719382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0719466Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0719718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0719801Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0720045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0720220Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0720224Z 2025-08-14T21:51:06.0720333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0720549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0720617Z return mod(**inputs) 2025-08-14T21:51:06.0720869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0720951Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0721200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0721282Z layer_outputs = layer_module( 2025-08-14T21:51:06.0721509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0721591Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0721849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0721933Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0722185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0722273Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0722500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0722583Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0722586Z 2025-08-14T21:51:06.0722663Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0722761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0722958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0723020Z return mod(**inputs) 2025-08-14T21:51:06.0723282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0723353Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0723590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0723669Z layer_outputs = layer_module( 2025-08-14T21:51:06.0723885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0723961Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0724205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0724284Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0724525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:51:06.0724633Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0724868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0724978Z return self.weight * hidden_states 2025-08-14T21:51:06.0724982Z 2025-08-14T21:51:06.0725095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0725288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0725357Z return mod(**inputs) 2025-08-14T21:51:06.0725587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0725662Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0725891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0725959Z layer_outputs = layer_module( 2025-08-14T21:51:06.0726198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0726274Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0726516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0726598Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0726839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0726925Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0727149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0727223Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0727226Z 2025-08-14T21:51:06.0727333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0727525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0727593Z return mod(**inputs) 2025-08-14T21:51:06.0727821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0727892Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0728129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0728198Z layer_outputs = layer_module( 2025-08-14T21:51:06.0728406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0728487Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0728713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0728799Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0729044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0729124Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0729360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0729435Z key_states = self.k(current_states) 2025-08-14T21:51:06.0729438Z 2025-08-14T21:51:06.0729546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0729735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0729798Z return mod(**inputs) 2025-08-14T21:51:06.0730039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0730111Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0730342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0730418Z layer_outputs = layer_module( 2025-08-14T21:51:06.0730626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0730734Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0730977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0731056Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0731293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0731373Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0731609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0731736Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0731755Z 2025-08-14T21:51:06.0731856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0732053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0732118Z return mod(**inputs) 2025-08-14T21:51:06.0732350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0732427Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0732656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0732730Z layer_outputs = layer_module( 2025-08-14T21:51:06.0732936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0733011Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0733247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0733325Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0733551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0733638Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0733862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0734016Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0734020Z 2025-08-14T21:51:06.0734117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0734304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0734376Z return mod(**inputs) 2025-08-14T21:51:06.0734605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0734698Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0734931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0735000Z layer_outputs = layer_module( 2025-08-14T21:51:06.0735221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0735296Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0735533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0735620Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0735857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0735958Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0736190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0736281Z value_states = self.v(current_states) 2025-08-14T21:51:06.0736284Z 2025-08-14T21:51:06.0736392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0736597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0736667Z return mod(**inputs) 2025-08-14T21:51:06.0736898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0736968Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0737205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0737272Z layer_outputs = layer_module( 2025-08-14T21:51:06.0737497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0737580Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0737805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0737890Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0738119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0738197Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0738430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0738537Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0738541Z 2025-08-14T21:51:06.0738647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0738841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0738907Z return mod(**inputs) 2025-08-14T21:51:06.0739159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0739230Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0739459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0739536Z layer_outputs = layer_module( 2025-08-14T21:51:06.0739743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0739822Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0740047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0740123Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0740358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0740454Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0740684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0740794Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0740797Z 2025-08-14T21:51:06.0740893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0741090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0741154Z return mod(**inputs) 2025-08-14T21:51:06.0741387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0741462Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0741697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0741773Z layer_outputs = layer_module( 2025-08-14T21:51:06.0741984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0742079Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0742347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0742428Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0742661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0742749Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0742984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0743096Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0743102Z 2025-08-14T21:51:06.0743220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0743424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0743502Z return mod(**inputs) 2025-08-14T21:51:06.0743756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0743840Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0744096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0744165Z layer_outputs = layer_module( 2025-08-14T21:51:06.0744387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0744463Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0744700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0744789Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0745025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0745114Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0745359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0745434Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0745438Z 2025-08-14T21:51:06.0745522Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0745621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0745810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0745881Z return mod(**inputs) 2025-08-14T21:51:06.0746118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0746216Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0746455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0746523Z layer_outputs = layer_module( 2025-08-14T21:51:06.0746746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0746819Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0747051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0747147Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0747379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0747478Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0747713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0747803Z return self.weight * hidden_states 2025-08-14T21:51:06.0747806Z 2025-08-14T21:51:06.0747911Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0748112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0748184Z return mod(**inputs) 2025-08-14T21:51:06.0748416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0748486Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0748726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0748794Z layer_outputs = layer_module( 2025-08-14T21:51:06.0749020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0749102Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0749329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0749424Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0749656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0749771Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0750011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0750111Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0750114Z 2025-08-14T21:51:06.0750219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0750416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0750483Z return mod(**inputs) 2025-08-14T21:51:06.0750724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0750798Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0751036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0751114Z layer_outputs = layer_module( 2025-08-14T21:51:06.0751329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0751413Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0751646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0751735Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0752000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0752121Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0752376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0752459Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0752462Z 2025-08-14T21:51:06.0752566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0752776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0752843Z return mod(**inputs) 2025-08-14T21:51:06.0753091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0753175Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0753426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0753507Z layer_outputs = layer_module( 2025-08-14T21:51:06.0753735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0753837Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0754111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0754208Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0754463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0754591Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0754846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0754950Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0754969Z 2025-08-14T21:51:06.0755081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0755295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0755375Z return mod(**inputs) 2025-08-14T21:51:06.0755633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0755792Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0756058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0756135Z layer_outputs = layer_module( 2025-08-14T21:51:06.0756378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0756462Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0756718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0756823Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0757081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0757212Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0757473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0757555Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0757559Z 2025-08-14T21:51:06.0757669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0757862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0757936Z return mod(**inputs) 2025-08-14T21:51:06.0758174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0758272Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0758518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0758595Z layer_outputs = layer_module( 2025-08-14T21:51:06.0758825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0758915Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0759162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0759262Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0759514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-08-14T21:51:06.0759649Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-08-14T21:51:06.0759656Z 2025-08-14T21:51:06.0759749Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0759856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0760094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0760161Z return mod(**inputs) 2025-08-14T21:51:06.0760428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0760513Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0760769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0760843Z layer_outputs = layer_module( 2025-08-14T21:51:06.0761094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0761171Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0761433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0761518Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0761750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0761862Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0762111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0762185Z return self.weight * hidden_states 2025-08-14T21:51:06.0762197Z 2025-08-14T21:51:06.0762299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0762506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0762580Z return mod(**inputs) 2025-08-14T21:51:06.0762835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0762910Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0763168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0763241Z layer_outputs = layer_module( 2025-08-14T21:51:06.0763486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0763568Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0763822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0763914Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0764168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0764287Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0764532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0764610Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0764614Z 2025-08-14T21:51:06.0764724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0764921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0764986Z return mod(**inputs) 2025-08-14T21:51:06.0765233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0765304Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0765540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0765616Z layer_outputs = layer_module( 2025-08-14T21:51:06.0765834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0765921Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0766186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0766265Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0766523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0766607Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0766844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0766918Z key_states = self.k(current_states) 2025-08-14T21:51:06.0766922Z 2025-08-14T21:51:06.0767021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0767237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0767306Z return mod(**inputs) 2025-08-14T21:51:06.0767543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0767622Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0767860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0767937Z layer_outputs = layer_module( 2025-08-14T21:51:06.0768154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0768229Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0768470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0768549Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0768794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0768878Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0769112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0769247Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0769252Z 2025-08-14T21:51:06.0769353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0769551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0769622Z return mod(**inputs) 2025-08-14T21:51:06.0769857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0769935Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0770172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0770261Z layer_outputs = layer_module( 2025-08-14T21:51:06.0770483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0770560Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0770793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0770878Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0771110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0771198Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0771429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0771581Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0771588Z 2025-08-14T21:51:06.0771695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0771888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0771979Z return mod(**inputs) 2025-08-14T21:51:06.0772230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0772304Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0772550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0772621Z layer_outputs = layer_module( 2025-08-14T21:51:06.0772844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0772932Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0773196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0773289Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0773540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0773625Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0773888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0773969Z value_states = self.v(current_states) 2025-08-14T21:51:06.0773973Z 2025-08-14T21:51:06.0774087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0774294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0774362Z return mod(**inputs) 2025-08-14T21:51:06.0774618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0774696Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0774945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0775029Z layer_outputs = layer_module( 2025-08-14T21:51:06.0775258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0775349Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0775596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0775681Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0775934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0776017Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0776265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0776407Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0776412Z 2025-08-14T21:51:06.0776519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0776734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0776801Z return mod(**inputs) 2025-08-14T21:51:06.0777051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0777137Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0777384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0777464Z layer_outputs = layer_module( 2025-08-14T21:51:06.0777691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0777777Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0778034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0778135Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0778403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0778497Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0778743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0778862Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0778867Z 2025-08-14T21:51:06.0778972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0779176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0779272Z return mod(**inputs) 2025-08-14T21:51:06.0779526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0779610Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0779859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0779932Z layer_outputs = layer_module( 2025-08-14T21:51:06.0780165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0780256Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0780486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0780573Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0780807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0780895Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0781132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0781239Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0781244Z 2025-08-14T21:51:06.0781350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0781547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0781620Z return mod(**inputs) 2025-08-14T21:51:06.0781855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0781926Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0782176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0782275Z layer_outputs = layer_module( 2025-08-14T21:51:06.0782502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0782592Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0782840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0782932Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0783177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0783264Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0783517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0783597Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0783601Z 2025-08-14T21:51:06.0783685Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0783799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0784002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0784095Z return mod(**inputs) 2025-08-14T21:51:06.0784370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0784448Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0784704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0784777Z layer_outputs = layer_module( 2025-08-14T21:51:06.0785009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0785098Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0785360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0785456Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0785702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:51:06.0785813Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0786070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0786149Z return self.weight * hidden_states 2025-08-14T21:51:06.0786153Z 2025-08-14T21:51:06.0786270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0786478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0786545Z return mod(**inputs) 2025-08-14T21:51:06.0786801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0786881Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0787135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0787218Z layer_outputs = layer_module( 2025-08-14T21:51:06.0787448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0787539Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0787787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0787872Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0788129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0788216Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0788469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0788577Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0788582Z 2025-08-14T21:51:06.0788691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0788912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0788979Z return mod(**inputs) 2025-08-14T21:51:06.0789236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0789322Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0789578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0789743Z layer_outputs = layer_module( 2025-08-14T21:51:06.0790005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0790115Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0790419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0790547Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0791078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0791227Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0791529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0791638Z key_states = self.k(current_states) 2025-08-14T21:51:06.0791642Z 2025-08-14T21:51:06.0791812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0792044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0792173Z return mod(**inputs) 2025-08-14T21:51:06.0792515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0792617Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0792899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0793039Z layer_outputs = layer_module( 2025-08-14T21:51:06.0793277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0793457Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0793728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0793836Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0794153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0794268Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0794586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0794762Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0794767Z 2025-08-14T21:51:06.0794899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0795169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0795262Z return mod(**inputs) 2025-08-14T21:51:06.0795543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0795777Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0796081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0796245Z layer_outputs = layer_module( 2025-08-14T21:51:06.0796506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0796631Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0796926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0797056Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0797368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0797486Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0797758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0797978Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0797982Z 2025-08-14T21:51:06.0798103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0798392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0798511Z return mod(**inputs) 2025-08-14T21:51:06.0798809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0798940Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0799212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0799298Z layer_outputs = layer_module( 2025-08-14T21:51:06.0799614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0799723Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0800023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0800166Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0800438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0800590Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0800875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0801007Z value_states = self.v(current_states) 2025-08-14T21:51:06.0801012Z 2025-08-14T21:51:06.0801143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0801373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0801482Z return mod(**inputs) 2025-08-14T21:51:06.0801782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0801921Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0802197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0802299Z layer_outputs = layer_module( 2025-08-14T21:51:06.0802581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0802679Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0802971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0803116Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0803398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0803535Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0803824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0803970Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0803975Z 2025-08-14T21:51:06.0804163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0804396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0804531Z return mod(**inputs) 2025-08-14T21:51:06.0804815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0804918Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0805227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0805339Z layer_outputs = layer_module( 2025-08-14T21:51:06.0805599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0805734Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0806013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0806157Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0806483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0806606Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0806926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0807060Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0807064Z 2025-08-14T21:51:06.0807226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0807446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0807568Z return mod(**inputs) 2025-08-14T21:51:06.0807913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0808013Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0808323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0808422Z layer_outputs = layer_module( 2025-08-14T21:51:06.0808915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0809167Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0809511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0809619Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0809940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0810058Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0810374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0810532Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0810537Z 2025-08-14T21:51:06.0810669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0810933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0811027Z return mod(**inputs) 2025-08-14T21:51:06.0811334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0811453Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0811756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0811886Z layer_outputs = layer_module( 2025-08-14T21:51:06.0812206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0812344Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0812603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0812739Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0813050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0813159Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0813430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0813564Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0813569Z 2025-08-14T21:51:06.0813664Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0813887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0814117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0814243Z return mod(**inputs) 2025-08-14T21:51:06.0814551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0814683Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0814991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0815093Z layer_outputs = layer_module( 2025-08-14T21:51:06.0815325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0815451Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0815718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0815861Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0816167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0816294Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0816569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0816672Z return self.weight * hidden_states 2025-08-14T21:51:06.0816676Z 2025-08-14T21:51:06.0816797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0817023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0817132Z return mod(**inputs) 2025-08-14T21:51:06.0817424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0817520Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0817785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0817909Z layer_outputs = layer_module( 2025-08-14T21:51:06.0818150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0818313Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0818565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0818682Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0818966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0819105Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0819350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0819549Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0819552Z 2025-08-14T21:51:06.0819690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0819937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0820026Z return mod(**inputs) 2025-08-14T21:51:06.0820284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0820423Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0820690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0820817Z layer_outputs = layer_module( 2025-08-14T21:51:06.0821057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0821170Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0821435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0821587Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0821895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0822030Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0822278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0822423Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0822427Z 2025-08-14T21:51:06.0822535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0822772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0822899Z return mod(**inputs) 2025-08-14T21:51:06.0823176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0823302Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0823564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0823650Z layer_outputs = layer_module( 2025-08-14T21:51:06.0823949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0824049Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0824347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0824482Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0824758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0824944Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0825212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0825350Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0825354Z 2025-08-14T21:51:06.0825480Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0825700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0825810Z return mod(**inputs) 2025-08-14T21:51:06.0826088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0826194Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0826480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0826615Z layer_outputs = layer_module( 2025-08-14T21:51:06.0826889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0826983Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0827260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0827407Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0827660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0827827Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0828080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0828170Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0828174Z 2025-08-14T21:51:06.0828344Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0828469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0828695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0828828Z return mod(**inputs) 2025-08-14T21:51:06.0829098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0829249Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0829516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0829617Z layer_outputs = layer_module( 2025-08-14T21:51:06.0829888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0829988Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0830278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0830404Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0830692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0830849Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0831109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0831235Z return self.weight * hidden_states 2025-08-14T21:51:06.0831239Z 2025-08-14T21:51:06.0831349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0831587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0831728Z return mod(**inputs) 2025-08-14T21:51:06.0832004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0832109Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0832415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0832504Z layer_outputs = layer_module( 2025-08-14T21:51:06.0832824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0832929Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0833199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0833338Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0833611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0833772Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0834057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0834182Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0834188Z 2025-08-14T21:51:06.0834347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0834576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0834669Z return mod(**inputs) 2025-08-14T21:51:06.0834994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0835128Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0835432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0835530Z layer_outputs = layer_module( 2025-08-14T21:51:06.0835847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0835990Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0836308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0836479Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0836774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0836892Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0837213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0837311Z key_states = self.k(current_states) 2025-08-14T21:51:06.0837315Z 2025-08-14T21:51:06.0837504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0837722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0837810Z return mod(**inputs) 2025-08-14T21:51:06.0838122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0838220Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0838467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0850222Z layer_outputs = layer_module( 2025-08-14T21:51:06.0850555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0850656Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0850924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0851015Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0851271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0851374Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0851617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0851763Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0851769Z 2025-08-14T21:51:06.0851886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0852112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0852188Z return mod(**inputs) 2025-08-14T21:51:06.0852454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0852549Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0852809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0852977Z layer_outputs = layer_module( 2025-08-14T21:51:06.0853224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0853316Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0853576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0853664Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0853913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0854014Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0854272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0854439Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0854444Z 2025-08-14T21:51:06.0854562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0854771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0854880Z return mod(**inputs) 2025-08-14T21:51:06.0855121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0855221Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0855466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0855537Z layer_outputs = layer_module( 2025-08-14T21:51:06.0855760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0855841Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0856076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0856196Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0856434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0856528Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0856763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0856840Z value_states = self.v(current_states) 2025-08-14T21:51:06.0856844Z 2025-08-14T21:51:06.0856958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0857162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0857229Z return mod(**inputs) 2025-08-14T21:51:06.0857473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0857545Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0857792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0857866Z layer_outputs = layer_module( 2025-08-14T21:51:06.0858082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0858171Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0858406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0858488Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0858729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0858810Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0859052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0859196Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0859200Z 2025-08-14T21:51:06.0859302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0859505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0859572Z return mod(**inputs) 2025-08-14T21:51:06.0859810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0859881Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0860113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0860193Z layer_outputs = layer_module( 2025-08-14T21:51:06.0860408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0860489Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0860733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0860850Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0861110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0861194Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0861431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0861546Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0861550Z 2025-08-14T21:51:06.0861652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0861858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0861925Z return mod(**inputs) 2025-08-14T21:51:06.0862186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0862268Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0862505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0862577Z layer_outputs = layer_module( 2025-08-14T21:51:06.0862800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0862877Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0863118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0863197Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0863428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0863522Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0863753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0863870Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0863873Z 2025-08-14T21:51:06.0863975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0864171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0864245Z return mod(**inputs) 2025-08-14T21:51:06.0864483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0864555Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0864797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0864867Z layer_outputs = layer_module( 2025-08-14T21:51:06.0865111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0865189Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0865429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0865520Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0865755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0865835Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0866078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0866156Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0866160Z 2025-08-14T21:51:06.0866269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0866471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0866537Z return mod(**inputs) 2025-08-14T21:51:06.0866786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0866877Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0867137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0867211Z layer_outputs = layer_module( 2025-08-14T21:51:06.0867430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0867518Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0867751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0867832Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0868117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:51:06.0868253Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:51:06.0868258Z 2025-08-14T21:51:06.0868351Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0868455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0868656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0868730Z return mod(**inputs) 2025-08-14T21:51:06.0868970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0869042Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0869287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0869358Z layer_outputs = layer_module( 2025-08-14T21:51:06.0869582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0869662Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0869897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0869986Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0870230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:51:06.0870346Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0870596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0870677Z return self.weight * hidden_states 2025-08-14T21:51:06.0870681Z 2025-08-14T21:51:06.0870796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0871025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0871096Z return mod(**inputs) 2025-08-14T21:51:06.0871362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0871439Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0871702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0871776Z layer_outputs = layer_module( 2025-08-14T21:51:06.0872010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0872099Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0872351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0872444Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0872697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0872802Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0873074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0873159Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0873163Z 2025-08-14T21:51:06.0873270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0873487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0873557Z return mod(**inputs) 2025-08-14T21:51:06.0873817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0873892Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0874161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0874248Z layer_outputs = layer_module( 2025-08-14T21:51:06.0874482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0874568Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0874826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0874912Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0875171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0875263Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0875517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0875613Z key_states = self.k(current_states) 2025-08-14T21:51:06.0875618Z 2025-08-14T21:51:06.0875847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0876082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0876155Z return mod(**inputs) 2025-08-14T21:51:06.0876416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0876503Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0876761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0876839Z layer_outputs = layer_module( 2025-08-14T21:51:06.0877083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0877167Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0877461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0877544Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0877781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0877877Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0878113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0878251Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0878255Z 2025-08-14T21:51:06.0878359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0878555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0878628Z return mod(**inputs) 2025-08-14T21:51:06.0878881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0878959Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0879219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0879311Z layer_outputs = layer_module( 2025-08-14T21:51:06.0879565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0879648Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0879901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0879995Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0880241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0880328Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0880601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0880769Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0880775Z 2025-08-14T21:51:06.0880887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0881088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0881159Z return mod(**inputs) 2025-08-14T21:51:06.0881423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0881500Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0881762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0881837Z layer_outputs = layer_module( 2025-08-14T21:51:06.0882072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0882163Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0882413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0882499Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0882758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0882845Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0883103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0883187Z value_states = self.v(current_states) 2025-08-14T21:51:06.0883192Z 2025-08-14T21:51:06.0883299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0883516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0883624Z return mod(**inputs) 2025-08-14T21:51:06.0883869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0883943Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0884183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0884264Z layer_outputs = layer_module( 2025-08-14T21:51:06.0884495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0884575Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0884831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0884913Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0885170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0885256Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0885521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0885658Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0885663Z 2025-08-14T21:51:06.0885770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0885984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0886052Z return mod(**inputs) 2025-08-14T21:51:06.0886303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0886385Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0886653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0886732Z layer_outputs = layer_module( 2025-08-14T21:51:06.0886968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0887050Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0887309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0887394Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0887644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0887738Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0887998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0888110Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0888117Z 2025-08-14T21:51:06.0888223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0888435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0888506Z return mod(**inputs) 2025-08-14T21:51:06.0888762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0888845Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0889101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0889183Z layer_outputs = layer_module( 2025-08-14T21:51:06.0889413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0889494Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0889753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0889855Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0890103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0890198Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0890444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0890564Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0890568Z 2025-08-14T21:51:06.0890674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0890881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0890959Z return mod(**inputs) 2025-08-14T21:51:06.0891211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0891296Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0891546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0891637Z layer_outputs = layer_module( 2025-08-14T21:51:06.0892119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0892205Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0892457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0892550Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0892799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0892894Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0893169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0893253Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0893258Z 2025-08-14T21:51:06.0893356Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0893466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0893679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0893748Z return mod(**inputs) 2025-08-14T21:51:06.0893999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0894084Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0894337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0894414Z layer_outputs = layer_module( 2025-08-14T21:51:06.0894656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0894736Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0894988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0895084Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0895332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0895441Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0895687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0895767Z return self.weight * hidden_states 2025-08-14T21:51:06.0895778Z 2025-08-14T21:51:06.0895884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0896092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0896189Z return mod(**inputs) 2025-08-14T21:51:06.0896444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0896521Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0896784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0896857Z layer_outputs = layer_module( 2025-08-14T21:51:06.0897094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0897176Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0897424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0897528Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0897782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0897910Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0898188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0898309Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0898314Z 2025-08-14T21:51:06.0898428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0898635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0898703Z return mod(**inputs) 2025-08-14T21:51:06.0898963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0899039Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0899313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0899392Z layer_outputs = layer_module( 2025-08-14T21:51:06.0899625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0899713Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0899961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0900056Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0900312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0900436Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0900691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0900777Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0900780Z 2025-08-14T21:51:06.0900887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0901106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0901175Z return mod(**inputs) 2025-08-14T21:51:06.0901425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0901507Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0901762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0901842Z layer_outputs = layer_module( 2025-08-14T21:51:06.0902071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0902152Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0902425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0902513Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0902755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0902869Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0903104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0903200Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0903204Z 2025-08-14T21:51:06.0903305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0903501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0903575Z return mod(**inputs) 2025-08-14T21:51:06.0903812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0903895Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0904128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0904220Z layer_outputs = layer_module( 2025-08-14T21:51:06.0904481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0904560Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0904810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0904901Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0905147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0905271Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0905544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0905631Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0905635Z 2025-08-14T21:51:06.0905726Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0905834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0906050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0906118Z return mod(**inputs) 2025-08-14T21:51:06.0906373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0906458Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0906709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0906786Z layer_outputs = layer_module( 2025-08-14T21:51:06.0907029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0907114Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0907378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0907462Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0907695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:51:06.0907812Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0908045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0908128Z return self.weight * hidden_states 2025-08-14T21:51:06.0908131Z 2025-08-14T21:51:06.0908233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0908450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0908526Z return mod(**inputs) 2025-08-14T21:51:06.0909066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0909178Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0909444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0909519Z layer_outputs = layer_module( 2025-08-14T21:51:06.0909753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0909840Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0910101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0910198Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0910455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0910613Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0910903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0910987Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0910991Z 2025-08-14T21:51:06.0911110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0911325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0911401Z return mod(**inputs) 2025-08-14T21:51:06.0911659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0911733Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0912030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0912106Z layer_outputs = layer_module( 2025-08-14T21:51:06.0912355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0912447Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0912712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0912804Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0913058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0913146Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0913405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0913488Z key_states = self.k(current_states) 2025-08-14T21:51:06.0913492Z 2025-08-14T21:51:06.0913609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0913825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0913896Z return mod(**inputs) 2025-08-14T21:51:06.0914172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0914248Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0914517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0914599Z layer_outputs = layer_module( 2025-08-14T21:51:06.0914836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0914925Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0915215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0915300Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0915564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0915650Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0915972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0916127Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0916131Z 2025-08-14T21:51:06.0916243Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0916464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0916535Z return mod(**inputs) 2025-08-14T21:51:06.0916808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0916893Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0917164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0917270Z layer_outputs = layer_module( 2025-08-14T21:51:06.0917538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0917623Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0917957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0918037Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0918274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0918363Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0918637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0918813Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0918819Z 2025-08-14T21:51:06.0918929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0919152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0919231Z return mod(**inputs) 2025-08-14T21:51:06.0919493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0919578Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0919839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0919917Z layer_outputs = layer_module( 2025-08-14T21:51:06.0920167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0920253Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0920511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0920606Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0920862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0920957Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0921218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0921301Z value_states = self.v(current_states) 2025-08-14T21:51:06.0921304Z 2025-08-14T21:51:06.0921423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0921650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0921743Z return mod(**inputs) 2025-08-14T21:51:06.0922010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0922090Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0922355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0922432Z layer_outputs = layer_module( 2025-08-14T21:51:06.0922674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0922766Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0923030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0923124Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0923389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0923474Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0923757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0923889Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0923893Z 2025-08-14T21:51:06.0924005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0924237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0924307Z return mod(**inputs) 2025-08-14T21:51:06.0924576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0924647Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0924902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0924983Z layer_outputs = layer_module( 2025-08-14T21:51:06.0925200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0925283Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0925525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0925604Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0925848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0925927Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0926160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0926274Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0926279Z 2025-08-14T21:51:06.0926381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0926585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0926652Z return mod(**inputs) 2025-08-14T21:51:06.0926894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0926973Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0927210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0927279Z layer_outputs = layer_module( 2025-08-14T21:51:06.0927502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0927578Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0927825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0927921Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0928152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0928241Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0928477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0928590Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0928593Z 2025-08-14T21:51:06.0928695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0928889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0928960Z return mod(**inputs) 2025-08-14T21:51:06.0929196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0929271Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0929513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0929600Z layer_outputs = layer_module( 2025-08-14T21:51:06.0929849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0929928Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0930162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:51:06.0930247Z self_attention_outputs = self.layer[0]( 2025-08-14T21:51:06.0930480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:51:06.0930567Z attention_output = self.SelfAttention( 2025-08-14T21:51:06.0930823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0930904Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0930909Z 2025-08-14T21:51:06.0930996Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0931097Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0931300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0931374Z return mod(**inputs) 2025-08-14T21:51:06.0931611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0931688Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0931925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0931994Z layer_outputs = layer_module( 2025-08-14T21:51:06.0932233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0932315Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0932578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0932670Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0932918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:51:06.0933032Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0933269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0933344Z return self.weight * hidden_states 2025-08-14T21:51:06.0933348Z 2025-08-14T21:51:06.0933457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0933653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0933745Z return mod(**inputs) 2025-08-14T21:51:06.0933981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0934054Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0934298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0934368Z layer_outputs = layer_module( 2025-08-14T21:51:06.0934586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0934670Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0934904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0934989Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0935227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0935317Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0935602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:51:06.0935700Z query_states = self.q(hidden_states) 2025-08-14T21:51:06.0935704Z 2025-08-14T21:51:06.0935829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0936023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0936087Z return mod(**inputs) 2025-08-14T21:51:06.0936332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0936404Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0936675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0936763Z layer_outputs = layer_module( 2025-08-14T21:51:06.0937002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0937091Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0937359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0937444Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0937713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0937801Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0938057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:51:06.0938143Z key_states = self.k(current_states) 2025-08-14T21:51:06.0938148Z 2025-08-14T21:51:06.0938255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0938468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0938538Z return mod(**inputs) 2025-08-14T21:51:06.0938789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0938870Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0939121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0939191Z layer_outputs = layer_module( 2025-08-14T21:51:06.0939414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0939490Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0939728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0939829Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0940063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0940156Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0940386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:51:06.0940520Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:51:06.0940524Z 2025-08-14T21:51:06.0940623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0940820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0940894Z return mod(**inputs) 2025-08-14T21:51:06.0941130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0941206Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0941447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0941537Z layer_outputs = layer_module( 2025-08-14T21:51:06.0941787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0941870Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0942117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0942207Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0942451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0942545Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0942806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:51:06.0942973Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:51:06.0942979Z 2025-08-14T21:51:06.0943094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0943302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0943371Z return mod(**inputs) 2025-08-14T21:51:06.0943629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0943705Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0943965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0944040Z layer_outputs = layer_module( 2025-08-14T21:51:06.0944272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0944362Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0944611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0944704Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0944953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0945040Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0945291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:51:06.0945373Z value_states = self.v(current_states) 2025-08-14T21:51:06.0945377Z 2025-08-14T21:51:06.0945483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0945698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0945788Z return mod(**inputs) 2025-08-14T21:51:06.0946051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0946129Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0946385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0946468Z layer_outputs = layer_module( 2025-08-14T21:51:06.0946699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0946778Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0947036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0947119Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0947379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0947467Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0947720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0947853Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0947857Z 2025-08-14T21:51:06.0947979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0948197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0948264Z return mod(**inputs) 2025-08-14T21:51:06.0948517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0948599Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0948850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0948946Z layer_outputs = layer_module( 2025-08-14T21:51:06.0949182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0949263Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0949519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0949603Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0949857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0949951Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0950197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:51:06.0950317Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:51:06.0950320Z 2025-08-14T21:51:06.0950432Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0950639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0950717Z return mod(**inputs) 2025-08-14T21:51:06.0950968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0951044Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0951300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0951374Z layer_outputs = layer_module( 2025-08-14T21:51:06.0951609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0951689Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0951936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0952052Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0952299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0952387Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0952639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:51:06.0952751Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:51:06.0952754Z 2025-08-14T21:51:06.0952867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0953073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0953142Z return mod(**inputs) 2025-08-14T21:51:06.0953398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0953475Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0953744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0953852Z layer_outputs = layer_module( 2025-08-14T21:51:06.0954104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0954197Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0954452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0954537Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0954798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:51:06.0954886Z attention_output = self.EncDecAttention( 2025-08-14T21:51:06.0955161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:51:06.0955247Z attn_output = self.o(attn_output) 2025-08-14T21:51:06.0955250Z 2025-08-14T21:51:06.0955358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0955584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0955654Z return mod(**inputs) 2025-08-14T21:51:06.0956001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0956087Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0956343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0956428Z layer_outputs = layer_module( 2025-08-14T21:51:06.0956664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0956747Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0957016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:51:06.0957104Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:51:06.0957374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 524, in forward 2025-08-14T21:51:06.0957513Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:51:06.0957517Z 2025-08-14T21:51:06.0957603Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0957719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0957926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0957995Z return mod(**inputs) 2025-08-14T21:51:06.0958253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0958360Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0958621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0958697Z layer_outputs = layer_module( 2025-08-14T21:51:06.0958928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0959017Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0959266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0959369Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0959616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:51:06.0959717Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:51:06.0959972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0960053Z return self.weight * hidden_states 2025-08-14T21:51:06.0960057Z 2025-08-14T21:51:06.0960186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0960406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0960485Z return mod(**inputs) 2025-08-14T21:51:06.0960731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0960802Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0961034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0961113Z layer_outputs = layer_module( 2025-08-14T21:51:06.0961335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0961428Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0961663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0961754Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0961991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0962104Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0962332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:51:06.0962434Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:51:06.0962438Z 2025-08-14T21:51:06.0962536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0962731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0962797Z return mod(**inputs) 2025-08-14T21:51:06.0963028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0963106Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0963345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0963415Z layer_outputs = layer_module( 2025-08-14T21:51:06.0963636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0963712Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0963961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0964045Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0964274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0964411Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0964636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:51:06.0964721Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:51:06.0964724Z 2025-08-14T21:51:06.0964822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0965008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0965076Z return mod(**inputs) 2025-08-14T21:51:06.0965304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0965372Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0965609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0965679Z layer_outputs = layer_module( 2025-08-14T21:51:06.0965896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0965988Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0966256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0966351Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0966577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0966695Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0966921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:51:06.0967007Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:51:06.0967011Z 2025-08-14T21:51:06.0967137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0967335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0967401Z return mod(**inputs) 2025-08-14T21:51:06.0967649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0967718Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0967958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:51:06.0968026Z layer_outputs = layer_module( 2025-08-14T21:51:06.0968236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:06.0968315Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:06.0968541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:51:06.0968627Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:51:06.0968859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:51:06.0968970Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:51:06.0969206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:51:06.0969287Z hidden_states = self.wo(hidden_states) 2025-08-14T21:51:06.0969291Z 2025-08-14T21:51:06.0969370Z cudagraph partition due to non gpu ops 2025-08-14T21:51:06.0969481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0969681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0969753Z return mod(**inputs) 2025-08-14T21:51:06.0969993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:51:06.0970087Z decoder_outputs = self.decoder( 2025-08-14T21:51:06.0970337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1115, in forward 2025-08-14T21:51:06.0970444Z hidden_states = self.final_layer_norm(hidden_states) 2025-08-14T21:51:06.0970685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:51:06.0970770Z return self.weight * hidden_states 2025-08-14T21:51:06.0970773Z 2025-08-14T21:51:06.0970873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0971077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0971154Z return mod(**inputs) 2025-08-14T21:51:06.0971390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1816, in forward 2025-08-14T21:51:06.0971488Z lm_logits = self.lm_head(sequence_output) 2025-08-14T21:51:06.0971491Z 2025-08-14T21:51:06.0971587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0971803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0971868Z return mod(**inputs) 2025-08-14T21:51:06.0972120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-08-14T21:51:06.0972270Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:51:06.0972274Z 2025-08-14T21:51:06.0972373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0972567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0972639Z return mod(**inputs) 2025-08-14T21:51:06.0972893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-08-14T21:51:06.0973037Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:51:06.0973040Z 2025-08-14T21:51:06.0973142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:06.0973334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:06.0973409Z return mod(**inputs) 2025-08-14T21:51:06.0973649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-08-14T21:51:06.0973783Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:51:06.0973786Z 2025-08-14T21:51:17.0356334Z Compilation time (from dynamo_timed): 22.475517257 2025-08-14T21:51:17.0530023Z pass 2025-08-14T21:51:17.0530462Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:17.0531350Z TIMING: _recursive_pre_grad_passes:0.0157 _recursive_joint_graph_passes:0.7904 _recursive_post_grad_passes:0.6092 async_compile.wait:0.78956 code_gen:10.33272 inductor_compile:13.05462 backend_compile:18.21085 gc:0.00016 entire_frame_compile:22.47552 total_wall_time:22.47552 2025-08-14T21:51:17.0532307Z STATS: call_* op count: 1189 | FakeTensorMode.__torch_dispatch__:29419 | FakeTensor.__torch_dispatch__:8702 | ProxyTorchDispatchMode.__torch_dispatch__:10618 2025-08-14T21:51:17.0532793Z Dynamo produced 1 graphs covering 1189 ops with 0 graph breaks (0 unique) 2025-08-14T21:51:22.6836725Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:51:22.6837755Z from pkg_resources import resource_filename 2025-08-14T21:51:23.3586555Z 2025-08-14T21:51:23.3709633Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-14T21:51:23.3710315Z WARNING:transformers.models.megatron_bert.modeling_megatron_bert:If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-14T21:51:26.6980400Z 2025-08-14T21:51:26.6985077Z loading model: 0it [00:03, ?it/s] 2025-08-14T21:51:26.7005854Z cpu eval MegatronBertForCausalLM 2025-08-14T21:51:28.3675530Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:28.9895136Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:29.6127052Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:44.0893791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0894283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0894637Z return mod(**inputs) 2025-08-14T21:51:44.0895092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0895907Z outputs = self.bert( 2025-08-14T21:51:44.0896444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0896921Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0897373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0897840Z layer_outputs = layer_module( 2025-08-14T21:51:44.0898252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0898652Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0899167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.0899626Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.0900088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.0900513Z self_outputs = self.self( 2025-08-14T21:51:44.0900926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.0901316Z return func(*args, **kwargs) 2025-08-14T21:51:44.0901733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.0902181Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.0902358Z 2025-08-14T21:51:44.0902469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0902840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0903168Z return mod(**inputs) 2025-08-14T21:51:44.0903608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0904074Z outputs = self.bert( 2025-08-14T21:51:44.0904499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0904947Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0905397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0905846Z layer_outputs = layer_module( 2025-08-14T21:51:44.0906208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0906692Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0907145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.0907608Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.0908062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.0908579Z self_outputs = self.self( 2025-08-14T21:51:44.0909211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.0909627Z return func(*args, **kwargs) 2025-08-14T21:51:44.0910060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.0910518Z key_layer = self.key(current_states) 2025-08-14T21:51:44.0910675Z 2025-08-14T21:51:44.0910833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0911214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0911599Z return mod(**inputs) 2025-08-14T21:51:44.0912074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0912538Z outputs = self.bert( 2025-08-14T21:51:44.0912964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0913435Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0913898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0914370Z layer_outputs = layer_module( 2025-08-14T21:51:44.0914768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0915171Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0916017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.0916510Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.0916972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.0917441Z self_outputs = self.self( 2025-08-14T21:51:44.0917839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.0918245Z return func(*args, **kwargs) 2025-08-14T21:51:44.0918678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.0919135Z value_layer = self.value(current_states) 2025-08-14T21:51:44.0919278Z 2025-08-14T21:51:44.0919374Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.0919602Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.0919857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0920239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0920583Z return mod(**inputs) 2025-08-14T21:51:44.0921001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0921444Z outputs = self.bert( 2025-08-14T21:51:44.0921870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0922392Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0922853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0923349Z layer_outputs = layer_module( 2025-08-14T21:51:44.0923712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0924093Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0924554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.0925002Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.0925460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.0925977Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.0926486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.0926944Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.0927091Z 2025-08-14T21:51:44.0927203Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0927603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0927966Z return mod(**inputs) 2025-08-14T21:51:44.0928385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0928812Z outputs = self.bert( 2025-08-14T21:51:44.0929225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0929673Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0930117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0930573Z layer_outputs = layer_module( 2025-08-14T21:51:44.0930941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0931320Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0931742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.0932182Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.0932597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.0933016Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.0933485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.0934000Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.0934493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.0934950Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.0935093Z 2025-08-14T21:51:44.0935204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0935584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0935926Z return mod(**inputs) 2025-08-14T21:51:44.0936346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0936782Z outputs = self.bert( 2025-08-14T21:51:44.0937195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0937633Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0938091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0938535Z layer_outputs = layer_module( 2025-08-14T21:51:44.0938898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0939279Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0939721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.0940174Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.0940607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.0941026Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.0941506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.0942018Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.0942604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.0943117Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.0943539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.0943903Z return self.act(input) 2025-08-14T21:51:44.0944026Z 2025-08-14T21:51:44.0944149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0944538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0944898Z return mod(**inputs) 2025-08-14T21:51:44.0945365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0945811Z outputs = self.bert( 2025-08-14T21:51:44.0946250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0946716Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0947177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0947626Z layer_outputs = layer_module( 2025-08-14T21:51:44.0948006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0948406Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0948873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.0949431Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.0949867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.0950297Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.0950774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.0951325Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.0951841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.0952329Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.0952481Z 2025-08-14T21:51:44.0952604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0952996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0953377Z return mod(**inputs) 2025-08-14T21:51:44.0953810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0954255Z outputs = self.bert( 2025-08-14T21:51:44.0954692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0955149Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0955761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0956222Z layer_outputs = layer_module( 2025-08-14T21:51:44.0956596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0956990Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0957466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.0957953Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.0958456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.0958941Z self_outputs = self.self( 2025-08-14T21:51:44.0959345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.0959769Z return func(*args, **kwargs) 2025-08-14T21:51:44.0960233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.0960712Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.0960866Z 2025-08-14T21:51:44.0960977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0961383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0961727Z return mod(**inputs) 2025-08-14T21:51:44.0962141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0962584Z outputs = self.bert( 2025-08-14T21:51:44.0963001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0963440Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0963872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0964316Z layer_outputs = layer_module( 2025-08-14T21:51:44.0964681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0965052Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0965490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.0965916Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.0966334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.0966738Z self_outputs = self.self( 2025-08-14T21:51:44.0967099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.0967471Z return func(*args, **kwargs) 2025-08-14T21:51:44.0967875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.0968293Z key_layer = self.key(current_states) 2025-08-14T21:51:44.0968441Z 2025-08-14T21:51:44.0968550Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0968951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0969283Z return mod(**inputs) 2025-08-14T21:51:44.0969699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0970151Z outputs = self.bert( 2025-08-14T21:51:44.0970553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0970958Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0971374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0971812Z layer_outputs = layer_module( 2025-08-14T21:51:44.0972169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0972542Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0972982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.0973457Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.0973927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.0974364Z self_outputs = self.self( 2025-08-14T21:51:44.0974744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.0975137Z return func(*args, **kwargs) 2025-08-14T21:51:44.0975562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.0976010Z value_layer = self.value(current_states) 2025-08-14T21:51:44.0976174Z 2025-08-14T21:51:44.0976267Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.0976490Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.0976746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0977125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0977472Z return mod(**inputs) 2025-08-14T21:51:44.0977885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0978331Z outputs = self.bert( 2025-08-14T21:51:44.0978756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0979212Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0979660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0980099Z layer_outputs = layer_module( 2025-08-14T21:51:44.0980468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0980853Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0981368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.0981828Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.0982273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.0982769Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.0983267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.0983750Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.0983899Z 2025-08-14T21:51:44.0984019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0984405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0984757Z return mod(**inputs) 2025-08-14T21:51:44.0985190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0985630Z outputs = self.bert( 2025-08-14T21:51:44.0986058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0986506Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0986949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0987391Z layer_outputs = layer_module( 2025-08-14T21:51:44.0987773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0989073Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0989553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.0990015Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.0990457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.0990890Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.0991373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.0991898Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.0992416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.0992883Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.0993039Z 2025-08-14T21:51:44.0993152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.0993546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.0993900Z return mod(**inputs) 2025-08-14T21:51:44.0994334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.0994780Z outputs = self.bert( 2025-08-14T21:51:44.0995220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.0995753Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.0996222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.0996675Z layer_outputs = layer_module( 2025-08-14T21:51:44.0997065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.0997456Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.0997920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.0998401Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.0998841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.0999270Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.0999746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1000339Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1000839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1001350Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1001774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1002159Z return self.act(input) 2025-08-14T21:51:44.1002283Z 2025-08-14T21:51:44.1002407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1002815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1003188Z return mod(**inputs) 2025-08-14T21:51:44.1003627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1004094Z outputs = self.bert( 2025-08-14T21:51:44.1004532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1005011Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1005483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1005943Z layer_outputs = layer_module( 2025-08-14T21:51:44.1006321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1006720Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1007188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1007653Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1008108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1008541Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1009219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1009757Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1010277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1010744Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1010892Z 2025-08-14T21:51:44.1011003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1011385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1011727Z return mod(**inputs) 2025-08-14T21:51:44.1012153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1012589Z outputs = self.bert( 2025-08-14T21:51:44.1013012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1013482Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1013930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1014380Z layer_outputs = layer_module( 2025-08-14T21:51:44.1014742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1015132Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1015601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1016118Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1016541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1016961Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1017428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1017961Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1018462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1018908Z return input_tensor + hidden_states 2025-08-14T21:51:44.1019050Z 2025-08-14T21:51:44.1019160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1019548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1019900Z return mod(**inputs) 2025-08-14T21:51:44.1020316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1020786Z outputs = self.bert( 2025-08-14T21:51:44.1021228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1021668Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1022097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1022535Z layer_outputs = layer_module( 2025-08-14T21:51:44.1022898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1023266Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1023744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1024222Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1024668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1025100Z self_outputs = self.self( 2025-08-14T21:51:44.1025486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1025878Z return func(*args, **kwargs) 2025-08-14T21:51:44.1026304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1026743Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1026895Z 2025-08-14T21:51:44.1027007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1027383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1027724Z return mod(**inputs) 2025-08-14T21:51:44.1028142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1028574Z outputs = self.bert( 2025-08-14T21:51:44.1028987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1029425Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1029926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1030378Z layer_outputs = layer_module( 2025-08-14T21:51:44.1030736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1031128Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1031580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1032044Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1032505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1032955Z self_outputs = self.self( 2025-08-14T21:51:44.1033349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1033761Z return func(*args, **kwargs) 2025-08-14T21:51:44.1034208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1034669Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1034819Z 2025-08-14T21:51:44.1034939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1035331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1035745Z return mod(**inputs) 2025-08-14T21:51:44.1036219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1036677Z outputs = self.bert( 2025-08-14T21:51:44.1037100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1037566Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1038028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1038493Z layer_outputs = layer_module( 2025-08-14T21:51:44.1038882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1039277Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1039739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1040200Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1040667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1041122Z self_outputs = self.self( 2025-08-14T21:51:44.1041514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1041911Z return func(*args, **kwargs) 2025-08-14T21:51:44.1042354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1042820Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1042968Z 2025-08-14T21:51:44.1043064Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1043295Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1043555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1043950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1044295Z return mod(**inputs) 2025-08-14T21:51:44.1044730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1045186Z outputs = self.bert( 2025-08-14T21:51:44.1045603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1046039Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1046497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1046934Z layer_outputs = layer_module( 2025-08-14T21:51:44.1047293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1047675Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1048116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1048569Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1049011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1049513Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1050010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1050463Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1050608Z 2025-08-14T21:51:44.1050743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1051138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1051486Z return mod(**inputs) 2025-08-14T21:51:44.1051906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1052324Z outputs = self.bert( 2025-08-14T21:51:44.1052735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1053224Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1053661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1054081Z layer_outputs = layer_module( 2025-08-14T21:51:44.1054424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1054782Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1055196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1055627Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1056034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1056432Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1056897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1057418Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1057908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1058358Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1058502Z 2025-08-14T21:51:44.1058607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1058967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1059293Z return mod(**inputs) 2025-08-14T21:51:44.1059688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1060110Z outputs = self.bert( 2025-08-14T21:51:44.1060514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1060951Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1061369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1061804Z layer_outputs = layer_module( 2025-08-14T21:51:44.1062180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1062559Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1063007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1063441Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1063840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1064237Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1064714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1065224Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1065715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1066217Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1066625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1066982Z return self.act(input) 2025-08-14T21:51:44.1067094Z 2025-08-14T21:51:44.1067200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1067570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1067904Z return mod(**inputs) 2025-08-14T21:51:44.1068326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1068740Z outputs = self.bert( 2025-08-14T21:51:44.1069133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1069550Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1069962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1070382Z layer_outputs = layer_module( 2025-08-14T21:51:44.1070727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1071086Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1071492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1071912Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1072315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1072701Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1073151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1073682Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1074184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1074632Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1074785Z 2025-08-14T21:51:44.1074896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1075275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1075648Z return mod(**inputs) 2025-08-14T21:51:44.1076151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1076612Z outputs = self.bert( 2025-08-14T21:51:44.1077041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1077481Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1077891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1078301Z layer_outputs = layer_module( 2025-08-14T21:51:44.1078655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1079003Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1079421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1079834Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1080270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1080686Z self_outputs = self.self( 2025-08-14T21:51:44.1081047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1081415Z return func(*args, **kwargs) 2025-08-14T21:51:44.1081809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1082229Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1082376Z 2025-08-14T21:51:44.1082481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1082858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1083177Z return mod(**inputs) 2025-08-14T21:51:44.1083577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1083988Z outputs = self.bert( 2025-08-14T21:51:44.1084383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1084795Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1085212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1085629Z layer_outputs = layer_module( 2025-08-14T21:51:44.1085972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1086328Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1086737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1087151Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1087558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1087961Z self_outputs = self.self( 2025-08-14T21:51:44.1088321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1088696Z return func(*args, **kwargs) 2025-08-14T21:51:44.1089100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1089512Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1089662Z 2025-08-14T21:51:44.1089773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1090111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1090427Z return mod(**inputs) 2025-08-14T21:51:44.1090813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1091289Z outputs = self.bert( 2025-08-14T21:51:44.1091669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1092085Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1092495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1092904Z layer_outputs = layer_module( 2025-08-14T21:51:44.1093253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1093613Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1094035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1094477Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1094924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1095335Z self_outputs = self.self( 2025-08-14T21:51:44.1095698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1096065Z return func(*args, **kwargs) 2025-08-14T21:51:44.1096472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1096913Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1097052Z 2025-08-14T21:51:44.1097133Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1097353Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1097592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1097946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1098263Z return mod(**inputs) 2025-08-14T21:51:44.1098659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1099075Z outputs = self.bert( 2025-08-14T21:51:44.1099460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1099878Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1100292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1100707Z layer_outputs = layer_module( 2025-08-14T21:51:44.1101049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1101410Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1101834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1102260Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1102674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1103148Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1103619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1104061Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1104206Z 2025-08-14T21:51:44.1104308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1104661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1104981Z return mod(**inputs) 2025-08-14T21:51:44.1105371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1105785Z outputs = self.bert( 2025-08-14T21:51:44.1106176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1106592Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1106995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1107412Z layer_outputs = layer_module( 2025-08-14T21:51:44.1107758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1108129Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1108578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1109151Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1109567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1109947Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1110394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1110881Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1111407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1111858Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1112014Z 2025-08-14T21:51:44.1112125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1112508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1112844Z return mod(**inputs) 2025-08-14T21:51:44.1113273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1113709Z outputs = self.bert( 2025-08-14T21:51:44.1114128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1114574Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1115023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1115470Z layer_outputs = layer_module( 2025-08-14T21:51:44.1115905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1116305Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1116765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1117252Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1117635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1118019Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1118466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1119019Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1119497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1119990Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1120410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1120774Z return self.act(input) 2025-08-14T21:51:44.1120895Z 2025-08-14T21:51:44.1121019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1121379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1121713Z return mod(**inputs) 2025-08-14T21:51:44.1122096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1122504Z outputs = self.bert( 2025-08-14T21:51:44.1122889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1123350Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1123823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1124239Z layer_outputs = layer_module( 2025-08-14T21:51:44.1124582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1124934Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1125355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1125786Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1126209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1126607Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1127111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1127662Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1128176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1128623Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1128770Z 2025-08-14T21:51:44.1128875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1129236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1129570Z return mod(**inputs) 2025-08-14T21:51:44.1129963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1130385Z outputs = self.bert( 2025-08-14T21:51:44.1130787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1131204Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1131628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1132048Z layer_outputs = layer_module( 2025-08-14T21:51:44.1132398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1132758Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1133189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1133644Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1134033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1134428Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1134873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1135373Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1135837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1136261Z return input_tensor + hidden_states 2025-08-14T21:51:44.1136400Z 2025-08-14T21:51:44.1136503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1136866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1137182Z return mod(**inputs) 2025-08-14T21:51:44.1137599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1138029Z outputs = self.bert( 2025-08-14T21:51:44.1138418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1138842Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1139260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1139677Z layer_outputs = layer_module( 2025-08-14T21:51:44.1140016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1140396Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1140813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1141236Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1141653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1142064Z self_outputs = self.self( 2025-08-14T21:51:44.1142426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1142795Z return func(*args, **kwargs) 2025-08-14T21:51:44.1143203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1143624Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1143762Z 2025-08-14T21:51:44.1143872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1144225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1144547Z return mod(**inputs) 2025-08-14T21:51:44.1144945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1145352Z outputs = self.bert( 2025-08-14T21:51:44.1145736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1146153Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1146588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1147022Z layer_outputs = layer_module( 2025-08-14T21:51:44.1147390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1147790Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1148216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1148638Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1149086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1149529Z self_outputs = self.self( 2025-08-14T21:51:44.1149916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1150308Z return func(*args, **kwargs) 2025-08-14T21:51:44.1150739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1151196Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1151336Z 2025-08-14T21:51:44.1151447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1151848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1152199Z return mod(**inputs) 2025-08-14T21:51:44.1152639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1153080Z outputs = self.bert( 2025-08-14T21:51:44.1153490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1153942Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1154382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1154837Z layer_outputs = layer_module( 2025-08-14T21:51:44.1155234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1155633Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1156182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1156657Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1157134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1157588Z self_outputs = self.self( 2025-08-14T21:51:44.1157967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1158368Z return func(*args, **kwargs) 2025-08-14T21:51:44.1158807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1159256Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1159408Z 2025-08-14T21:51:44.1159495Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1160228Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1160481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1160855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1161198Z return mod(**inputs) 2025-08-14T21:51:44.1161621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1162056Z outputs = self.bert( 2025-08-14T21:51:44.1162465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1162914Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1163404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1163836Z layer_outputs = layer_module( 2025-08-14T21:51:44.1164199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1164570Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1164989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1165407Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1165830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1166303Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1166773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1167194Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1167380Z 2025-08-14T21:51:44.1167487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1167869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1168188Z return mod(**inputs) 2025-08-14T21:51:44.1168590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1169009Z outputs = self.bert( 2025-08-14T21:51:44.1169401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1169810Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1170238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1170656Z layer_outputs = layer_module( 2025-08-14T21:51:44.1170991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1171346Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1171766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1172193Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1172591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1173001Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1173474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1173986Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1174451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1174907Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1175051Z 2025-08-14T21:51:44.1175169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1175551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1175884Z return mod(**inputs) 2025-08-14T21:51:44.1176302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1176739Z outputs = self.bert( 2025-08-14T21:51:44.1177152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1177592Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1178006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1178429Z layer_outputs = layer_module( 2025-08-14T21:51:44.1178789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1179169Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1179615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1180058Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1180480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1180899Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1181378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1181876Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1182391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1182874Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1183275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1183631Z return self.act(input) 2025-08-14T21:51:44.1183756Z 2025-08-14T21:51:44.1183866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1184253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1184597Z return mod(**inputs) 2025-08-14T21:51:44.1185051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1185509Z outputs = self.bert( 2025-08-14T21:51:44.1185926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1186377Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1186816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1187282Z layer_outputs = layer_module( 2025-08-14T21:51:44.1187649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1188029Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1188489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1188961Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1189391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1189823Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1190298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1190855Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1191376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1191853Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1192008Z 2025-08-14T21:51:44.1192121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1192513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1192897Z return mod(**inputs) 2025-08-14T21:51:44.1193331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1193783Z outputs = self.bert( 2025-08-14T21:51:44.1194214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1194677Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1195132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1195593Z layer_outputs = layer_module( 2025-08-14T21:51:44.1196053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1196455Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1196935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1197443Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1197985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1198447Z self_outputs = self.self( 2025-08-14T21:51:44.1198846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1199252Z return func(*args, **kwargs) 2025-08-14T21:51:44.1199704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1200171Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1200319Z 2025-08-14T21:51:44.1200439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1200845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1201196Z return mod(**inputs) 2025-08-14T21:51:44.1201629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1202078Z outputs = self.bert( 2025-08-14T21:51:44.1202499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1202956Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1203411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1203867Z layer_outputs = layer_module( 2025-08-14T21:51:44.1204238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1204602Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1205017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1205438Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1205864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1206281Z self_outputs = self.self( 2025-08-14T21:51:44.1206635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1207013Z return func(*args, **kwargs) 2025-08-14T21:51:44.1207428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1207851Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1208008Z 2025-08-14T21:51:44.1208116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1208477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1208926Z return mod(**inputs) 2025-08-14T21:51:44.1209335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1209746Z outputs = self.bert( 2025-08-14T21:51:44.1210138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1210561Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1210971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1211392Z layer_outputs = layer_module( 2025-08-14T21:51:44.1211746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1212111Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1212573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1213030Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1213457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1213866Z self_outputs = self.self( 2025-08-14T21:51:44.1214220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1214591Z return func(*args, **kwargs) 2025-08-14T21:51:44.1214998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1215461Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1215605Z 2025-08-14T21:51:44.1215687Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1215903Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1216142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1216491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1216813Z return mod(**inputs) 2025-08-14T21:51:44.1217207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1217609Z outputs = self.bert( 2025-08-14T21:51:44.1217998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1218412Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1218822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1219229Z layer_outputs = layer_module( 2025-08-14T21:51:44.1219579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1219934Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1220345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1220768Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1221189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1221690Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1222183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1222689Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1222842Z 2025-08-14T21:51:44.1222953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1223329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1223669Z return mod(**inputs) 2025-08-14T21:51:44.1224094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1224542Z outputs = self.bert( 2025-08-14T21:51:44.1224958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1225415Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1225866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1226319Z layer_outputs = layer_module( 2025-08-14T21:51:44.1226683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1227081Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1227551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1228004Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1228425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1228843Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1229331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1229844Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1230338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1230791Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1230936Z 2025-08-14T21:51:44.1231056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1231435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1231782Z return mod(**inputs) 2025-08-14T21:51:44.1232208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1232654Z outputs = self.bert( 2025-08-14T21:51:44.1233065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1233515Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1233961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1234400Z layer_outputs = layer_module( 2025-08-14T21:51:44.1234774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1235161Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1235619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1236156Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1236595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1237025Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1237513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1238053Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1238532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1239025Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1239436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1239793Z return self.act(input) 2025-08-14T21:51:44.1239919Z 2025-08-14T21:51:44.1240031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1240415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1240759Z return mod(**inputs) 2025-08-14T21:51:44.1241157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1241575Z outputs = self.bert( 2025-08-14T21:51:44.1241970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1242401Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1242834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1243251Z layer_outputs = layer_module( 2025-08-14T21:51:44.1243608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1244000Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1244418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1244845Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1245268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1245666Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1246112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1246615Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1247082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1247511Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1247656Z 2025-08-14T21:51:44.1247774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1248144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1248490Z return mod(**inputs) 2025-08-14T21:51:44.1248904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1249348Z outputs = self.bert( 2025-08-14T21:51:44.1249756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1250202Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1250617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1251035Z layer_outputs = layer_module( 2025-08-14T21:51:44.1251373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1251731Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1252154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1252591Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1252996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1253412Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1253883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1254402Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1254883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1255335Z return input_tensor + hidden_states 2025-08-14T21:51:44.1255467Z 2025-08-14T21:51:44.1255580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1255934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1256262Z return mod(**inputs) 2025-08-14T21:51:44.1256678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1257099Z outputs = self.bert( 2025-08-14T21:51:44.1257500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1257916Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1258346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1258772Z layer_outputs = layer_module( 2025-08-14T21:51:44.1259134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1259543Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1260007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1260452Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1260894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1261307Z self_outputs = self.self( 2025-08-14T21:51:44.1261664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1262042Z return func(*args, **kwargs) 2025-08-14T21:51:44.1262450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1262876Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1263015Z 2025-08-14T21:51:44.1263121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1263482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1263812Z return mod(**inputs) 2025-08-14T21:51:44.1264226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1264664Z outputs = self.bert( 2025-08-14T21:51:44.1265058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1265480Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1265912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1266361Z layer_outputs = layer_module( 2025-08-14T21:51:44.1266751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1267134Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1267575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1268031Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1268474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1268912Z self_outputs = self.self( 2025-08-14T21:51:44.1269294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1269685Z return func(*args, **kwargs) 2025-08-14T21:51:44.1270118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1270562Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1270712Z 2025-08-14T21:51:44.1270822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1271228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1271573Z return mod(**inputs) 2025-08-14T21:51:44.1272015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1272470Z outputs = self.bert( 2025-08-14T21:51:44.1272893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1273336Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1273786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1274275Z layer_outputs = layer_module( 2025-08-14T21:51:44.1274653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1275041Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1275501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1276055Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1276524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1276971Z self_outputs = self.self( 2025-08-14T21:51:44.1277362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1277770Z return func(*args, **kwargs) 2025-08-14T21:51:44.1278210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1278678Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1278837Z 2025-08-14T21:51:44.1278926Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1279161Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1279415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1279807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1280161Z return mod(**inputs) 2025-08-14T21:51:44.1280586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1281037Z outputs = self.bert( 2025-08-14T21:51:44.1281465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1281948Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1282390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1282846Z layer_outputs = layer_module( 2025-08-14T21:51:44.1283224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1283576Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1284013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1284465Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1284909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1285403Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1285906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1286374Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1286539Z 2025-08-14T21:51:44.1286656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1287046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1287391Z return mod(**inputs) 2025-08-14T21:51:44.1287809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1288243Z outputs = self.bert( 2025-08-14T21:51:44.1288660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1289106Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1289562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1289995Z layer_outputs = layer_module( 2025-08-14T21:51:44.1290365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1290746Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1291200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1291657Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1292100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1292527Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1292994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1293512Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1293992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1294452Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1294596Z 2025-08-14T21:51:44.1294707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1295092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1295442Z return mod(**inputs) 2025-08-14T21:51:44.1295864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1296290Z outputs = self.bert( 2025-08-14T21:51:44.1296712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1297184Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1297623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1298072Z layer_outputs = layer_module( 2025-08-14T21:51:44.1298439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1298828Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1299275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1299740Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1300172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1300598Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1301066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1301598Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1302095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1302582Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1302982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1303346Z return self.act(input) 2025-08-14T21:51:44.1303467Z 2025-08-14T21:51:44.1303586Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1303965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1304316Z return mod(**inputs) 2025-08-14T21:51:44.1304762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1305200Z outputs = self.bert( 2025-08-14T21:51:44.1305604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1306049Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1306486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1306919Z layer_outputs = layer_module( 2025-08-14T21:51:44.1307290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1307668Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1308123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1308579Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1309166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1309603Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1310100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1310647Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1311178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1311648Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1311801Z 2025-08-14T21:51:44.1311922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1312362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1312715Z return mod(**inputs) 2025-08-14T21:51:44.1313147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1313593Z outputs = self.bert( 2025-08-14T21:51:44.1314023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1314485Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1314938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1315380Z layer_outputs = layer_module( 2025-08-14T21:51:44.1315842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1316243Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1316695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1317201Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1317691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1318146Z self_outputs = self.self( 2025-08-14T21:51:44.1318535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1318942Z return func(*args, **kwargs) 2025-08-14T21:51:44.1319386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1319846Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1319996Z 2025-08-14T21:51:44.1320141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1320533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1320888Z return mod(**inputs) 2025-08-14T21:51:44.1321311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1321393Z outputs = self.bert( 2025-08-14T21:51:44.1321704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1321794Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1322109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1322188Z layer_outputs = layer_module( 2025-08-14T21:51:44.1322435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1322524Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1322830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1322932Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1323238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1323322Z self_outputs = self.self( 2025-08-14T21:51:44.1323579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1323665Z return func(*args, **kwargs) 2025-08-14T21:51:44.1323972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1324079Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1324085Z 2025-08-14T21:51:44.1324200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1324415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1324482Z return mod(**inputs) 2025-08-14T21:51:44.1324783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1324860Z outputs = self.bert( 2025-08-14T21:51:44.1325142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1325221Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1325502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1325580Z layer_outputs = layer_module( 2025-08-14T21:51:44.1325798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1325872Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1326174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1326279Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1326571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1326640Z self_outputs = self.self( 2025-08-14T21:51:44.1326878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1326955Z return func(*args, **kwargs) 2025-08-14T21:51:44.1327255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1327338Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1327350Z 2025-08-14T21:51:44.1327430Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1327512Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1327622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1327818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1327886Z return mod(**inputs) 2025-08-14T21:51:44.1328194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1328262Z outputs = self.bert( 2025-08-14T21:51:44.1328540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1328623Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1328904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1328985Z layer_outputs = layer_module( 2025-08-14T21:51:44.1329205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1329285Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1329569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1329650Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1329933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1330062Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1330342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1330451Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1330456Z 2025-08-14T21:51:44.1330555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1330755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1330820Z return mod(**inputs) 2025-08-14T21:51:44.1331098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1331167Z outputs = self.bert( 2025-08-14T21:51:44.1331442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1331513Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1331800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1331870Z layer_outputs = layer_module( 2025-08-14T21:51:44.1332092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1332183Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1332474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1332565Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1332820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1332901Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1333218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1333347Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1333640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1333723Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1333726Z 2025-08-14T21:51:44.1333830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1334032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1334098Z return mod(**inputs) 2025-08-14T21:51:44.1334385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1334450Z outputs = self.bert( 2025-08-14T21:51:44.1334729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1334812Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1335095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1335174Z layer_outputs = layer_module( 2025-08-14T21:51:44.1335390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1335477Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1335757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1335836Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1336084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1336164Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1336469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1336604Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1336888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1337003Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1337224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1337298Z return self.act(input) 2025-08-14T21:51:44.1337302Z 2025-08-14T21:51:44.1337420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1337627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1337695Z return mod(**inputs) 2025-08-14T21:51:44.1338004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1338076Z outputs = self.bert( 2025-08-14T21:51:44.1338378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1338518Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1338842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1338929Z layer_outputs = layer_module( 2025-08-14T21:51:44.1339160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1339241Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1339551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1339659Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1339942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1340024Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1340357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1340506Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1340807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1340893Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1340905Z 2025-08-14T21:51:44.1341013Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1341219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1341301Z return mod(**inputs) 2025-08-14T21:51:44.1341605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1341677Z outputs = self.bert( 2025-08-14T21:51:44.1341986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1342064Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1342370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1342443Z layer_outputs = layer_module( 2025-08-14T21:51:44.1342672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1342760Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1343080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1343168Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1343450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1343531Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1343869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1344007Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1344307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1344397Z return input_tensor + hidden_states 2025-08-14T21:51:44.1344400Z 2025-08-14T21:51:44.1344511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1344724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1344814Z return mod(**inputs) 2025-08-14T21:51:44.1345143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1345223Z outputs = self.bert( 2025-08-14T21:51:44.1345519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1345595Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1345907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1345984Z layer_outputs = layer_module( 2025-08-14T21:51:44.1346239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1346324Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1346620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1346716Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1347015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1347095Z self_outputs = self.self( 2025-08-14T21:51:44.1347351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1347427Z return func(*args, **kwargs) 2025-08-14T21:51:44.1347739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1347825Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1347831Z 2025-08-14T21:51:44.1347946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1348149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1348218Z return mod(**inputs) 2025-08-14T21:51:44.1348527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1348597Z outputs = self.bert( 2025-08-14T21:51:44.1348895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1348981Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1349278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1349358Z layer_outputs = layer_module( 2025-08-14T21:51:44.1349609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1349688Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1349993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1350081Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1350376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1350456Z self_outputs = self.self( 2025-08-14T21:51:44.1350708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1350791Z return func(*args, **kwargs) 2025-08-14T21:51:44.1351098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1351183Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1351187Z 2025-08-14T21:51:44.1351300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1351543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1351633Z return mod(**inputs) 2025-08-14T21:51:44.1351935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1352002Z outputs = self.bert( 2025-08-14T21:51:44.1352312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1352388Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1352696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1352796Z layer_outputs = layer_module( 2025-08-14T21:51:44.1353031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1353124Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1353440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1353529Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1353852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1353928Z self_outputs = self.self( 2025-08-14T21:51:44.1354199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1354276Z return func(*args, **kwargs) 2025-08-14T21:51:44.1354598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1354692Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1354698Z 2025-08-14T21:51:44.1354787Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1354876Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1354997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1355210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1355288Z return mod(**inputs) 2025-08-14T21:51:44.1355612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1355754Z outputs = self.bert( 2025-08-14T21:51:44.1356084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1356197Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1356515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1356600Z layer_outputs = layer_module( 2025-08-14T21:51:44.1356847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1356938Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1357246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1357331Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1357651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1357786Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1358103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1358216Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1358220Z 2025-08-14T21:51:44.1358332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1358572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1358646Z return mod(**inputs) 2025-08-14T21:51:44.1358980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1359052Z outputs = self.bert( 2025-08-14T21:51:44.1359371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1359457Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1359801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1359881Z layer_outputs = layer_module( 2025-08-14T21:51:44.1360129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1360213Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1360543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1360632Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1360932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1361024Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1361373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1361495Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1361824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1361913Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1361917Z 2025-08-14T21:51:44.1362034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1362250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1362322Z return mod(**inputs) 2025-08-14T21:51:44.1362653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1362722Z outputs = self.bert( 2025-08-14T21:51:44.1363059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1363158Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1363464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1363550Z layer_outputs = layer_module( 2025-08-14T21:51:44.1363787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1363877Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1364188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1364273Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1364552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1364633Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1364961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1365096Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1365416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1365538Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1365747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1365817Z return self.act(input) 2025-08-14T21:51:44.1365820Z 2025-08-14T21:51:44.1365928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1366121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1366194Z return mod(**inputs) 2025-08-14T21:51:44.1366503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1366573Z outputs = self.bert( 2025-08-14T21:51:44.1366883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1366958Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1367266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1367346Z layer_outputs = layer_module( 2025-08-14T21:51:44.1367561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1367646Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1367929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1368013Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1368279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1368354Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1368675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1368806Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1369091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1369179Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1369183Z 2025-08-14T21:51:44.1369283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1369498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1369572Z return mod(**inputs) 2025-08-14T21:51:44.1369858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1369933Z outputs = self.bert( 2025-08-14T21:51:44.1370214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1370287Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1370576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1370647Z layer_outputs = layer_module( 2025-08-14T21:51:44.1370871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1370952Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1371236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1371344Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1371641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1371714Z self_outputs = self.self( 2025-08-14T21:51:44.1371966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1372036Z return func(*args, **kwargs) 2025-08-14T21:51:44.1372327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1372406Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1372411Z 2025-08-14T21:51:44.1372531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1372733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1372801Z return mod(**inputs) 2025-08-14T21:51:44.1373094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1373162Z outputs = self.bert( 2025-08-14T21:51:44.1373441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1373523Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1373804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1373874Z layer_outputs = layer_module( 2025-08-14T21:51:44.1374099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1374178Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1374469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1374553Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1374856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1374937Z self_outputs = self.self( 2025-08-14T21:51:44.1375199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1375275Z return func(*args, **kwargs) 2025-08-14T21:51:44.1375560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1375684Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1375687Z 2025-08-14T21:51:44.1375798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1376008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1376076Z return mod(**inputs) 2025-08-14T21:51:44.1376396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1376466Z outputs = self.bert( 2025-08-14T21:51:44.1376780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1376856Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1377162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1377246Z layer_outputs = layer_module( 2025-08-14T21:51:44.1377487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1377594Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1377907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1377993Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1378296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1378368Z self_outputs = self.self( 2025-08-14T21:51:44.1378619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1378701Z return func(*args, **kwargs) 2025-08-14T21:51:44.1379038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1379132Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1379135Z 2025-08-14T21:51:44.1379221Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1379304Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1379421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1379625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1379693Z return mod(**inputs) 2025-08-14T21:51:44.1380008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1380077Z outputs = self.bert( 2025-08-14T21:51:44.1380385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1380458Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1380746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1380830Z layer_outputs = layer_module( 2025-08-14T21:51:44.1381062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1381148Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1381447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1381530Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1381838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1381970Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1382295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1382391Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1382396Z 2025-08-14T21:51:44.1382504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1382717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1382786Z return mod(**inputs) 2025-08-14T21:51:44.1383088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1383165Z outputs = self.bert( 2025-08-14T21:51:44.1383464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1383547Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1383847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1383923Z layer_outputs = layer_module( 2025-08-14T21:51:44.1384177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1384256Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1384572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1384668Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1384942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1385030Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1385359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1385487Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1385796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1385885Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1385888Z 2025-08-14T21:51:44.1386004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1386210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1386280Z return mod(**inputs) 2025-08-14T21:51:44.1386587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1386655Z outputs = self.bert( 2025-08-14T21:51:44.1386956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1387044Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1387342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1387425Z layer_outputs = layer_module( 2025-08-14T21:51:44.1387656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1387739Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1388049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1388137Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1388417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1388497Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1388854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1388971Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1389271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1389401Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1389624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1389700Z return self.act(input) 2025-08-14T21:51:44.1389704Z 2025-08-14T21:51:44.1389818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1390025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1390093Z return mod(**inputs) 2025-08-14T21:51:44.1390406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1390474Z outputs = self.bert( 2025-08-14T21:51:44.1390803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1390899Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1391201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1391285Z layer_outputs = layer_module( 2025-08-14T21:51:44.1391514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1391595Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1391912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1392021Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1392305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1392385Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1392718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1392864Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1393160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1393250Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1393254Z 2025-08-14T21:51:44.1393359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1393567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1393645Z return mod(**inputs) 2025-08-14T21:51:44.1393958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1394036Z outputs = self.bert( 2025-08-14T21:51:44.1394353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1394431Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1394750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1394824Z layer_outputs = layer_module( 2025-08-14T21:51:44.1395060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1395149Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1395479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1395573Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1395937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1396025Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1396376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1396517Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1396849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1396934Z return input_tensor + hidden_states 2025-08-14T21:51:44.1396940Z 2025-08-14T21:51:44.1397053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1397274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1397374Z return mod(**inputs) 2025-08-14T21:51:44.1397715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1397795Z outputs = self.bert( 2025-08-14T21:51:44.1398104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1398190Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1398499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1398574Z layer_outputs = layer_module( 2025-08-14T21:51:44.1398849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1398937Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1399262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1399352Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1399666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1399749Z self_outputs = self.self( 2025-08-14T21:51:44.1400019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1400096Z return func(*args, **kwargs) 2025-08-14T21:51:44.1400420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1400511Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1400515Z 2025-08-14T21:51:44.1400634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1400846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1400915Z return mod(**inputs) 2025-08-14T21:51:44.1401232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1401303Z outputs = self.bert( 2025-08-14T21:51:44.1401625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1401703Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1402021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1402128Z layer_outputs = layer_module( 2025-08-14T21:51:44.1402374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1402460Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1402785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1402873Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1403182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1403257Z self_outputs = self.self( 2025-08-14T21:51:44.1403525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1403610Z return func(*args, **kwargs) 2025-08-14T21:51:44.1403930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1404025Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1404029Z 2025-08-14T21:51:44.1404162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1404389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1404471Z return mod(**inputs) 2025-08-14T21:51:44.1404785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1404855Z outputs = self.bert( 2025-08-14T21:51:44.1405188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1405267Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1405611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1405691Z layer_outputs = layer_module( 2025-08-14T21:51:44.1405926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1406022Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1406329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1406425Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1406732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1406807Z self_outputs = self.self( 2025-08-14T21:51:44.1407079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1407152Z return func(*args, **kwargs) 2025-08-14T21:51:44.1407453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1407545Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1407549Z 2025-08-14T21:51:44.1407633Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1407724Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1407832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1408041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1408119Z return mod(**inputs) 2025-08-14T21:51:44.1408430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1408499Z outputs = self.bert( 2025-08-14T21:51:44.1408971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1409100Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1409405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1409482Z layer_outputs = layer_module( 2025-08-14T21:51:44.1409713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1409804Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1410111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1410205Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1410503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1410641Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1410948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1411062Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1411066Z 2025-08-14T21:51:44.1411202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1411419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1411487Z return mod(**inputs) 2025-08-14T21:51:44.1411801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1411871Z outputs = self.bert( 2025-08-14T21:51:44.1412176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1412264Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1412589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1412672Z layer_outputs = layer_module( 2025-08-14T21:51:44.1412900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1412981Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1413284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1413371Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1413642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1413731Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1414064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1414182Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1414478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1414564Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1414568Z 2025-08-14T21:51:44.1414684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1414890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1414966Z return mod(**inputs) 2025-08-14T21:51:44.1415267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1415334Z outputs = self.bert( 2025-08-14T21:51:44.1415639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1415735Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1416033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1416115Z layer_outputs = layer_module( 2025-08-14T21:51:44.1416343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1416430Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1416724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1416809Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1417082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1417165Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1417501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1417635Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1417948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1418076Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1418297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1418370Z return self.act(input) 2025-08-14T21:51:44.1418380Z 2025-08-14T21:51:44.1418487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1418696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1418798Z return mod(**inputs) 2025-08-14T21:51:44.1419094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1419164Z outputs = self.bert( 2025-08-14T21:51:44.1419459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1419534Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1419830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1419903Z layer_outputs = layer_module( 2025-08-14T21:51:44.1420125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1420213Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1420502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1420588Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1420857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1420936Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1421269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1421405Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1421703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1421797Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1421800Z 2025-08-14T21:51:44.1421909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1422147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1422218Z return mod(**inputs) 2025-08-14T21:51:44.1422518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1422598Z outputs = self.bert( 2025-08-14T21:51:44.1422896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1422981Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1423277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1423353Z layer_outputs = layer_module( 2025-08-14T21:51:44.1423588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1423673Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1423971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1424087Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1424418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1424500Z self_outputs = self.self( 2025-08-14T21:51:44.1424744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1424815Z return func(*args, **kwargs) 2025-08-14T21:51:44.1425110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1425193Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1425214Z 2025-08-14T21:51:44.1425330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1425529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1425598Z return mod(**inputs) 2025-08-14T21:51:44.1425906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1425975Z outputs = self.bert( 2025-08-14T21:51:44.1426273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1426359Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1426654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1426734Z layer_outputs = layer_module( 2025-08-14T21:51:44.1426966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1427046Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1427352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1427438Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1427744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1427815Z self_outputs = self.self( 2025-08-14T21:51:44.1428059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1428136Z return func(*args, **kwargs) 2025-08-14T21:51:44.1428423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1428847Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1428850Z 2025-08-14T21:51:44.1428966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1429169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1429243Z return mod(**inputs) 2025-08-14T21:51:44.1429541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1429608Z outputs = self.bert( 2025-08-14T21:51:44.1429910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1429988Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1430294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1430379Z layer_outputs = layer_module( 2025-08-14T21:51:44.1430610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1430726Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1431057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1431144Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1431451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1431525Z self_outputs = self.self( 2025-08-14T21:51:44.1431787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1431862Z return func(*args, **kwargs) 2025-08-14T21:51:44.1432178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1432273Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1432278Z 2025-08-14T21:51:44.1432364Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1432456Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1432576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1432786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1432865Z return mod(**inputs) 2025-08-14T21:51:44.1433168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1433241Z outputs = self.bert( 2025-08-14T21:51:44.1433549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1433633Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1433941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1434021Z layer_outputs = layer_module( 2025-08-14T21:51:44.1434253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1434347Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1434651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1434740Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1435047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1435185Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1435514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1435602Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1435606Z 2025-08-14T21:51:44.1435772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1435998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1436067Z return mod(**inputs) 2025-08-14T21:51:44.1436383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1436455Z outputs = self.bert( 2025-08-14T21:51:44.1436762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1436851Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1437165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1437240Z layer_outputs = layer_module( 2025-08-14T21:51:44.1437500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1437602Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1437914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1438004Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1438281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1438372Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1438721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1438842Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1439142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1439230Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1439236Z 2025-08-14T21:51:44.1439350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1439559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1439630Z return mod(**inputs) 2025-08-14T21:51:44.1439941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1440011Z outputs = self.bert( 2025-08-14T21:51:44.1440318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1440396Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1440698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1440782Z layer_outputs = layer_module( 2025-08-14T21:51:44.1441012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1441100Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1441397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1441486Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1441764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1441844Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1442233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1442351Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1442654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1442781Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1443007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1443080Z return self.act(input) 2025-08-14T21:51:44.1443084Z 2025-08-14T21:51:44.1443210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1443425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1443495Z return mod(**inputs) 2025-08-14T21:51:44.1443784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1443869Z outputs = self.bert( 2025-08-14T21:51:44.1444172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1444247Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1444530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1444610Z layer_outputs = layer_module( 2025-08-14T21:51:44.1444827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1444913Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1445215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1445300Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1445562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1445641Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1445960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1446089Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1446380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1446471Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1446474Z 2025-08-14T21:51:44.1446576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1446782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1446848Z return mod(**inputs) 2025-08-14T21:51:44.1447133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1447209Z outputs = self.bert( 2025-08-14T21:51:44.1447488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1447561Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1447848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1447919Z layer_outputs = layer_module( 2025-08-14T21:51:44.1448142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1448237Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1448521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1448615Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1448884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1448965Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1449299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1449434Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1449744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1449824Z return input_tensor + hidden_states 2025-08-14T21:51:44.1449831Z 2025-08-14T21:51:44.1449938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1450204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1450309Z return mod(**inputs) 2025-08-14T21:51:44.1450631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1450703Z outputs = self.bert( 2025-08-14T21:51:44.1451002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1451087Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1451384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1451465Z layer_outputs = layer_module( 2025-08-14T21:51:44.1451717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1451803Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1452121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1452211Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1452519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1452600Z self_outputs = self.self( 2025-08-14T21:51:44.1452865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1452949Z return func(*args, **kwargs) 2025-08-14T21:51:44.1453267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1453360Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1453364Z 2025-08-14T21:51:44.1453486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1453712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1453789Z return mod(**inputs) 2025-08-14T21:51:44.1454089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1454157Z outputs = self.bert( 2025-08-14T21:51:44.1454465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1454536Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1454810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1454906Z layer_outputs = layer_module( 2025-08-14T21:51:44.1455119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1455204Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1455483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1455561Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1455847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1455914Z self_outputs = self.self( 2025-08-14T21:51:44.1456149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1456227Z return func(*args, **kwargs) 2025-08-14T21:51:44.1456504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1456589Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1456612Z 2025-08-14T21:51:44.1456713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1456932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1457012Z return mod(**inputs) 2025-08-14T21:51:44.1457325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1457401Z outputs = self.bert( 2025-08-14T21:51:44.1457708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1457785Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1458113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1458192Z layer_outputs = layer_module( 2025-08-14T21:51:44.1458430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1458520Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1458824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1458920Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1459205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1459274Z self_outputs = self.self( 2025-08-14T21:51:44.1459518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1459591Z return func(*args, **kwargs) 2025-08-14T21:51:44.1459881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1459964Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1459967Z 2025-08-14T21:51:44.1460053Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1460147Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1460257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1460459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1460534Z return mod(**inputs) 2025-08-14T21:51:44.1460844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1460918Z outputs = self.bert( 2025-08-14T21:51:44.1461227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1461325Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1461646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1461723Z layer_outputs = layer_module( 2025-08-14T21:51:44.1461974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1462062Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1462380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1462473Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1462801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1462944Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1463281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1463389Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1463392Z 2025-08-14T21:51:44.1463528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1463743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1463814Z return mod(**inputs) 2025-08-14T21:51:44.1464137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1464208Z outputs = self.bert( 2025-08-14T21:51:44.1464561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1464654Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1464962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1465046Z layer_outputs = layer_module( 2025-08-14T21:51:44.1465286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1465366Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1465681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1465768Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1466048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1466128Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1466464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1466586Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1466895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1466990Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1466994Z 2025-08-14T21:51:44.1467105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1467317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1467394Z return mod(**inputs) 2025-08-14T21:51:44.1467712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1467781Z outputs = self.bert( 2025-08-14T21:51:44.1468107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1468182Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1468489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1468564Z layer_outputs = layer_module( 2025-08-14T21:51:44.1468796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1468883Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1469190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1469283Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1469554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1469635Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1469971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1470099Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1470412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1470539Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1470759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1470840Z return self.act(input) 2025-08-14T21:51:44.1470843Z 2025-08-14T21:51:44.1470951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1471175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1471256Z return mod(**inputs) 2025-08-14T21:51:44.1471569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1471647Z outputs = self.bert( 2025-08-14T21:51:44.1471953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1472032Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1472350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1472427Z layer_outputs = layer_module( 2025-08-14T21:51:44.1472666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1472758Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1473070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1473168Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1473463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1473545Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1473887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1474027Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1474346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1474433Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1474462Z 2025-08-14T21:51:44.1474574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1474791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1474863Z return mod(**inputs) 2025-08-14T21:51:44.1475167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1475244Z outputs = self.bert( 2025-08-14T21:51:44.1475546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1475632Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1476242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1476326Z layer_outputs = layer_module( 2025-08-14T21:51:44.1476576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1476663Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1476982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1477100Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1477446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1477535Z self_outputs = self.self( 2025-08-14T21:51:44.1477814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1477890Z return func(*args, **kwargs) 2025-08-14T21:51:44.1478200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1478306Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1478311Z 2025-08-14T21:51:44.1478426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1478633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1478700Z return mod(**inputs) 2025-08-14T21:51:44.1479014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1479084Z outputs = self.bert( 2025-08-14T21:51:44.1479394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1479473Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1479776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1479860Z layer_outputs = layer_module( 2025-08-14T21:51:44.1480096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1480180Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1480498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1480585Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1480898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1480974Z self_outputs = self.self( 2025-08-14T21:51:44.1481231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1481313Z return func(*args, **kwargs) 2025-08-14T21:51:44.1481619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1481730Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1481734Z 2025-08-14T21:51:44.1481848Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1482060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1482138Z return mod(**inputs) 2025-08-14T21:51:44.1482449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1482520Z outputs = self.bert( 2025-08-14T21:51:44.1482832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1482909Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1483225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1483305Z layer_outputs = layer_module( 2025-08-14T21:51:44.1483538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1483648Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1483972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1484068Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1484377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1484451Z self_outputs = self.self( 2025-08-14T21:51:44.1484720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1484795Z return func(*args, **kwargs) 2025-08-14T21:51:44.1485124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1485220Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1485225Z 2025-08-14T21:51:44.1485311Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1485402Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1485513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1485723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1485801Z return mod(**inputs) 2025-08-14T21:51:44.1486113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1486183Z outputs = self.bert( 2025-08-14T21:51:44.1486497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1486580Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1486897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1486975Z layer_outputs = layer_module( 2025-08-14T21:51:44.1487212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1487302Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1487610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1487704Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1488024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1488163Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1488503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1488594Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1488598Z 2025-08-14T21:51:44.1488716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1488940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1489009Z return mod(**inputs) 2025-08-14T21:51:44.1489318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1489387Z outputs = self.bert( 2025-08-14T21:51:44.1489694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1489778Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1490076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1490175Z layer_outputs = layer_module( 2025-08-14T21:51:44.1490402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1490499Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1490807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1490897Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1491170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1491256Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1491611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1491731Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1492029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1492115Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1492119Z 2025-08-14T21:51:44.1492232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1492438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1492516Z return mod(**inputs) 2025-08-14T21:51:44.1492824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1492893Z outputs = self.bert( 2025-08-14T21:51:44.1493203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1493282Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1493589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1493664Z layer_outputs = layer_module( 2025-08-14T21:51:44.1493896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1493985Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1494282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1494373Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1494651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1494750Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1495085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1495193Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1495492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1495618Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1495841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1495921Z return self.act(input) 2025-08-14T21:51:44.1495925Z 2025-08-14T21:51:44.1496032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1496239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1496316Z return mod(**inputs) 2025-08-14T21:51:44.1496619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1496711Z outputs = self.bert( 2025-08-14T21:51:44.1497035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1497114Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1497421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1497496Z layer_outputs = layer_module( 2025-08-14T21:51:44.1497725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1497813Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1498139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1498236Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1498506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1498586Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1498922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1499058Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1499355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1499448Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1499451Z 2025-08-14T21:51:44.1499558Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1499772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1499841Z return mod(**inputs) 2025-08-14T21:51:44.1500143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1500222Z outputs = self.bert( 2025-08-14T21:51:44.1500520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1500603Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1500898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1500972Z layer_outputs = layer_module( 2025-08-14T21:51:44.1501205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1501304Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1501606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1501701Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1501973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1502057Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1502386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1502520Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1502829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1502913Z return input_tensor + hidden_states 2025-08-14T21:51:44.1502917Z 2025-08-14T21:51:44.1503029Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1503254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1503323Z return mod(**inputs) 2025-08-14T21:51:44.1503648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1503720Z outputs = self.bert( 2025-08-14T21:51:44.1504016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1504103Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1504398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1504478Z layer_outputs = layer_module( 2025-08-14T21:51:44.1504728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1504813Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1505120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1505206Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1505509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1505584Z self_outputs = self.self( 2025-08-14T21:51:44.1505836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1505921Z return func(*args, **kwargs) 2025-08-14T21:51:44.1506221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1506310Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1506322Z 2025-08-14T21:51:44.1506431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1506635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1506712Z return mod(**inputs) 2025-08-14T21:51:44.1507012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1507082Z outputs = self.bert( 2025-08-14T21:51:44.1507390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1507467Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1507776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1507871Z layer_outputs = layer_module( 2025-08-14T21:51:44.1508108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1508201Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1508509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1508594Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1509071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1509151Z self_outputs = self.self( 2025-08-14T21:51:44.1509417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1509492Z return func(*args, **kwargs) 2025-08-14T21:51:44.1509799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1509892Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1509945Z 2025-08-14T21:51:44.1510055Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1510292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1510364Z return mod(**inputs) 2025-08-14T21:51:44.1510669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1510759Z outputs = self.bert( 2025-08-14T21:51:44.1511074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1511152Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1511489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1511570Z layer_outputs = layer_module( 2025-08-14T21:51:44.1511810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1511892Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1512191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1512286Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1512584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1512663Z self_outputs = self.self( 2025-08-14T21:51:44.1512916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1512993Z return func(*args, **kwargs) 2025-08-14T21:51:44.1513303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1513388Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1513391Z 2025-08-14T21:51:44.1513477Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1513568Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1513673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1513884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1513953Z return mod(**inputs) 2025-08-14T21:51:44.1514253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1514330Z outputs = self.bert( 2025-08-14T21:51:44.1514629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1514733Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1515041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1515118Z layer_outputs = layer_module( 2025-08-14T21:51:44.1515351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1515432Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1515798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1515901Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1516220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1516374Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1516685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1516796Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1516818Z 2025-08-14T21:51:44.1516938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1517157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1517227Z return mod(**inputs) 2025-08-14T21:51:44.1517533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1517603Z outputs = self.bert( 2025-08-14T21:51:44.1517923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1518021Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1518333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1518421Z layer_outputs = layer_module( 2025-08-14T21:51:44.1518661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1518753Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1519062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1519152Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1519437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1519519Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1519868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1519989Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1520299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1520393Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1520397Z 2025-08-14T21:51:44.1520506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1520719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1520796Z return mod(**inputs) 2025-08-14T21:51:44.1521104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1521183Z outputs = self.bert( 2025-08-14T21:51:44.1521515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1521595Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1521915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1521992Z layer_outputs = layer_module( 2025-08-14T21:51:44.1522229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1522321Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1522630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1522726Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1523010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1523093Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1523444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1523605Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1523919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1524042Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1524268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1524351Z return self.act(input) 2025-08-14T21:51:44.1524355Z 2025-08-14T21:51:44.1524465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1524705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1524780Z return mod(**inputs) 2025-08-14T21:51:44.1525095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1525175Z outputs = self.bert( 2025-08-14T21:51:44.1525481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1525558Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1525863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1525937Z layer_outputs = layer_module( 2025-08-14T21:51:44.1526174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1526255Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1526568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1526665Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1526939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1527017Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1527358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1527494Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1527803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1527888Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1527912Z 2025-08-14T21:51:44.1528023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1528238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1528307Z return mod(**inputs) 2025-08-14T21:51:44.1528621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1528690Z outputs = self.bert( 2025-08-14T21:51:44.1528988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1529073Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1529372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1529453Z layer_outputs = layer_module( 2025-08-14T21:51:44.1529687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1529768Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1530093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1530195Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1530497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1530579Z self_outputs = self.self( 2025-08-14T21:51:44.1530835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1530914Z return func(*args, **kwargs) 2025-08-14T21:51:44.1531211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1531317Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1531321Z 2025-08-14T21:51:44.1531435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1531639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1531714Z return mod(**inputs) 2025-08-14T21:51:44.1532013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1532082Z outputs = self.bert( 2025-08-14T21:51:44.1532387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1532463Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1532778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1532862Z layer_outputs = layer_module( 2025-08-14T21:51:44.1533089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1533175Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1533455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1533536Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1533822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1533890Z self_outputs = self.self( 2025-08-14T21:51:44.1534134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1534204Z return func(*args, **kwargs) 2025-08-14T21:51:44.1534484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1534586Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1534592Z 2025-08-14T21:51:44.1534693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1534889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1534959Z return mod(**inputs) 2025-08-14T21:51:44.1535247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1535316Z outputs = self.bert( 2025-08-14T21:51:44.1535611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1535681Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1535967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1536036Z layer_outputs = layer_module( 2025-08-14T21:51:44.1536252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1536353Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1536651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1536739Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1537017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1537085Z self_outputs = self.self( 2025-08-14T21:51:44.1537330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1537399Z return func(*args, **kwargs) 2025-08-14T21:51:44.1537704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1537786Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1537789Z 2025-08-14T21:51:44.1537868Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1537956Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1538068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1538254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1538324Z return mod(**inputs) 2025-08-14T21:51:44.1538601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1538672Z outputs = self.bert( 2025-08-14T21:51:44.1538951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1539023Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1539308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1539381Z layer_outputs = layer_module( 2025-08-14T21:51:44.1539604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1539680Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1539964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1540051Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1540335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1540481Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1540774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1540858Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1540862Z 2025-08-14T21:51:44.1540973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1541179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1541245Z return mod(**inputs) 2025-08-14T21:51:44.1541529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1541593Z outputs = self.bert( 2025-08-14T21:51:44.1541876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1541953Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1542234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1542329Z layer_outputs = layer_module( 2025-08-14T21:51:44.1542568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1542646Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1542942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1543024Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1543286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1543362Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1543724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1543837Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1544120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1544214Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1544217Z 2025-08-14T21:51:44.1544324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1544530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1544607Z return mod(**inputs) 2025-08-14T21:51:44.1544918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1544985Z outputs = self.bert( 2025-08-14T21:51:44.1545275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1545349Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1545647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1545723Z layer_outputs = layer_module( 2025-08-14T21:51:44.1545956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1546045Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1546347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1546436Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1546692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1546792Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1547130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1547240Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1547558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1547679Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1547892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1547969Z return self.act(input) 2025-08-14T21:51:44.1547973Z 2025-08-14T21:51:44.1548076Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1548275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1548352Z return mod(**inputs) 2025-08-14T21:51:44.1548639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1548743Z outputs = self.bert( 2025-08-14T21:51:44.1549052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1549128Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1549439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1549513Z layer_outputs = layer_module( 2025-08-14T21:51:44.1549753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1549841Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1550169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1550266Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1550542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1550621Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1550970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1551107Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1551426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1551512Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1551515Z 2025-08-14T21:51:44.1551627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1551841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1551913Z return mod(**inputs) 2025-08-14T21:51:44.1552223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1552293Z outputs = self.bert( 2025-08-14T21:51:44.1552592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1552676Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1552987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1553063Z layer_outputs = layer_module( 2025-08-14T21:51:44.1553300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1553402Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1553712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1553808Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1554094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1554190Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1554537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1554688Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1555004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1555095Z return input_tensor + hidden_states 2025-08-14T21:51:44.1555099Z 2025-08-14T21:51:44.1555222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1555456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1555526Z return mod(**inputs) 2025-08-14T21:51:44.1555961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1556040Z outputs = self.bert( 2025-08-14T21:51:44.1556359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1556438Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1556747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1556852Z layer_outputs = layer_module( 2025-08-14T21:51:44.1557092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1557181Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1557465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1557549Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1557849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1557925Z self_outputs = self.self( 2025-08-14T21:51:44.1558182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1558268Z return func(*args, **kwargs) 2025-08-14T21:51:44.1558578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1558675Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1558680Z 2025-08-14T21:51:44.1558790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1559000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1559079Z return mod(**inputs) 2025-08-14T21:51:44.1559385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1559459Z outputs = self.bert( 2025-08-14T21:51:44.1559762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1559839Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1560148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1560251Z layer_outputs = layer_module( 2025-08-14T21:51:44.1560473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1560561Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1560854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1560941Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1561228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1561297Z self_outputs = self.self( 2025-08-14T21:51:44.1561548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1561619Z return func(*args, **kwargs) 2025-08-14T21:51:44.1561905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1562008Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1562012Z 2025-08-14T21:51:44.1562115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1562332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1562398Z return mod(**inputs) 2025-08-14T21:51:44.1562681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1562755Z outputs = self.bert( 2025-08-14T21:51:44.1563036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1563115Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1563420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1563494Z layer_outputs = layer_module( 2025-08-14T21:51:44.1563716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1563793Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1564074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1564164Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1564443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1564517Z self_outputs = self.self( 2025-08-14T21:51:44.1564756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1564827Z return func(*args, **kwargs) 2025-08-14T21:51:44.1565122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1565206Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1565210Z 2025-08-14T21:51:44.1565306Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1565389Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1565494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1565703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1565771Z return mod(**inputs) 2025-08-14T21:51:44.1566072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1566147Z outputs = self.bert( 2025-08-14T21:51:44.1566468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1566549Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1566852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1566928Z layer_outputs = layer_module( 2025-08-14T21:51:44.1567165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1567240Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1567532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1567617Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1567913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1568053Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1568353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1568475Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1568487Z 2025-08-14T21:51:44.1568594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1568801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1568879Z return mod(**inputs) 2025-08-14T21:51:44.1569178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1569245Z outputs = self.bert( 2025-08-14T21:51:44.1569566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1569646Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1569951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1570025Z layer_outputs = layer_module( 2025-08-14T21:51:44.1570255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1570344Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1570652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1570739Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1571017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1571100Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1571439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1571549Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1571846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1571942Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1571945Z 2025-08-14T21:51:44.1572051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1572267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1572335Z return mod(**inputs) 2025-08-14T21:51:44.1572646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1572755Z outputs = self.bert( 2025-08-14T21:51:44.1573058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1573137Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1573452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1573527Z layer_outputs = layer_module( 2025-08-14T21:51:44.1573768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1573849Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1574162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1574257Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1574537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1574627Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1574994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1575122Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1575431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1575551Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1575771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1575853Z return self.act(input) 2025-08-14T21:51:44.1575857Z 2025-08-14T21:51:44.1575962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1576194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1576267Z return mod(**inputs) 2025-08-14T21:51:44.1576573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1576650Z outputs = self.bert( 2025-08-14T21:51:44.1576967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1577052Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1577350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1577425Z layer_outputs = layer_module( 2025-08-14T21:51:44.1577659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1577747Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1578051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1578148Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1578421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1578510Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1578843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1578980Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1579298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1579384Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1579411Z 2025-08-14T21:51:44.1579527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1579733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1579804Z return mod(**inputs) 2025-08-14T21:51:44.1580115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1580184Z outputs = self.bert( 2025-08-14T21:51:44.1580480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1580567Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1580875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1580956Z layer_outputs = layer_module( 2025-08-14T21:51:44.1581189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1581271Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1581595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1581698Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1582017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1582087Z self_outputs = self.self( 2025-08-14T21:51:44.1582326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1582405Z return func(*args, **kwargs) 2025-08-14T21:51:44.1582709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1582794Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1582805Z 2025-08-14T21:51:44.1582906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1583101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1583174Z return mod(**inputs) 2025-08-14T21:51:44.1583461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1583527Z outputs = self.bert( 2025-08-14T21:51:44.1583817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1583889Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1584179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1584250Z layer_outputs = layer_module( 2025-08-14T21:51:44.1584471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1584557Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1584840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1584920Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1585210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1585278Z self_outputs = self.self( 2025-08-14T21:51:44.1585524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1585595Z return func(*args, **kwargs) 2025-08-14T21:51:44.1585912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1585999Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1586005Z 2025-08-14T21:51:44.1586111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1586325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1586394Z return mod(**inputs) 2025-08-14T21:51:44.1586694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1586771Z outputs = self.bert( 2025-08-14T21:51:44.1587069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1587147Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1587454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1587529Z layer_outputs = layer_module( 2025-08-14T21:51:44.1587786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1587866Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1588179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1588273Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1588575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1588655Z self_outputs = self.self( 2025-08-14T21:51:44.1588906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1589000Z return func(*args, **kwargs) 2025-08-14T21:51:44.1589311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1589398Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1589401Z 2025-08-14T21:51:44.1589486Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1589578Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1589686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1589901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1589969Z return mod(**inputs) 2025-08-14T21:51:44.1590273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1590352Z outputs = self.bert( 2025-08-14T21:51:44.1590652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1590731Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1591038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1591114Z layer_outputs = layer_module( 2025-08-14T21:51:44.1591350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1591429Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1591730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1591823Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1592121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1592284Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1592584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1592672Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1592675Z 2025-08-14T21:51:44.1592793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1593000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1593074Z return mod(**inputs) 2025-08-14T21:51:44.1593385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1593455Z outputs = self.bert( 2025-08-14T21:51:44.1593768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1593852Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1594160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1594265Z layer_outputs = layer_module( 2025-08-14T21:51:44.1594515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1594605Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1594913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1595003Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1595290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1595371Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1595830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1595953Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1596270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1596367Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1596371Z 2025-08-14T21:51:44.1596482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1596696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1596775Z return mod(**inputs) 2025-08-14T21:51:44.1597086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1597165Z outputs = self.bert( 2025-08-14T21:51:44.1597488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1597565Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1597876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1597952Z layer_outputs = layer_module( 2025-08-14T21:51:44.1598193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1598276Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1598576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1598671Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1598946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1599045Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1599386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1599497Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1599806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1599925Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1600146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1600226Z return self.act(input) 2025-08-14T21:51:44.1600230Z 2025-08-14T21:51:44.1600336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1600552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1600623Z return mod(**inputs) 2025-08-14T21:51:44.1600934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1601035Z outputs = self.bert( 2025-08-14T21:51:44.1601363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1601444Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1601773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1601850Z layer_outputs = layer_module( 2025-08-14T21:51:44.1602086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1602166Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1602489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1602587Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1602859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1602944Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1603276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1603413Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1603723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1603809Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1603814Z 2025-08-14T21:51:44.1603923Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1604135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1604206Z return mod(**inputs) 2025-08-14T21:51:44.1604518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1604588Z outputs = self.bert( 2025-08-14T21:51:44.1604886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1604970Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1605271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1605352Z layer_outputs = layer_module( 2025-08-14T21:51:44.1605584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1605686Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1605990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1606080Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1606350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1606435Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1606760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1606902Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1607201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1607283Z return input_tensor + hidden_states 2025-08-14T21:51:44.1607287Z 2025-08-14T21:51:44.1607402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1607628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1607722Z return mod(**inputs) 2025-08-14T21:51:44.1608027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1608096Z outputs = self.bert( 2025-08-14T21:51:44.1608402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1608481Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1608904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1609052Z layer_outputs = layer_module( 2025-08-14T21:51:44.1609286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1609379Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1609680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1609770Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1610078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1610154Z self_outputs = self.self( 2025-08-14T21:51:44.1610418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1610495Z return func(*args, **kwargs) 2025-08-14T21:51:44.1610797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1610889Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1610894Z 2025-08-14T21:51:44.1611004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1611213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1611295Z return mod(**inputs) 2025-08-14T21:51:44.1611598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1611677Z outputs = self.bert( 2025-08-14T21:51:44.1611973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1612051Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1612362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1612465Z layer_outputs = layer_module( 2025-08-14T21:51:44.1612706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1612785Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1613087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1613180Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1613480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1613563Z self_outputs = self.self( 2025-08-14T21:51:44.1613811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1613887Z return func(*args, **kwargs) 2025-08-14T21:51:44.1614187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1614290Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1614294Z 2025-08-14T21:51:44.1614418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1614619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1614682Z return mod(**inputs) 2025-08-14T21:51:44.1614969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1615034Z outputs = self.bert( 2025-08-14T21:51:44.1615312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1615393Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1615682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1615754Z layer_outputs = layer_module( 2025-08-14T21:51:44.1615975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1616050Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1616330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1616408Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1616685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1616759Z self_outputs = self.self( 2025-08-14T21:51:44.1616992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1617067Z return func(*args, **kwargs) 2025-08-14T21:51:44.1617342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1617419Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1617424Z 2025-08-14T21:51:44.1617507Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1617584Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1617681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1617880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1617945Z return mod(**inputs) 2025-08-14T21:51:44.1618235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1618319Z outputs = self.bert( 2025-08-14T21:51:44.1618609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1618691Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1618978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1619048Z layer_outputs = layer_module( 2025-08-14T21:51:44.1619271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1619347Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1619647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1619724Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1620004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1620135Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1620424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1620533Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1620537Z 2025-08-14T21:51:44.1620637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1620827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1620900Z return mod(**inputs) 2025-08-14T21:51:44.1621180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1621245Z outputs = self.bert( 2025-08-14T21:51:44.1621544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1621617Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1621898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1621968Z layer_outputs = layer_module( 2025-08-14T21:51:44.1622180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1622261Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1622536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1622624Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1622873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1622950Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1623266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1623371Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1623663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1623745Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1623748Z 2025-08-14T21:51:44.1623857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1624054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1624117Z return mod(**inputs) 2025-08-14T21:51:44.1624394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1624488Z outputs = self.bert( 2025-08-14T21:51:44.1624772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1624854Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1625138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1625207Z layer_outputs = layer_module( 2025-08-14T21:51:44.1625433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1625509Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1625804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1625885Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1626148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1626232Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1626570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1626692Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1626998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1627117Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1627344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1627418Z return self.act(input) 2025-08-14T21:51:44.1627422Z 2025-08-14T21:51:44.1627527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1627771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1627837Z return mod(**inputs) 2025-08-14T21:51:44.1628130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1628197Z outputs = self.bert( 2025-08-14T21:51:44.1628484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1628560Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1628833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1628900Z layer_outputs = layer_module( 2025-08-14T21:51:44.1629120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1629197Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1629483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1629568Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1629824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1629907Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1630218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1630356Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1630641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1630749Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1630753Z 2025-08-14T21:51:44.1630867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1631076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1631145Z return mod(**inputs) 2025-08-14T21:51:44.1631454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1631525Z outputs = self.bert( 2025-08-14T21:51:44.1631831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1631906Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1632203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1632285Z layer_outputs = layer_module( 2025-08-14T21:51:44.1632517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1632623Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1632933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1633021Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1633328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1633400Z self_outputs = self.self( 2025-08-14T21:51:44.1633655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1633734Z return func(*args, **kwargs) 2025-08-14T21:51:44.1634050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1634145Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1634149Z 2025-08-14T21:51:44.1634258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1634461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1634539Z return mod(**inputs) 2025-08-14T21:51:44.1634843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1634919Z outputs = self.bert( 2025-08-14T21:51:44.1635216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1635293Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1635598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1635738Z layer_outputs = layer_module( 2025-08-14T21:51:44.1635985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1636079Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1636389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1636483Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1636794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1636869Z self_outputs = self.self( 2025-08-14T21:51:44.1637140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1637228Z return func(*args, **kwargs) 2025-08-14T21:51:44.1637563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1637651Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1637655Z 2025-08-14T21:51:44.1637763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1637981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1638051Z return mod(**inputs) 2025-08-14T21:51:44.1638353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1638433Z outputs = self.bert( 2025-08-14T21:51:44.1638732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1638816Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1639121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1639196Z layer_outputs = layer_module( 2025-08-14T21:51:44.1639455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1639552Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1639865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1639951Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1640252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1640333Z self_outputs = self.self( 2025-08-14T21:51:44.1640585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1640676Z return func(*args, **kwargs) 2025-08-14T21:51:44.1640985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1641071Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1641075Z 2025-08-14T21:51:44.1641168Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1641254Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1641361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1641574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1641643Z return mod(**inputs) 2025-08-14T21:51:44.1641942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1642018Z outputs = self.bert( 2025-08-14T21:51:44.1642318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1642416Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1642714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1642790Z layer_outputs = layer_module( 2025-08-14T21:51:44.1643025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1643106Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1643411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1643497Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1643795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1643961Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1644264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1644352Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1644364Z 2025-08-14T21:51:44.1644471Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1644676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1644752Z return mod(**inputs) 2025-08-14T21:51:44.1645054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1645123Z outputs = self.bert( 2025-08-14T21:51:44.1645431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1645510Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1645815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1645915Z layer_outputs = layer_module( 2025-08-14T21:51:44.1646184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1646275Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1646572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1646654Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1646916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1646992Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1647327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1647433Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1647722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1647817Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1647821Z 2025-08-14T21:51:44.1647928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1648141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1648209Z return mod(**inputs) 2025-08-14T21:51:44.1648512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1648590Z outputs = self.bert( 2025-08-14T21:51:44.1648892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1648969Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1649276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1649350Z layer_outputs = layer_module( 2025-08-14T21:51:44.1649590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1649670Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1649971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1650067Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1650347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1650445Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1650757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1650862Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1651261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1651400Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1651680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1651782Z return self.act(input) 2025-08-14T21:51:44.1651786Z 2025-08-14T21:51:44.1651930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1668946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1669180Z return mod(**inputs) 2025-08-14T21:51:44.1669554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1669736Z outputs = self.bert( 2025-08-14T21:51:44.1670080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1670161Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1670450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1670537Z layer_outputs = layer_module( 2025-08-14T21:51:44.1670764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1670857Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1671185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1671280Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1671556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1671649Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1671968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1672101Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1672379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1672472Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1672481Z 2025-08-14T21:51:44.1672595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1672804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1672883Z return mod(**inputs) 2025-08-14T21:51:44.1673176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1673253Z outputs = self.bert( 2025-08-14T21:51:44.1673538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1673616Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1673911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1673983Z layer_outputs = layer_module( 2025-08-14T21:51:44.1674242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1674328Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1674613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1674706Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1674965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1675042Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1675363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1675496Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1675878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1675972Z return input_tensor + hidden_states 2025-08-14T21:51:44.1675978Z 2025-08-14T21:51:44.1676122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1676373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1676448Z return mod(**inputs) 2025-08-14T21:51:44.1676771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1676845Z outputs = self.bert( 2025-08-14T21:51:44.1677163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1677255Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1677565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1677641Z layer_outputs = layer_module( 2025-08-14T21:51:44.1677864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1677946Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1678232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1678315Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1678589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1678670Z self_outputs = self.self( 2025-08-14T21:51:44.1678909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1678991Z return func(*args, **kwargs) 2025-08-14T21:51:44.1679268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1679350Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1679354Z 2025-08-14T21:51:44.1679466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1679662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1679728Z return mod(**inputs) 2025-08-14T21:51:44.1680014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1680078Z outputs = self.bert( 2025-08-14T21:51:44.1680365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1680440Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1680744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1680825Z layer_outputs = layer_module( 2025-08-14T21:51:44.1681057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1681144Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1681423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1681507Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1681800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1681870Z self_outputs = self.self( 2025-08-14T21:51:44.1682124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1682204Z return func(*args, **kwargs) 2025-08-14T21:51:44.1682479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1682579Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1682583Z 2025-08-14T21:51:44.1682701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1682898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1682970Z return mod(**inputs) 2025-08-14T21:51:44.1683252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1683322Z outputs = self.bert( 2025-08-14T21:51:44.1683599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1683690Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1683974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1684046Z layer_outputs = layer_module( 2025-08-14T21:51:44.1684259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1684342Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1684619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1684704Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1684980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1685048Z self_outputs = self.self( 2025-08-14T21:51:44.1685294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1685364Z return func(*args, **kwargs) 2025-08-14T21:51:44.1685652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1685733Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1685736Z 2025-08-14T21:51:44.1685822Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1685908Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1686014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1686213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1686291Z return mod(**inputs) 2025-08-14T21:51:44.1686597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1686712Z outputs = self.bert( 2025-08-14T21:51:44.1687011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1687093Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1687398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1687474Z layer_outputs = layer_module( 2025-08-14T21:51:44.1687709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1687795Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1688074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1688175Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1688451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1688580Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1688920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1689013Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1689017Z 2025-08-14T21:51:44.1689135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1689345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1689414Z return mod(**inputs) 2025-08-14T21:51:44.1689722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1689793Z outputs = self.bert( 2025-08-14T21:51:44.1690118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1690206Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1690509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1690591Z layer_outputs = layer_module( 2025-08-14T21:51:44.1690825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1690907Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1691217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1691306Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1691594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1691679Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1692015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1692140Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1692444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1692540Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1692544Z 2025-08-14T21:51:44.1692652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1692863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1692938Z return mod(**inputs) 2025-08-14T21:51:44.1693248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1693339Z outputs = self.bert( 2025-08-14T21:51:44.1693648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1693726Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1694037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1694113Z layer_outputs = layer_module( 2025-08-14T21:51:44.1694344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1694436Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1694740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1694836Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1695117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1695218Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1695569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1695684Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1695985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1696114Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1696337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1696422Z return self.act(input) 2025-08-14T21:51:44.1696426Z 2025-08-14T21:51:44.1697402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1697621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1697700Z return mod(**inputs) 2025-08-14T21:51:44.1697993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1698067Z outputs = self.bert( 2025-08-14T21:51:44.1698352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1698437Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1698720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1698788Z layer_outputs = layer_module( 2025-08-14T21:51:44.1699004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1699090Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1699367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1699457Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1699709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1699790Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1700106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1700238Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1700520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1700632Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1700635Z 2025-08-14T21:51:44.1700735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1700937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1701001Z return mod(**inputs) 2025-08-14T21:51:44.1701281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1701353Z outputs = self.bert( 2025-08-14T21:51:44.1701627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1701698Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1701983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1702057Z layer_outputs = layer_module( 2025-08-14T21:51:44.1702279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1702380Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1702693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1702785Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1703061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1703139Z self_outputs = self.self( 2025-08-14T21:51:44.1703379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1703450Z return func(*args, **kwargs) 2025-08-14T21:51:44.1703755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1703841Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1703846Z 2025-08-14T21:51:44.1703953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1704150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1704216Z return mod(**inputs) 2025-08-14T21:51:44.1704506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1704572Z outputs = self.bert( 2025-08-14T21:51:44.1704857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1704937Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1705218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1705296Z layer_outputs = layer_module( 2025-08-14T21:51:44.1705512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1705589Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1705887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1705966Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1706242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1706320Z self_outputs = self.self( 2025-08-14T21:51:44.1706562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1706657Z return func(*args, **kwargs) 2025-08-14T21:51:44.1706940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1707021Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1707024Z 2025-08-14T21:51:44.1707133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1707331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1707403Z return mod(**inputs) 2025-08-14T21:51:44.1707716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1707786Z outputs = self.bert( 2025-08-14T21:51:44.1708094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1708181Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1708469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1708565Z layer_outputs = layer_module( 2025-08-14T21:51:44.1708973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1709126Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1709447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1709535Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1709850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1709923Z self_outputs = self.self( 2025-08-14T21:51:44.1710220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1710297Z return func(*args, **kwargs) 2025-08-14T21:51:44.1710608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1710703Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1710707Z 2025-08-14T21:51:44.1710795Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1710878Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1710997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1711224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1711301Z return mod(**inputs) 2025-08-14T21:51:44.1711618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1711687Z outputs = self.bert( 2025-08-14T21:51:44.1712005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1712084Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1712397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1712480Z layer_outputs = layer_module( 2025-08-14T21:51:44.1712721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1712807Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1713119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1713204Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1713525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1713690Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1714013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1714102Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1714106Z 2025-08-14T21:51:44.1714213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1714435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1714504Z return mod(**inputs) 2025-08-14T21:51:44.1714820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1714896Z outputs = self.bert( 2025-08-14T21:51:44.1715210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1715296Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1715611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1715766Z layer_outputs = layer_module( 2025-08-14T21:51:44.1716054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1716138Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1716454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1716544Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1716831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1716945Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1717290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1717424Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1717734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1717823Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1717827Z 2025-08-14T21:51:44.1717943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1718153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1718224Z return mod(**inputs) 2025-08-14T21:51:44.1718547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1718620Z outputs = self.bert( 2025-08-14T21:51:44.1718925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1719004Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1719304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1719387Z layer_outputs = layer_module( 2025-08-14T21:51:44.1719619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1719707Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1720015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1720103Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1720403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1720484Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1720823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1720932Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1721202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1721317Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1721517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1721584Z return self.act(input) 2025-08-14T21:51:44.1721588Z 2025-08-14T21:51:44.1721692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1721883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1721953Z return mod(**inputs) 2025-08-14T21:51:44.1722240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1722318Z outputs = self.bert( 2025-08-14T21:51:44.1722595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1722664Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1722932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1723006Z layer_outputs = layer_module( 2025-08-14T21:51:44.1723214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1723315Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1723585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1723673Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1723926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1723999Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1724304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1724430Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1724702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1724790Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1724793Z 2025-08-14T21:51:44.1724890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1725085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1725149Z return mod(**inputs) 2025-08-14T21:51:44.1725423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1725493Z outputs = self.bert( 2025-08-14T21:51:44.1725771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1725841Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1726128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1726196Z layer_outputs = layer_module( 2025-08-14T21:51:44.1726439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1726514Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1726793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1726880Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1727132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1727214Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1727521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1727648Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1727933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1728010Z return input_tensor + hidden_states 2025-08-14T21:51:44.1728033Z 2025-08-14T21:51:44.1728142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1728349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1728413Z return mod(**inputs) 2025-08-14T21:51:44.1728703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1728767Z outputs = self.bert( 2025-08-14T21:51:44.1729043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1729121Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1729411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1729490Z layer_outputs = layer_module( 2025-08-14T21:51:44.1729706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1729783Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1730073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1730152Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1730439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1730507Z self_outputs = self.self( 2025-08-14T21:51:44.1730744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1730822Z return func(*args, **kwargs) 2025-08-14T21:51:44.1731104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1731183Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1731187Z 2025-08-14T21:51:44.1731291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1731482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1731552Z return mod(**inputs) 2025-08-14T21:51:44.1731840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1731905Z outputs = self.bert( 2025-08-14T21:51:44.1732200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1732274Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1732844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1732926Z layer_outputs = layer_module( 2025-08-14T21:51:44.1733155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1733248Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1733545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1733634Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1733940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1734014Z self_outputs = self.self( 2025-08-14T21:51:44.1734277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1734355Z return func(*args, **kwargs) 2025-08-14T21:51:44.1734655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1734767Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1734787Z 2025-08-14T21:51:44.1734898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1735109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1735179Z return mod(**inputs) 2025-08-14T21:51:44.1735478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1735556Z outputs = self.bert( 2025-08-14T21:51:44.1735853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1735949Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1736261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1736337Z layer_outputs = layer_module( 2025-08-14T21:51:44.1736577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1736658Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1736956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1737050Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1737349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1737424Z self_outputs = self.self( 2025-08-14T21:51:44.1737688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1737763Z return func(*args, **kwargs) 2025-08-14T21:51:44.1738070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1738155Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1738158Z 2025-08-14T21:51:44.1738245Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1738340Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1738449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1738662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1738731Z return mod(**inputs) 2025-08-14T21:51:44.1739032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1739131Z outputs = self.bert( 2025-08-14T21:51:44.1739430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1739510Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1739820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1739892Z layer_outputs = layer_module( 2025-08-14T21:51:44.1740126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1740206Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1740513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1740604Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1740908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1741078Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1741408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1741498Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1741502Z 2025-08-14T21:51:44.1741614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1741820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1741888Z return mod(**inputs) 2025-08-14T21:51:44.1742210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1742280Z outputs = self.bert( 2025-08-14T21:51:44.1742632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1742712Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1743014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1743097Z layer_outputs = layer_module( 2025-08-14T21:51:44.1743328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1743416Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1743727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1743814Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1744095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1744178Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1744518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1744640Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1744941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1745036Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1745040Z 2025-08-14T21:51:44.1745145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1745352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1745429Z return mod(**inputs) 2025-08-14T21:51:44.1745745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1745847Z outputs = self.bert( 2025-08-14T21:51:44.1746145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1746222Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1746532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1746606Z layer_outputs = layer_module( 2025-08-14T21:51:44.1746835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1746924Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1747233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1747329Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1747598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1747697Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1748049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1748161Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1748471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1748595Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1748820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1748907Z return self.act(input) 2025-08-14T21:51:44.1748913Z 2025-08-14T21:51:44.1749040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1749246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1749323Z return mod(**inputs) 2025-08-14T21:51:44.1749627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1749705Z outputs = self.bert( 2025-08-14T21:51:44.1750003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1750079Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1750388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1750463Z layer_outputs = layer_module( 2025-08-14T21:51:44.1750701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1750785Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1751086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1751185Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1751459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1751548Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1751874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1752007Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1752312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1752422Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1752426Z 2025-08-14T21:51:44.1752536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1752754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1752825Z return mod(**inputs) 2025-08-14T21:51:44.1753135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1753204Z outputs = self.bert( 2025-08-14T21:51:44.1753511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1753596Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1753896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1753975Z layer_outputs = layer_module( 2025-08-14T21:51:44.1754215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1754316Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1754641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1754730Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1755028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1755111Z self_outputs = self.self( 2025-08-14T21:51:44.1755367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1755450Z return func(*args, **kwargs) 2025-08-14T21:51:44.1755848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:51:44.1755944Z query_layer = self.query(hidden_states) 2025-08-14T21:51:44.1755950Z 2025-08-14T21:51:44.1756070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1756286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1756359Z return mod(**inputs) 2025-08-14T21:51:44.1756682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1756755Z outputs = self.bert( 2025-08-14T21:51:44.1757071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1757151Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1757449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1757530Z layer_outputs = layer_module( 2025-08-14T21:51:44.1757751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1757839Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1758123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1758204Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1758496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1758566Z self_outputs = self.self( 2025-08-14T21:51:44.1758807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1758909Z return func(*args, **kwargs) 2025-08-14T21:51:44.1759198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:51:44.1759287Z key_layer = self.key(current_states) 2025-08-14T21:51:44.1759290Z 2025-08-14T21:51:44.1759395Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1759589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1759661Z return mod(**inputs) 2025-08-14T21:51:44.1759951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1760022Z outputs = self.bert( 2025-08-14T21:51:44.1760307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1760380Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1760676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1760766Z layer_outputs = layer_module( 2025-08-14T21:51:44.1760999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1761084Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1761364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1761452Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1761734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:51:44.1761801Z self_outputs = self.self( 2025-08-14T21:51:44.1762060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:51:44.1762132Z return func(*args, **kwargs) 2025-08-14T21:51:44.1762429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:51:44.1762511Z value_layer = self.value(current_states) 2025-08-14T21:51:44.1762514Z 2025-08-14T21:51:44.1762594Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1762678Z cudagraph partition due to non gpu ops 2025-08-14T21:51:44.1762780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1762971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1763041Z return mod(**inputs) 2025-08-14T21:51:44.1763325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1763399Z outputs = self.bert( 2025-08-14T21:51:44.1763685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1763759Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1764052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1764120Z layer_outputs = layer_module( 2025-08-14T21:51:44.1764341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1764423Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1764705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:51:44.1764791Z self_attention_outputs = self.attention( 2025-08-14T21:51:44.1765075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:51:44.1765238Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:51:44.1765527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:51:44.1765608Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1765611Z 2025-08-14T21:51:44.1765719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1765916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1765982Z return mod(**inputs) 2025-08-14T21:51:44.1766278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1766344Z outputs = self.bert( 2025-08-14T21:51:44.1766634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1766714Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1767013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1767105Z layer_outputs = layer_module( 2025-08-14T21:51:44.1767323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1767398Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1767688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1767770Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1768031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1768126Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1768444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1768557Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1768846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:51:44.1768935Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1768939Z 2025-08-14T21:51:44.1769042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1769236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1769307Z return mod(**inputs) 2025-08-14T21:51:44.1769596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1769666Z outputs = self.bert( 2025-08-14T21:51:44.1769957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1770033Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1770336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1770404Z layer_outputs = layer_module( 2025-08-14T21:51:44.1770618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1770701Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1770980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1771066Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1771340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1771414Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1771728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:51:44.1771828Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:51:44.1772101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:51:44.1772221Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:44.1772424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:51:44.1772502Z return self.act(input) 2025-08-14T21:51:44.1772506Z 2025-08-14T21:51:44.1772605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1772795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1772865Z return mod(**inputs) 2025-08-14T21:51:44.1773163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1773250Z outputs = self.bert( 2025-08-14T21:51:44.1773525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1773595Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1773875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1773945Z layer_outputs = layer_module( 2025-08-14T21:51:44.1774154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1774257Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1774533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1774624Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1774878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1774955Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1775274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1775405Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1775695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:51:44.1775782Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1775786Z 2025-08-14T21:51:44.1775888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1776093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1776159Z return mod(**inputs) 2025-08-14T21:51:44.1776444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:51:44.1776516Z outputs = self.bert( 2025-08-14T21:51:44.1776797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:51:44.1776875Z encoder_outputs = self.encoder( 2025-08-14T21:51:44.1777155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:51:44.1777245Z layer_outputs = layer_module( 2025-08-14T21:51:44.1777469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:51:44.1777549Z return super().__call__(*args, **kwargs) 2025-08-14T21:51:44.1777837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:51:44.1777920Z layer_output = apply_chunking_to_forward( 2025-08-14T21:51:44.1778173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:51:44.1778255Z return forward_fn(*input_tensors) 2025-08-14T21:51:44.1778566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:51:44.1778694Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:51:44.1778985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:51:44.1779061Z return input_tensor + hidden_states 2025-08-14T21:51:44.1779083Z 2025-08-14T21:51:44.1779192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1779406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1779472Z return mod(**inputs) 2025-08-14T21:51:44.1779765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1082, in forward 2025-08-14T21:51:44.1779861Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:51:44.1780152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 652, in forward 2025-08-14T21:51:44.1780262Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:51:44.1780562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 640, in forward 2025-08-14T21:51:44.1780665Z hidden_states = self.transform(hidden_states) 2025-08-14T21:51:44.1780950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 615, in forward 2025-08-14T21:51:44.1781039Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:44.1781043Z 2025-08-14T21:51:44.1781143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1781335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1781405Z return mod(**inputs) 2025-08-14T21:51:44.1781690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1082, in forward 2025-08-14T21:51:44.1781782Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:51:44.1782076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 652, in forward 2025-08-14T21:51:44.1782189Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:51:44.1782479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 641, in forward 2025-08-14T21:51:44.1782573Z hidden_states = self.decoder(hidden_states) 2025-08-14T21:51:44.1782577Z 2025-08-14T21:51:44.1782676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:44.1782877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:44.1782942Z return mod(**inputs) 2025-08-14T21:51:44.1783242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1086, in forward 2025-08-14T21:51:44.1783331Z lm_loss = self.loss_function( 2025-08-14T21:51:44.1783574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:51:44.1783753Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:51:44.1784005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:51:44.1784197Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:51:44.1784207Z 2025-08-14T21:51:55.5637687Z Compilation time (from dynamo_timed): 24.237056263 2025-08-14T21:51:55.5669664Z pass 2025-08-14T21:51:55.5670066Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:55.5670926Z TIMING: _recursive_pre_grad_passes:0.01154 _recursive_joint_graph_passes:1.1403 _recursive_post_grad_passes:0.1366 async_compile.wait:0.89748 code_gen:9.98063 inductor_compile:12.19921 backend_compile:18.62943 gc:0.00077 entire_frame_compile:24.23706 total_wall_time:24.23706 2025-08-14T21:51:55.5672056Z STATS: call_* op count: 723 | FakeTensorMode.__torch_dispatch__:28473 | FakeTensor.__torch_dispatch__:8903 | ProxyTorchDispatchMode.__torch_dispatch__:10946 2025-08-14T21:51:55.5672910Z Dynamo produced 1 graphs covering 723 ops with 0 graph breaks (0 unique) 2025-08-14T21:52:01.3257655Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:52:01.3258741Z from pkg_resources import resource_filename 2025-08-14T21:52:02.0985149Z 2025-08-14T21:52:05.1981948Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:52:05.1983402Z loading model: 0it [00:03, ?it/s] 2025-08-14T21:52:05.2005041Z cpu eval MegatronBertForQuestionAnswering 2025-08-14T21:52:06.7492773Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:07.4040226Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:07.9988640Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:22.5444816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5445473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5445829Z return mod(**inputs) 2025-08-14T21:52:22.5446311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5446752Z outputs = self.bert( 2025-08-14T21:52:22.5447186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5447616Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5448099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5448760Z layer_outputs = layer_module( 2025-08-14T21:52:22.5449570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5450125Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5450735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5451230Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5451689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5452474Z self_outputs = self.self( 2025-08-14T21:52:22.5452875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5453290Z return func(*args, **kwargs) 2025-08-14T21:52:22.5453785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5454251Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5454405Z 2025-08-14T21:52:22.5454521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5454919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5455268Z return mod(**inputs) 2025-08-14T21:52:22.5455688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5456153Z outputs = self.bert( 2025-08-14T21:52:22.5456587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5457151Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5457667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5458126Z layer_outputs = layer_module( 2025-08-14T21:52:22.5458523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5458921Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5459371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5459832Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5460346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5460793Z self_outputs = self.self( 2025-08-14T21:52:22.5461177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5461575Z return func(*args, **kwargs) 2025-08-14T21:52:22.5462008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5462559Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5462718Z 2025-08-14T21:52:22.5462831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5463217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5463583Z return mod(**inputs) 2025-08-14T21:52:22.5464004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5464424Z outputs = self.bert( 2025-08-14T21:52:22.5464839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5465321Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5465748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5466170Z layer_outputs = layer_module( 2025-08-14T21:52:22.5466536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5466922Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5467374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5467831Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5468377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5468801Z self_outputs = self.self( 2025-08-14T21:52:22.5469166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5469541Z return func(*args, **kwargs) 2025-08-14T21:52:22.5469961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5470420Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5470571Z 2025-08-14T21:52:22.5470659Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5470890Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5471142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5471539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5471893Z return mod(**inputs) 2025-08-14T21:52:22.5472322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5472798Z outputs = self.bert( 2025-08-14T21:52:22.5473245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5473697Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5474150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5474613Z layer_outputs = layer_module( 2025-08-14T21:52:22.5474988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5475378Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5475932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5476411Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5476888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5477393Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5477861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5478296Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5478437Z 2025-08-14T21:52:22.5478551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5478908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5479239Z return mod(**inputs) 2025-08-14T21:52:22.5479639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5480055Z outputs = self.bert( 2025-08-14T21:52:22.5480441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5480883Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5481322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5481771Z layer_outputs = layer_module( 2025-08-14T21:52:22.5482147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5482530Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5482953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5483422Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5483858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5484286Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5484770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5485281Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5485765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5486239Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5486392Z 2025-08-14T21:52:22.5486515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5486905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5487268Z return mod(**inputs) 2025-08-14T21:52:22.5487707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5488179Z outputs = self.bert( 2025-08-14T21:52:22.5488609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5489059Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5489500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5489933Z layer_outputs = layer_module( 2025-08-14T21:52:22.5490298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5490698Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5491139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5491594Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5492022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5492464Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5492941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5493454Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5493918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5494401Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5494806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5495165Z return self.act(input) 2025-08-14T21:52:22.5495286Z 2025-08-14T21:52:22.5495395Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5496055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5496398Z return mod(**inputs) 2025-08-14T21:52:22.5496808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5497245Z outputs = self.bert( 2025-08-14T21:52:22.5497657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5498106Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5498544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5499013Z layer_outputs = layer_module( 2025-08-14T21:52:22.5499380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5499754Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5500199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5500650Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5501073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5501488Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5501957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5502490Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5502993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5503462Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5503637Z 2025-08-14T21:52:22.5503753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5504137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5504485Z return mod(**inputs) 2025-08-14T21:52:22.5504907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5505351Z outputs = self.bert( 2025-08-14T21:52:22.5505769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5506245Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5506691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5507131Z layer_outputs = layer_module( 2025-08-14T21:52:22.5507498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5507871Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5508316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5509115Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5509581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5510027Z self_outputs = self.self( 2025-08-14T21:52:22.5510424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5510833Z return func(*args, **kwargs) 2025-08-14T21:52:22.5511282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5511764Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5511923Z 2025-08-14T21:52:22.5512039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5512434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5512781Z return mod(**inputs) 2025-08-14T21:52:22.5513210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5513669Z outputs = self.bert( 2025-08-14T21:52:22.5514095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5514617Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5515077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5515537Z layer_outputs = layer_module( 2025-08-14T21:52:22.5516085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5516479Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5516943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5517413Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5517870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5518342Z self_outputs = self.self( 2025-08-14T21:52:22.5518743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5519231Z return func(*args, **kwargs) 2025-08-14T21:52:22.5520407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5520904Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5521053Z 2025-08-14T21:52:22.5521177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5521569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5521926Z return mod(**inputs) 2025-08-14T21:52:22.5522357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5522813Z outputs = self.bert( 2025-08-14T21:52:22.5523276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5523746Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5524209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5524667Z layer_outputs = layer_module( 2025-08-14T21:52:22.5525051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5525446Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5525905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5526365Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5526836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5527307Z self_outputs = self.self( 2025-08-14T21:52:22.5527700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5528103Z return func(*args, **kwargs) 2025-08-14T21:52:22.5528559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5529032Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5529180Z 2025-08-14T21:52:22.5529286Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5529519Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5529776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5530178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5530551Z return mod(**inputs) 2025-08-14T21:52:22.5530982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5531425Z outputs = self.bert( 2025-08-14T21:52:22.5531847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5532291Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5532737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5533183Z layer_outputs = layer_module( 2025-08-14T21:52:22.5533543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5533930Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5534385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5534833Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5535295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5535814Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5536321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5536783Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5536928Z 2025-08-14T21:52:22.5537038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5537422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5537764Z return mod(**inputs) 2025-08-14T21:52:22.5538192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5538634Z outputs = self.bert( 2025-08-14T21:52:22.5539049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5539504Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5539935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5540378Z layer_outputs = layer_module( 2025-08-14T21:52:22.5540747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5541121Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5541578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5542039Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5542466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5542880Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5543355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5543865Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5544339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5544783Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5544936Z 2025-08-14T21:52:22.5545045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5545427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5545790Z return mod(**inputs) 2025-08-14T21:52:22.5546207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5546652Z outputs = self.bert( 2025-08-14T21:52:22.5547073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5547517Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5547958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5548404Z layer_outputs = layer_module( 2025-08-14T21:52:22.5548753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5549111Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5549547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5550007Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5550442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5550874Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5551346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5551849Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5552323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5552824Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5553256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5553635Z return self.act(input) 2025-08-14T21:52:22.5553755Z 2025-08-14T21:52:22.5553871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5554263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5554620Z return mod(**inputs) 2025-08-14T21:52:22.5555048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5555504Z outputs = self.bert( 2025-08-14T21:52:22.5556034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5556509Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5556969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5557409Z layer_outputs = layer_module( 2025-08-14T21:52:22.5557777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5558142Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5558558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5558990Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5559391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5559781Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5560255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5560823Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5561333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5561754Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5561900Z 2025-08-14T21:52:22.5562007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5562371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5562711Z return mod(**inputs) 2025-08-14T21:52:22.5563131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5563570Z outputs = self.bert( 2025-08-14T21:52:22.5563982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5564420Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5564871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5565340Z layer_outputs = layer_module( 2025-08-14T21:52:22.5565728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5566113Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5566564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5567024Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5567446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5567851Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5568356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5568903Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5569411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.5569868Z return input_tensor + hidden_states 2025-08-14T21:52:22.5570014Z 2025-08-14T21:52:22.5570124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5570496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5570817Z return mod(**inputs) 2025-08-14T21:52:22.5571210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5571623Z outputs = self.bert( 2025-08-14T21:52:22.5572017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5572430Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5572845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5573264Z layer_outputs = layer_module( 2025-08-14T21:52:22.5573602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5573963Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5574422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5574865Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5575287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5575730Z self_outputs = self.self( 2025-08-14T21:52:22.5576097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5576475Z return func(*args, **kwargs) 2025-08-14T21:52:22.5576879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5577312Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5577451Z 2025-08-14T21:52:22.5577563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5577914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5578241Z return mod(**inputs) 2025-08-14T21:52:22.5578642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5579148Z outputs = self.bert( 2025-08-14T21:52:22.5579546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5579990Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5580428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5580843Z layer_outputs = layer_module( 2025-08-14T21:52:22.5581184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5581549Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5581979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5582400Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5582848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5583265Z self_outputs = self.self( 2025-08-14T21:52:22.5583625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5583993Z return func(*args, **kwargs) 2025-08-14T21:52:22.5584405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5584832Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5584967Z 2025-08-14T21:52:22.5585077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5585423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5585745Z return mod(**inputs) 2025-08-14T21:52:22.5586139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5586545Z outputs = self.bert( 2025-08-14T21:52:22.5586938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5587363Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5587765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5588157Z layer_outputs = layer_module( 2025-08-14T21:52:22.5588491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5588835Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5589234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5589668Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5590093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5590511Z self_outputs = self.self( 2025-08-14T21:52:22.5590865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5591245Z return func(*args, **kwargs) 2025-08-14T21:52:22.5591643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5592055Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5592188Z 2025-08-14T21:52:22.5592270Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5592501Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5592754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5593129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5593476Z return mod(**inputs) 2025-08-14T21:52:22.5593899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5594364Z outputs = self.bert( 2025-08-14T21:52:22.5594792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5595237Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5595747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5596198Z layer_outputs = layer_module( 2025-08-14T21:52:22.5596581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5596979Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5597454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5597892Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5598313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5598775Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5599238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5599683Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5599827Z 2025-08-14T21:52:22.5599928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5600278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5600596Z return mod(**inputs) 2025-08-14T21:52:22.5600981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5601385Z outputs = self.bert( 2025-08-14T21:52:22.5601768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5602170Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5602577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5602994Z layer_outputs = layer_module( 2025-08-14T21:52:22.5603340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5603691Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5604123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5604571Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5604960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5605346Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5605786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5606272Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5606716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5607158Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5607296Z 2025-08-14T21:52:22.5607411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5607774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5608090Z return mod(**inputs) 2025-08-14T21:52:22.5608501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5609174Z outputs = self.bert( 2025-08-14T21:52:22.5609628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5610048Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5610461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5610878Z layer_outputs = layer_module( 2025-08-14T21:52:22.5611219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5611614Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5612038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5612470Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5612867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5613261Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5613709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5614181Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5614632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5615093Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5615477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5615812Z return self.act(input) 2025-08-14T21:52:22.5615934Z 2025-08-14T21:52:22.5616037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5616399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5616714Z return mod(**inputs) 2025-08-14T21:52:22.5617111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5617527Z outputs = self.bert( 2025-08-14T21:52:22.5617916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5618330Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5618781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5619200Z layer_outputs = layer_module( 2025-08-14T21:52:22.5619546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5619903Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5620302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5620716Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5621096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5621479Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5621915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5622417Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5622886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5623382Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5623531Z 2025-08-14T21:52:22.5623634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5623992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5624310Z return mod(**inputs) 2025-08-14T21:52:22.5624709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5625122Z outputs = self.bert( 2025-08-14T21:52:22.5625501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5625908Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5626310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5626716Z layer_outputs = layer_module( 2025-08-14T21:52:22.5627046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5627408Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5627812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5628235Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5628652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5629067Z self_outputs = self.self( 2025-08-14T21:52:22.5629432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5629799Z return func(*args, **kwargs) 2025-08-14T21:52:22.5630206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5630638Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5630777Z 2025-08-14T21:52:22.5630888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5631245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5631573Z return mod(**inputs) 2025-08-14T21:52:22.5631976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5632411Z outputs = self.bert( 2025-08-14T21:52:22.5632848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5633294Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5633740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5634177Z layer_outputs = layer_module( 2025-08-14T21:52:22.5634544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5634927Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5635378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5635895Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5636360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5636805Z self_outputs = self.self( 2025-08-14T21:52:22.5637186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5637592Z return func(*args, **kwargs) 2025-08-14T21:52:22.5638029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5638450Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5638591Z 2025-08-14T21:52:22.5638703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5639085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5639429Z return mod(**inputs) 2025-08-14T21:52:22.5639850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5640300Z outputs = self.bert( 2025-08-14T21:52:22.5640717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5641168Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5641615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5642060Z layer_outputs = layer_module( 2025-08-14T21:52:22.5642432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5642820Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5643270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5643733Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5644188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5644624Z self_outputs = self.self( 2025-08-14T21:52:22.5644999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5645401Z return func(*args, **kwargs) 2025-08-14T21:52:22.5645837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5646287Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5646438Z 2025-08-14T21:52:22.5646525Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5646753Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5647003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5647374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5647742Z return mod(**inputs) 2025-08-14T21:52:22.5648155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5648551Z outputs = self.bert( 2025-08-14T21:52:22.5648942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5649370Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5649775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5650173Z layer_outputs = layer_module( 2025-08-14T21:52:22.5650511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5650859Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5651272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5651683Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5652119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5652603Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5653071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5653503Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5653656Z 2025-08-14T21:52:22.5653767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5654152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5654490Z return mod(**inputs) 2025-08-14T21:52:22.5654916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5655331Z outputs = self.bert( 2025-08-14T21:52:22.5655728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5656126Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5656528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5656930Z layer_outputs = layer_module( 2025-08-14T21:52:22.5657260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5657608Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5658016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5658438Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5658826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5659214Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5659650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5660118Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5660553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5660982Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5661119Z 2025-08-14T21:52:22.5661231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5661583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5661936Z return mod(**inputs) 2025-08-14T21:52:22.5662343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5662746Z outputs = self.bert( 2025-08-14T21:52:22.5663123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5663528Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5663929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5664336Z layer_outputs = layer_module( 2025-08-14T21:52:22.5664663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5665010Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5665418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5665849Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5666253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5666637Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5667065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5667523Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5667958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5668404Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5668802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5669133Z return self.act(input) 2025-08-14T21:52:22.5669249Z 2025-08-14T21:52:22.5669349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5669716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5670034Z return mod(**inputs) 2025-08-14T21:52:22.5670461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5670906Z outputs = self.bert( 2025-08-14T21:52:22.5671336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5671789Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5672244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5672734Z layer_outputs = layer_module( 2025-08-14T21:52:22.5673112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5673523Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5673999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5674480Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5674915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5675371Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5675966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5676580Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5677160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5677650Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5677797Z 2025-08-14T21:52:22.5677921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5678305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5678660Z return mod(**inputs) 2025-08-14T21:52:22.5679140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5679597Z outputs = self.bert( 2025-08-14T21:52:22.5680013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5680468Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5680902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5681378Z layer_outputs = layer_module( 2025-08-14T21:52:22.5681756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5682138Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5682581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5683031Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5683458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5683873Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5684362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5684843Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5685309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.5685720Z return input_tensor + hidden_states 2025-08-14T21:52:22.5685853Z 2025-08-14T21:52:22.5685961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5686313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5686635Z return mod(**inputs) 2025-08-14T21:52:22.5687030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5687436Z outputs = self.bert( 2025-08-14T21:52:22.5687842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5688250Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5688657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5689053Z layer_outputs = layer_module( 2025-08-14T21:52:22.5689392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5689751Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5690205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5690643Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5691090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5691568Z self_outputs = self.self( 2025-08-14T21:52:22.5691919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5692290Z return func(*args, **kwargs) 2025-08-14T21:52:22.5692694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5693122Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5693261Z 2025-08-14T21:52:22.5693364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5693726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5694051Z return mod(**inputs) 2025-08-14T21:52:22.5694450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5694860Z outputs = self.bert( 2025-08-14T21:52:22.5695263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5695702Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5696124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5696544Z layer_outputs = layer_module( 2025-08-14T21:52:22.5696892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5697254Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5697670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5698102Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5698575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5699013Z self_outputs = self.self( 2025-08-14T21:52:22.5699370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5699750Z return func(*args, **kwargs) 2025-08-14T21:52:22.5700167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5700593Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5700735Z 2025-08-14T21:52:22.5700841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5701203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5701532Z return mod(**inputs) 2025-08-14T21:52:22.5701937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5702385Z outputs = self.bert( 2025-08-14T21:52:22.5702807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5703257Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5703706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5704143Z layer_outputs = layer_module( 2025-08-14T21:52:22.5704494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5704856Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5705286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5705738Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5706159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5706566Z self_outputs = self.self( 2025-08-14T21:52:22.5706926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5707300Z return func(*args, **kwargs) 2025-08-14T21:52:22.5707700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5708124Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5708266Z 2025-08-14T21:52:22.5708347Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5708565Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5708937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5709306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5709636Z return mod(**inputs) 2025-08-14T21:52:22.5710082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5710504Z outputs = self.bert( 2025-08-14T21:52:22.5710933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5711378Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5711824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5712278Z layer_outputs = layer_module( 2025-08-14T21:52:22.5712660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5713091Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5713554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5714025Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5714490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5715005Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5715530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5716069Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5716221Z 2025-08-14T21:52:22.5716339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5716723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5717097Z return mod(**inputs) 2025-08-14T21:52:22.5717527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5717978Z outputs = self.bert( 2025-08-14T21:52:22.5718404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5718864Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5719322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5719777Z layer_outputs = layer_module( 2025-08-14T21:52:22.5720164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5720562Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5721067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5721527Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5721962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5722402Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5722888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5723402Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5723893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5724341Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5724486Z 2025-08-14T21:52:22.5724599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5724976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5725337Z return mod(**inputs) 2025-08-14T21:52:22.5725833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5726268Z outputs = self.bert( 2025-08-14T21:52:22.5726687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5727136Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5727582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5728025Z layer_outputs = layer_module( 2025-08-14T21:52:22.5728391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5728788Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5729230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5729686Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5730112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5730525Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5730985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5731491Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5731972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5732458Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5732852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5733213Z return self.act(input) 2025-08-14T21:52:22.5733331Z 2025-08-14T21:52:22.5733448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5733824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5734165Z return mod(**inputs) 2025-08-14T21:52:22.5734584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5735019Z outputs = self.bert( 2025-08-14T21:52:22.5735427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5735846Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5736294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5736706Z layer_outputs = layer_module( 2025-08-14T21:52:22.5737052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5737416Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5737839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5738261Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5738666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5739056Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5739507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5740031Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5740560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5741029Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5741167Z 2025-08-14T21:52:22.5741279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5741625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5741945Z return mod(**inputs) 2025-08-14T21:52:22.5742338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5742737Z outputs = self.bert( 2025-08-14T21:52:22.5743148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5743566Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5743983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5744391Z layer_outputs = layer_module( 2025-08-14T21:52:22.5744735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5745095Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5745512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5745943Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5746368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5746787Z self_outputs = self.self( 2025-08-14T21:52:22.5747143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5747523Z return func(*args, **kwargs) 2025-08-14T21:52:22.5747932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5748364Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5748505Z 2025-08-14T21:52:22.5748610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5748971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5749297Z return mod(**inputs) 2025-08-14T21:52:22.5749705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5750203Z outputs = self.bert( 2025-08-14T21:52:22.5750626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5751076Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5751516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5751953Z layer_outputs = layer_module( 2025-08-14T21:52:22.5752322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5752701Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5753135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5753591Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5754043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5754475Z self_outputs = self.self( 2025-08-14T21:52:22.5754893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5755298Z return func(*args, **kwargs) 2025-08-14T21:52:22.5755834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5756313Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5756470Z 2025-08-14T21:52:22.5756584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5756983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5757341Z return mod(**inputs) 2025-08-14T21:52:22.5757793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5758239Z outputs = self.bert( 2025-08-14T21:52:22.5758662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5759111Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5759577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5760024Z layer_outputs = layer_module( 2025-08-14T21:52:22.5760401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5760784Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5761236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5761695Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5762128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5762536Z self_outputs = self.self( 2025-08-14T21:52:22.5762902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5763276Z return func(*args, **kwargs) 2025-08-14T21:52:22.5763679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5764110Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5764246Z 2025-08-14T21:52:22.5764336Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5764562Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5764788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5765140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5765481Z return mod(**inputs) 2025-08-14T21:52:22.5765859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5766269Z outputs = self.bert( 2025-08-14T21:52:22.5766657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5767066Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5767463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5767878Z layer_outputs = layer_module( 2025-08-14T21:52:22.5768226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5768581Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5769004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5769452Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5769891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5770359Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5770823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5771245Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5771381Z 2025-08-14T21:52:22.5771491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5771838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5772171Z return mod(**inputs) 2025-08-14T21:52:22.5772589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5772999Z outputs = self.bert( 2025-08-14T21:52:22.5773399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5773831Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5774239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5774643Z layer_outputs = layer_module( 2025-08-14T21:52:22.5774984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5775349Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5775785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5776216Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5776622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5777032Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5777467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5777938Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5778377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5778795Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5778930Z 2025-08-14T21:52:22.5779035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5779408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5779725Z return mod(**inputs) 2025-08-14T21:52:22.5780112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5780504Z outputs = self.bert( 2025-08-14T21:52:22.5780895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5781301Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5781694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5782099Z layer_outputs = layer_module( 2025-08-14T21:52:22.5782435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5782795Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5783200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5783639Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5784058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5784457Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5784901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5785387Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5785845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5786306Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5786723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5787072Z return self.act(input) 2025-08-14T21:52:22.5787186Z 2025-08-14T21:52:22.5787297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5787653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5787979Z return mod(**inputs) 2025-08-14T21:52:22.5788377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5788793Z outputs = self.bert( 2025-08-14T21:52:22.5789180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5789611Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5790032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5790450Z layer_outputs = layer_module( 2025-08-14T21:52:22.5790820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5791204Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5791656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5792109Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5792537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5792954Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5793420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5793977Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5794485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5794937Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5795085Z 2025-08-14T21:52:22.5795195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5795574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5795993Z return mod(**inputs) 2025-08-14T21:52:22.5796428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5796875Z outputs = self.bert( 2025-08-14T21:52:22.5797317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5797739Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5798148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5798588Z layer_outputs = layer_module( 2025-08-14T21:52:22.5798974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5799363Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5799811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5800263Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5800660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5801051Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5801508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5802014Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5802489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.5802912Z return input_tensor + hidden_states 2025-08-14T21:52:22.5803044Z 2025-08-14T21:52:22.5803148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5803507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5803836Z return mod(**inputs) 2025-08-14T21:52:22.5804228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5804642Z outputs = self.bert( 2025-08-14T21:52:22.5805035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5805452Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5805861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5806284Z layer_outputs = layer_module( 2025-08-14T21:52:22.5806630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5806982Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5807407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5807833Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5808259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5808905Z self_outputs = self.self( 2025-08-14T21:52:22.5809313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5809713Z return func(*args, **kwargs) 2025-08-14T21:52:22.5810147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5810567Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5810713Z 2025-08-14T21:52:22.5810817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5811188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5811522Z return mod(**inputs) 2025-08-14T21:52:22.5811947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5812386Z outputs = self.bert( 2025-08-14T21:52:22.5812799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5813295Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5813769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5814211Z layer_outputs = layer_module( 2025-08-14T21:52:22.5814578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5814959Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5815416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5815876Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5816349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5816786Z self_outputs = self.self( 2025-08-14T21:52:22.5817171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5817564Z return func(*args, **kwargs) 2025-08-14T21:52:22.5818000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5818446Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5818585Z 2025-08-14T21:52:22.5818705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5819080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5819399Z return mod(**inputs) 2025-08-14T21:52:22.5819786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5820185Z outputs = self.bert( 2025-08-14T21:52:22.5820557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5820963Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5821374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5821786Z layer_outputs = layer_module( 2025-08-14T21:52:22.5822122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5822481Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5822899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5823350Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5823772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5824191Z self_outputs = self.self( 2025-08-14T21:52:22.5824555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5824922Z return func(*args, **kwargs) 2025-08-14T21:52:22.5825333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5825762Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5825895Z 2025-08-14T21:52:22.5825982Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5826188Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5826419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5826780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5827121Z return mod(**inputs) 2025-08-14T21:52:22.5827566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5828029Z outputs = self.bert( 2025-08-14T21:52:22.5828451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5828888Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5829308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5829726Z layer_outputs = layer_module( 2025-08-14T21:52:22.5830067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5830446Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5830872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5831300Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5831722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5832220Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5832725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5833189Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5833334Z 2025-08-14T21:52:22.5833444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5833826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5834175Z return mod(**inputs) 2025-08-14T21:52:22.5834587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5835031Z outputs = self.bert( 2025-08-14T21:52:22.5835455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5835965Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5836402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5836846Z layer_outputs = layer_module( 2025-08-14T21:52:22.5837213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5837601Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5838079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5838541Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5838970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5839383Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5839858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5840366Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5840841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5841291Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5841447Z 2025-08-14T21:52:22.5841560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5841943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5842311Z return mod(**inputs) 2025-08-14T21:52:22.5842761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5843203Z outputs = self.bert( 2025-08-14T21:52:22.5843620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5844057Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5844496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5844935Z layer_outputs = layer_module( 2025-08-14T21:52:22.5845323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5845708Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5846158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5846621Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5847055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5847462Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5847914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5848398Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5848846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5849315Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5849705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5850055Z return self.act(input) 2025-08-14T21:52:22.5850170Z 2025-08-14T21:52:22.5850279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5850648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5850979Z return mod(**inputs) 2025-08-14T21:52:22.5851376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5851796Z outputs = self.bert( 2025-08-14T21:52:22.5852201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5852642Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5853047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5853461Z layer_outputs = layer_module( 2025-08-14T21:52:22.5853807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5854175Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5854612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5855068Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5855485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5855870Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5856318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5856821Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5857335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5857756Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5857902Z 2025-08-14T21:52:22.5858006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5858364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5858690Z return mod(**inputs) 2025-08-14T21:52:22.5859078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5859490Z outputs = self.bert( 2025-08-14T21:52:22.5859904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5860320Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5860740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5861163Z layer_outputs = layer_module( 2025-08-14T21:52:22.5861517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5861859Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5862268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5862683Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5863099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5863504Z self_outputs = self.self( 2025-08-14T21:52:22.5863871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5864245Z return func(*args, **kwargs) 2025-08-14T21:52:22.5864651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5865082Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5865224Z 2025-08-14T21:52:22.5865328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5865696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5866011Z return mod(**inputs) 2025-08-14T21:52:22.5866403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5866497Z outputs = self.bert( 2025-08-14T21:52:22.5866788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5866865Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5867162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5867232Z layer_outputs = layer_module( 2025-08-14T21:52:22.5867448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5867532Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5867806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5867893Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5868168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5868238Z self_outputs = self.self( 2025-08-14T21:52:22.5868503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5868593Z return func(*args, **kwargs) 2025-08-14T21:52:22.5868870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5868955Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5868959Z 2025-08-14T21:52:22.5869060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5869258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5869322Z return mod(**inputs) 2025-08-14T21:52:22.5869628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5869705Z outputs = self.bert( 2025-08-14T21:52:22.5869988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5870072Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5870356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5870429Z layer_outputs = layer_module( 2025-08-14T21:52:22.5870652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5870729Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5871008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5871100Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5871381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5871459Z self_outputs = self.self( 2025-08-14T21:52:22.5871698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5871772Z return func(*args, **kwargs) 2025-08-14T21:52:22.5872076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5872158Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5872161Z 2025-08-14T21:52:22.5872252Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5872334Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5872441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5872676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5872746Z return mod(**inputs) 2025-08-14T21:52:22.5873047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5873125Z outputs = self.bert( 2025-08-14T21:52:22.5873423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5873507Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5873803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5873878Z layer_outputs = layer_module( 2025-08-14T21:52:22.5874116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5874196Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5874502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5874617Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5874936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5875084Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5875392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5875482Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5875485Z 2025-08-14T21:52:22.5875603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5875912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5876026Z return mod(**inputs) 2025-08-14T21:52:22.5876339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5876413Z outputs = self.bert( 2025-08-14T21:52:22.5876731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5876819Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5877100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5877180Z layer_outputs = layer_module( 2025-08-14T21:52:22.5877397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5877481Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5877764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5877855Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5878124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5878203Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5878526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5878634Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5878917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5879008Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5879012Z 2025-08-14T21:52:22.5879113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5879343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5879412Z return mod(**inputs) 2025-08-14T21:52:22.5879716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5879793Z outputs = self.bert( 2025-08-14T21:52:22.5880092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5880169Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5880483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5880556Z layer_outputs = layer_module( 2025-08-14T21:52:22.5880794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5880877Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5881174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5881300Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5881594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5881675Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5882020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5882130Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5882436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5882577Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5882803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5882883Z return self.act(input) 2025-08-14T21:52:22.5882887Z 2025-08-14T21:52:22.5882989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5883197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5883266Z return mod(**inputs) 2025-08-14T21:52:22.5883571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5883648Z outputs = self.bert( 2025-08-14T21:52:22.5883950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5884026Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5884342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5884419Z layer_outputs = layer_module( 2025-08-14T21:52:22.5884666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5884747Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5885034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5885125Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5885383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5885467Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5885781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5885932Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5886227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5886309Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5886313Z 2025-08-14T21:52:22.5886422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5886618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5886682Z return mod(**inputs) 2025-08-14T21:52:22.5886974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5887038Z outputs = self.bert( 2025-08-14T21:52:22.5887323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5887402Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5887681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5887778Z layer_outputs = layer_module( 2025-08-14T21:52:22.5888015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5888092Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5888393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5888479Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5888756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5888862Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5889193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5889341Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5889640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.5889722Z return input_tensor + hidden_states 2025-08-14T21:52:22.5889732Z 2025-08-14T21:52:22.5889841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5890046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5890122Z return mod(**inputs) 2025-08-14T21:52:22.5890417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5890486Z outputs = self.bert( 2025-08-14T21:52:22.5890777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5890852Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5891140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5891211Z layer_outputs = layer_module( 2025-08-14T21:52:22.5891427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5891511Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5891792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5891875Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5892188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5892258Z self_outputs = self.self( 2025-08-14T21:52:22.5892508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5892581Z return func(*args, **kwargs) 2025-08-14T21:52:22.5892865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5892954Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5892958Z 2025-08-14T21:52:22.5893061Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5893265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5893330Z return mod(**inputs) 2025-08-14T21:52:22.5893625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5893701Z outputs = self.bert( 2025-08-14T21:52:22.5893991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5894080Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5894399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5894471Z layer_outputs = layer_module( 2025-08-14T21:52:22.5894697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5894773Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5895056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5895167Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5895451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5895528Z self_outputs = self.self( 2025-08-14T21:52:22.5895770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5895841Z return func(*args, **kwargs) 2025-08-14T21:52:22.5896128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5896206Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5896210Z 2025-08-14T21:52:22.5896312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5896516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5896582Z return mod(**inputs) 2025-08-14T21:52:22.5896879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5896945Z outputs = self.bert( 2025-08-14T21:52:22.5897230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5897311Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5897593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5897669Z layer_outputs = layer_module( 2025-08-14T21:52:22.5897887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5897963Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5898253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5898355Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5898637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5898714Z self_outputs = self.self( 2025-08-14T21:52:22.5898955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5899033Z return func(*args, **kwargs) 2025-08-14T21:52:22.5899314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5899392Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5899395Z 2025-08-14T21:52:22.5899481Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5899562Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5899666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5899868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5899973Z return mod(**inputs) 2025-08-14T21:52:22.5900284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5900352Z outputs = self.bert( 2025-08-14T21:52:22.5900634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5900713Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5900995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5901071Z layer_outputs = layer_module( 2025-08-14T21:52:22.5901307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5901386Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5901676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5901759Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5902046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5902183Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5902467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5902554Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5902557Z 2025-08-14T21:52:22.5902655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5902849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5902922Z return mod(**inputs) 2025-08-14T21:52:22.5903234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5903311Z outputs = self.bert( 2025-08-14T21:52:22.5903618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5903693Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5904000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5904073Z layer_outputs = layer_module( 2025-08-14T21:52:22.5904311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5904430Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5904711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5904802Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5905069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5905142Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5905453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5905555Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5905837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5905917Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5905921Z 2025-08-14T21:52:22.5906020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5906217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5906303Z return mod(**inputs) 2025-08-14T21:52:22.5906609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5906684Z outputs = self.bert( 2025-08-14T21:52:22.5906968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5907049Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5907334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5907403Z layer_outputs = layer_module( 2025-08-14T21:52:22.5907650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5907728Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5908023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5908109Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5908368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5908456Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5908909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5909020Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5909314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5909431Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5909651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5909724Z return self.act(input) 2025-08-14T21:52:22.5909728Z 2025-08-14T21:52:22.5909832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5910035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5910101Z return mod(**inputs) 2025-08-14T21:52:22.5910443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5910513Z outputs = self.bert( 2025-08-14T21:52:22.5910828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5910973Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5911283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5911360Z layer_outputs = layer_module( 2025-08-14T21:52:22.5911611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5911694Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5912005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5912091Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5912383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5912470Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5912808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5912984Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5913326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5913414Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5913418Z 2025-08-14T21:52:22.5913534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5913740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5913818Z return mod(**inputs) 2025-08-14T21:52:22.5914123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5914193Z outputs = self.bert( 2025-08-14T21:52:22.5914563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5914642Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5914938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5915022Z layer_outputs = layer_module( 2025-08-14T21:52:22.5915254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5915341Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5915699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5915796Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5916110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5916188Z self_outputs = self.self( 2025-08-14T21:52:22.5916457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5916536Z return func(*args, **kwargs) 2025-08-14T21:52:22.5916844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5916938Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5916943Z 2025-08-14T21:52:22.5917053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5917263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5917334Z return mod(**inputs) 2025-08-14T21:52:22.5917610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5917704Z outputs = self.bert( 2025-08-14T21:52:22.5917979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5918051Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5918346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5918422Z layer_outputs = layer_module( 2025-08-14T21:52:22.5918660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5918751Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5919069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5919196Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5919508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5919601Z self_outputs = self.self( 2025-08-14T21:52:22.5919874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5919967Z return func(*args, **kwargs) 2025-08-14T21:52:22.5920285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5920370Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5920373Z 2025-08-14T21:52:22.5920483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5920703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5920772Z return mod(**inputs) 2025-08-14T21:52:22.5921096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5921177Z outputs = self.bert( 2025-08-14T21:52:22.5921486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5921572Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5921882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5921956Z layer_outputs = layer_module( 2025-08-14T21:52:22.5922205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5922287Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5922602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5922692Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5923000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5923083Z self_outputs = self.self( 2025-08-14T21:52:22.5923345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5923422Z return func(*args, **kwargs) 2025-08-14T21:52:22.5923736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5923822Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5923826Z 2025-08-14T21:52:22.5923919Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5924002Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5924112Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5924358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5924429Z return mod(**inputs) 2025-08-14T21:52:22.5924750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5924825Z outputs = self.bert( 2025-08-14T21:52:22.5925137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5925221Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5925529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5925605Z layer_outputs = layer_module( 2025-08-14T21:52:22.5925849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5925935Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5926253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5926361Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5926670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5926808Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5927082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5927172Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5927175Z 2025-08-14T21:52:22.5927277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5927473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5927573Z return mod(**inputs) 2025-08-14T21:52:22.5927865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5927932Z outputs = self.bert( 2025-08-14T21:52:22.5928212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5928283Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5928564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5928633Z layer_outputs = layer_module( 2025-08-14T21:52:22.5928844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5928927Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5929212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5929304Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5929563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5929640Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5929959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5930062Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5930343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5930429Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5930432Z 2025-08-14T21:52:22.5930554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5930769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5930836Z return mod(**inputs) 2025-08-14T21:52:22.5931121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5931195Z outputs = self.bert( 2025-08-14T21:52:22.5931483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5931560Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5931833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5931902Z layer_outputs = layer_module( 2025-08-14T21:52:22.5932128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5932206Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5932487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5932598Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5932867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5932951Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5933268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5933370Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5933659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5933791Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5934020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5934095Z return self.act(input) 2025-08-14T21:52:22.5934099Z 2025-08-14T21:52:22.5934208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5934421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5934499Z return mod(**inputs) 2025-08-14T21:52:22.5934777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5934847Z outputs = self.bert( 2025-08-14T21:52:22.5935129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5935207Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5935480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5935550Z layer_outputs = layer_module( 2025-08-14T21:52:22.5935767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5935842Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5936124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5936202Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5936451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5936534Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5936846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5936996Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5937287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5937369Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5937372Z 2025-08-14T21:52:22.5937481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5937675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5937739Z return mod(**inputs) 2025-08-14T21:52:22.5938033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5938097Z outputs = self.bert( 2025-08-14T21:52:22.5938394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5938465Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5938756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5938852Z layer_outputs = layer_module( 2025-08-14T21:52:22.5939067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5939142Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5939429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5939510Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5939774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5939867Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5940183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5940321Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5940607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.5940691Z return input_tensor + hidden_states 2025-08-14T21:52:22.5940695Z 2025-08-14T21:52:22.5940798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5940996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5941069Z return mod(**inputs) 2025-08-14T21:52:22.5941363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5941434Z outputs = self.bert( 2025-08-14T21:52:22.5941711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5941783Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5942067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5942135Z layer_outputs = layer_module( 2025-08-14T21:52:22.5942345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5942429Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5942705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5942792Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5943091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5943162Z self_outputs = self.self( 2025-08-14T21:52:22.5943411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5943484Z return func(*args, **kwargs) 2025-08-14T21:52:22.5943766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5943855Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5943859Z 2025-08-14T21:52:22.5943960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5944162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5944227Z return mod(**inputs) 2025-08-14T21:52:22.5944513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5944587Z outputs = self.bert( 2025-08-14T21:52:22.5944887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5944985Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5945267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5945337Z layer_outputs = layer_module( 2025-08-14T21:52:22.5945560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5945636Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5945916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5946025Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5946318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5946397Z self_outputs = self.self( 2025-08-14T21:52:22.5946653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5946726Z return func(*args, **kwargs) 2025-08-14T21:52:22.5947032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5947114Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5947118Z 2025-08-14T21:52:22.5947230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5947438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5947511Z return mod(**inputs) 2025-08-14T21:52:22.5947823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5947894Z outputs = self.bert( 2025-08-14T21:52:22.5948194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5948277Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5948574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5948655Z layer_outputs = layer_module( 2025-08-14T21:52:22.5948883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5948965Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5949275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5949383Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5949691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5949764Z self_outputs = self.self( 2025-08-14T21:52:22.5950028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5950105Z return func(*args, **kwargs) 2025-08-14T21:52:22.5950391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5950469Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5950479Z 2025-08-14T21:52:22.5950560Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5950638Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5950750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5950943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5951024Z return mod(**inputs) 2025-08-14T21:52:22.5951352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5951422Z outputs = self.bert( 2025-08-14T21:52:22.5951724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5951806Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5952108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5952188Z layer_outputs = layer_module( 2025-08-14T21:52:22.5952439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5952524Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5952834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5952919Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5953226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5953359Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5953662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5953755Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5953760Z 2025-08-14T21:52:22.5953867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5954083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5954150Z return mod(**inputs) 2025-08-14T21:52:22.5954456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5954533Z outputs = self.bert( 2025-08-14T21:52:22.5954831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5954907Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5955215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5955287Z layer_outputs = layer_module( 2025-08-14T21:52:22.5955523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5955627Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5956002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5956108Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5956385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5956475Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5956816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5956929Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5957236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5957322Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5957327Z 2025-08-14T21:52:22.5957430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5957634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5957723Z return mod(**inputs) 2025-08-14T21:52:22.5958037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5958105Z outputs = self.bert( 2025-08-14T21:52:22.5958391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5958472Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5958759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5958837Z layer_outputs = layer_module( 2025-08-14T21:52:22.5959085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5959163Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5959453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5959538Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5959797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5959879Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5960191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5960301Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5960584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5960698Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5960918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5960988Z return self.act(input) 2025-08-14T21:52:22.5960993Z 2025-08-14T21:52:22.5961103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5961298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5961364Z return mod(**inputs) 2025-08-14T21:52:22.5961660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5961725Z outputs = self.bert( 2025-08-14T21:52:22.5962009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5962109Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5962396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5962475Z layer_outputs = layer_module( 2025-08-14T21:52:22.5962695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5962771Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5963062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5963143Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5963415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5963496Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5963829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5964598Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5964929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5965016Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5965028Z 2025-08-14T21:52:22.5965136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5965347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5965424Z return mod(**inputs) 2025-08-14T21:52:22.5965725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5965796Z outputs = self.bert( 2025-08-14T21:52:22.5966128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5966208Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5966514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5966589Z layer_outputs = layer_module( 2025-08-14T21:52:22.5966818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5966909Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5967206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5967290Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5967596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5967671Z self_outputs = self.self( 2025-08-14T21:52:22.5967933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5968007Z return func(*args, **kwargs) 2025-08-14T21:52:22.5968307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5968398Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5968402Z 2025-08-14T21:52:22.5968511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5968723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5968793Z return mod(**inputs) 2025-08-14T21:52:22.5969092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5969191Z outputs = self.bert( 2025-08-14T21:52:22.5969490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5969566Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5969876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5969949Z layer_outputs = layer_module( 2025-08-14T21:52:22.5970185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5970266Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5970561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5970654Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5970951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5971050Z self_outputs = self.self( 2025-08-14T21:52:22.5971326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5971402Z return func(*args, **kwargs) 2025-08-14T21:52:22.5971708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.5971790Z key_layer = self.key(current_states) 2025-08-14T21:52:22.5971794Z 2025-08-14T21:52:22.5971900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5972114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5972182Z return mod(**inputs) 2025-08-14T21:52:22.5972517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5972585Z outputs = self.bert( 2025-08-14T21:52:22.5972882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5972966Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5973263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5973341Z layer_outputs = layer_module( 2025-08-14T21:52:22.5973570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5973652Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5973963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5974046Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5974326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5974401Z self_outputs = self.self( 2025-08-14T21:52:22.5974640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5974721Z return func(*args, **kwargs) 2025-08-14T21:52:22.5975018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.5975101Z value_layer = self.value(current_states) 2025-08-14T21:52:22.5975104Z 2025-08-14T21:52:22.5975195Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5975289Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.5975391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5975613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5975679Z return mod(**inputs) 2025-08-14T21:52:22.5975975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5976045Z outputs = self.bert( 2025-08-14T21:52:22.5976331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5976415Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5976700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5976780Z layer_outputs = layer_module( 2025-08-14T21:52:22.5977004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5977086Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5977381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5977491Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5977788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.5977926Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.5978206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.5978296Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5978300Z 2025-08-14T21:52:22.5978401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5978610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5978687Z return mod(**inputs) 2025-08-14T21:52:22.5978970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5979044Z outputs = self.bert( 2025-08-14T21:52:22.5979328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5979399Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5979699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5979773Z layer_outputs = layer_module( 2025-08-14T21:52:22.5980003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5980095Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5980456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5980557Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5980830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5980908Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5981252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5981366Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5981679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.5981768Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5981794Z 2025-08-14T21:52:22.5981909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5982134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5982207Z return mod(**inputs) 2025-08-14T21:52:22.5982532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5982603Z outputs = self.bert( 2025-08-14T21:52:22.5982920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5983006Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5983305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5983379Z layer_outputs = layer_module( 2025-08-14T21:52:22.5983618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5983700Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5984028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5984130Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5984404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5984494Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5984826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.5984943Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.5985268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.5985395Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.5985642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.5985717Z return self.act(input) 2025-08-14T21:52:22.5985721Z 2025-08-14T21:52:22.5985830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5986045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5986113Z return mod(**inputs) 2025-08-14T21:52:22.5986427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5986498Z outputs = self.bert( 2025-08-14T21:52:22.5986804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5986903Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5987201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5987283Z layer_outputs = layer_module( 2025-08-14T21:52:22.5987512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5987595Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5987898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5987985Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5988254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5988339Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5988669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5988831Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5989136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.5989221Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.5989224Z 2025-08-14T21:52:22.5989342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5989546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5989621Z return mod(**inputs) 2025-08-14T21:52:22.5989919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5989987Z outputs = self.bert( 2025-08-14T21:52:22.5990294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5990370Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5990686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5990784Z layer_outputs = layer_module( 2025-08-14T21:52:22.5991018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5991106Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5991407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.5991492Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.5991790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.5991874Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.5992210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.5992351Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.5992649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.5992739Z return input_tensor + hidden_states 2025-08-14T21:52:22.5992743Z 2025-08-14T21:52:22.5993184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5993404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5993475Z return mod(**inputs) 2025-08-14T21:52:22.5993789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5993870Z outputs = self.bert( 2025-08-14T21:52:22.5994179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5994259Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5994576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5994653Z layer_outputs = layer_module( 2025-08-14T21:52:22.5994898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5994981Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5995290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5995451Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5995828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5995921Z self_outputs = self.self( 2025-08-14T21:52:22.5996187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5996264Z return func(*args, **kwargs) 2025-08-14T21:52:22.5996583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.5996673Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.5996677Z 2025-08-14T21:52:22.5996795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.5997017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.5997089Z return mod(**inputs) 2025-08-14T21:52:22.5997416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.5997507Z outputs = self.bert( 2025-08-14T21:52:22.5997818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.5997924Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.5998228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.5998312Z layer_outputs = layer_module( 2025-08-14T21:52:22.5998542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.5998623Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.5998993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.5999085Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.5999382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.5999466Z self_outputs = self.self( 2025-08-14T21:52:22.5999718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.5999799Z return func(*args, **kwargs) 2025-08-14T21:52:22.6000099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6000181Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6000185Z 2025-08-14T21:52:22.6000298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6000507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6000587Z return mod(**inputs) 2025-08-14T21:52:22.6000897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6000968Z outputs = self.bert( 2025-08-14T21:52:22.6001278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6001354Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6001661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6001742Z layer_outputs = layer_module( 2025-08-14T21:52:22.6001974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6002061Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6002395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6002480Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6002788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6002861Z self_outputs = self.self( 2025-08-14T21:52:22.6003114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6003196Z return func(*args, **kwargs) 2025-08-14T21:52:22.6003508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6003599Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6003602Z 2025-08-14T21:52:22.6003688Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6003776Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6003895Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6004113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6004210Z return mod(**inputs) 2025-08-14T21:52:22.6004531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6004603Z outputs = self.bert( 2025-08-14T21:52:22.6004919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6004999Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6005306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6005390Z layer_outputs = layer_module( 2025-08-14T21:52:22.6005652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6005743Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6006058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6006143Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6006445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6006578Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6006881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6006967Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6006971Z 2025-08-14T21:52:22.6007082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6007293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6007363Z return mod(**inputs) 2025-08-14T21:52:22.6007685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6007761Z outputs = self.bert( 2025-08-14T21:52:22.6008078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6008160Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6008481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6008554Z layer_outputs = layer_module( 2025-08-14T21:52:22.6009026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6009173Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6009503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6009594Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6009871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6009960Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6010290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6010400Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6010726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6010816Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6010820Z 2025-08-14T21:52:22.6010935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6011177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6011249Z return mod(**inputs) 2025-08-14T21:52:22.6011615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6011688Z outputs = self.bert( 2025-08-14T21:52:22.6012076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6012164Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6012484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6012577Z layer_outputs = layer_module( 2025-08-14T21:52:22.6012821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6012900Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6013193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6013275Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6013543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6013620Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6013935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6014047Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6014348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6014468Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6014680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6014752Z return self.act(input) 2025-08-14T21:52:22.6014755Z 2025-08-14T21:52:22.6014865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6015060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6015125Z return mod(**inputs) 2025-08-14T21:52:22.6015445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6015514Z outputs = self.bert( 2025-08-14T21:52:22.6015830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6015928Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6016244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6016329Z layer_outputs = layer_module( 2025-08-14T21:52:22.6016575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6016666Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6017046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6017131Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6017404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6017482Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6017805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6017961Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6018286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6018376Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6018380Z 2025-08-14T21:52:22.6018482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6018679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6018752Z return mod(**inputs) 2025-08-14T21:52:22.6019041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6019132Z outputs = self.bert( 2025-08-14T21:52:22.6019410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6019485Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6019772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6019842Z layer_outputs = layer_module( 2025-08-14T21:52:22.6020059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6020145Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6020429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6020516Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6020805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6020870Z self_outputs = self.self( 2025-08-14T21:52:22.6021113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6021185Z return func(*args, **kwargs) 2025-08-14T21:52:22.6021467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6021546Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6021550Z 2025-08-14T21:52:22.6021650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6025146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6025223Z return mod(**inputs) 2025-08-14T21:52:22.6025520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6026768Z outputs = self.bert( 2025-08-14T21:52:22.6027066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6027146Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6027438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6027514Z layer_outputs = layer_module( 2025-08-14T21:52:22.6027749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6027842Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6028185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6028269Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6028558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6028644Z self_outputs = self.self( 2025-08-14T21:52:22.6028937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6029024Z return func(*args, **kwargs) 2025-08-14T21:52:22.6029327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6029415Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6029419Z 2025-08-14T21:52:22.6029532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6029743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6029820Z return mod(**inputs) 2025-08-14T21:52:22.6030142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6030217Z outputs = self.bert( 2025-08-14T21:52:22.6030533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6030610Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6030923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6030997Z layer_outputs = layer_module( 2025-08-14T21:52:22.6031230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6031322Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6031629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6031725Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6032030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6032105Z self_outputs = self.self( 2025-08-14T21:52:22.6032374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6032450Z return func(*args, **kwargs) 2025-08-14T21:52:22.6032755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6032848Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6032920Z 2025-08-14T21:52:22.6033008Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6033101Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6033212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6033420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6033522Z return mod(**inputs) 2025-08-14T21:52:22.6033827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6033895Z outputs = self.bert( 2025-08-14T21:52:22.6034214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6034291Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6034608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6034685Z layer_outputs = layer_module( 2025-08-14T21:52:22.6034925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6035017Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6035329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6035422Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6036034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6036180Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6036497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6036589Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6036593Z 2025-08-14T21:52:22.6036712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6036944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6037022Z return mod(**inputs) 2025-08-14T21:52:22.6037351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6037422Z outputs = self.bert( 2025-08-14T21:52:22.6037721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6037804Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6038102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6038187Z layer_outputs = layer_module( 2025-08-14T21:52:22.6038415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6038498Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6038813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6038903Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6039177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6039266Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6039596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6039711Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6040053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6040138Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6040144Z 2025-08-14T21:52:22.6040258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6040493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6040567Z return mod(**inputs) 2025-08-14T21:52:22.6040870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6040939Z outputs = self.bert( 2025-08-14T21:52:22.6041244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6041318Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6041615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6041698Z layer_outputs = layer_module( 2025-08-14T21:52:22.6041928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6042027Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6042302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6042397Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6042657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6042733Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6043052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6043157Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6043456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6043576Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6043786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6043863Z return self.act(input) 2025-08-14T21:52:22.6043866Z 2025-08-14T21:52:22.6043969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6044166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6044236Z return mod(**inputs) 2025-08-14T21:52:22.6044519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6044593Z outputs = self.bert( 2025-08-14T21:52:22.6044872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6044942Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6045219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6045290Z layer_outputs = layer_module( 2025-08-14T21:52:22.6045501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6045585Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6045861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6045942Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6046226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6046304Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6046625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6046775Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6047063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6047154Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6047158Z 2025-08-14T21:52:22.6047260Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6047465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6047530Z return mod(**inputs) 2025-08-14T21:52:22.6047819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6047893Z outputs = self.bert( 2025-08-14T21:52:22.6048182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6048265Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6048567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6048640Z layer_outputs = layer_module( 2025-08-14T21:52:22.6048864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6048941Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6049225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6049318Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6049588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6049674Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6049998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6050125Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6050407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.6050482Z return input_tensor + hidden_states 2025-08-14T21:52:22.6050485Z 2025-08-14T21:52:22.6050592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6050787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6050851Z return mod(**inputs) 2025-08-14T21:52:22.6051143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6051212Z outputs = self.bert( 2025-08-14T21:52:22.6051499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6051577Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6051850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6051924Z layer_outputs = layer_module( 2025-08-14T21:52:22.6052134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6052208Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6052508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6052590Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6052878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6052975Z self_outputs = self.self( 2025-08-14T21:52:22.6053237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6053320Z return func(*args, **kwargs) 2025-08-14T21:52:22.6053627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6053713Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6053725Z 2025-08-14T21:52:22.6053836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6054047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6054124Z return mod(**inputs) 2025-08-14T21:52:22.6054447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6054514Z outputs = self.bert( 2025-08-14T21:52:22.6054833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6054907Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6055203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6055271Z layer_outputs = layer_module( 2025-08-14T21:52:22.6055481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6055566Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6055858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6055939Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6056223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6056291Z self_outputs = self.self( 2025-08-14T21:52:22.6056540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6056612Z return func(*args, **kwargs) 2025-08-14T21:52:22.6056892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6056980Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6056985Z 2025-08-14T21:52:22.6057086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6057289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6057354Z return mod(**inputs) 2025-08-14T21:52:22.6057638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6057710Z outputs = self.bert( 2025-08-14T21:52:22.6057993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6058064Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6058351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6058418Z layer_outputs = layer_module( 2025-08-14T21:52:22.6058662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6058738Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6059025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6059131Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6059411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6059486Z self_outputs = self.self( 2025-08-14T21:52:22.6059721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6059791Z return func(*args, **kwargs) 2025-08-14T21:52:22.6060077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6060157Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6060161Z 2025-08-14T21:52:22.6060240Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6060334Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6060435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6060635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6060700Z return mod(**inputs) 2025-08-14T21:52:22.6061005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6061080Z outputs = self.bert( 2025-08-14T21:52:22.6061362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6061435Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6061726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6061795Z layer_outputs = layer_module( 2025-08-14T21:52:22.6062034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6062114Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6062402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6062490Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6062774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6062915Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6063225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6063316Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6063320Z 2025-08-14T21:52:22.6063435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6063642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6063711Z return mod(**inputs) 2025-08-14T21:52:22.6064022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6064090Z outputs = self.bert( 2025-08-14T21:52:22.6064397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6064475Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6064783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6064882Z layer_outputs = layer_module( 2025-08-14T21:52:22.6065100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6065187Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6065488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6065572Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6065839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6065917Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6066226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6066339Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6066623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6066725Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6066728Z 2025-08-14T21:52:22.6066836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6067040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6067136Z return mod(**inputs) 2025-08-14T21:52:22.6067444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6067518Z outputs = self.bert( 2025-08-14T21:52:22.6067803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6067878Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6068185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6068275Z layer_outputs = layer_module( 2025-08-14T21:52:22.6068508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6068608Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6068891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6068980Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6069234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6069309Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6069628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6069732Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6070022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6070138Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6070352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6070431Z return self.act(input) 2025-08-14T21:52:22.6070435Z 2025-08-14T21:52:22.6070537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6070740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6070805Z return mod(**inputs) 2025-08-14T21:52:22.6071101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6071198Z outputs = self.bert( 2025-08-14T21:52:22.6071498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6071592Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6071907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6071984Z layer_outputs = layer_module( 2025-08-14T21:52:22.6072224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6072306Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6072609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6072706Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6072983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6073066Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6073414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6073557Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6073900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6073993Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6073996Z 2025-08-14T21:52:22.6074107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6074325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6074397Z return mod(**inputs) 2025-08-14T21:52:22.6074732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6074806Z outputs = self.bert( 2025-08-14T21:52:22.6075114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6075204Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6075513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6075596Z layer_outputs = layer_module( 2025-08-14T21:52:22.6075933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6076025Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6076350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6076438Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6076749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6076836Z self_outputs = self.self( 2025-08-14T21:52:22.6077104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6077190Z return func(*args, **kwargs) 2025-08-14T21:52:22.6077501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6077590Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6077595Z 2025-08-14T21:52:22.6077714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6077962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6078042Z return mod(**inputs) 2025-08-14T21:52:22.6078356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6078451Z outputs = self.bert( 2025-08-14T21:52:22.6078772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6078853Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6079169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6079254Z layer_outputs = layer_module( 2025-08-14T21:52:22.6079499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6079591Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6079904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6079992Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6080314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6080392Z self_outputs = self.self( 2025-08-14T21:52:22.6080682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6080769Z return func(*args, **kwargs) 2025-08-14T21:52:22.6081078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6081170Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6081176Z 2025-08-14T21:52:22.6081286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6081495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6081594Z return mod(**inputs) 2025-08-14T21:52:22.6081907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6081988Z outputs = self.bert( 2025-08-14T21:52:22.6082294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6082373Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6082688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6082765Z layer_outputs = layer_module( 2025-08-14T21:52:22.6083001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6083104Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6083404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6083498Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6083802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6083872Z self_outputs = self.self( 2025-08-14T21:52:22.6084119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6084189Z return func(*args, **kwargs) 2025-08-14T21:52:22.6084482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6084579Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6084583Z 2025-08-14T21:52:22.6084662Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6084748Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6084851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6085044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6085137Z return mod(**inputs) 2025-08-14T21:52:22.6085428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6085502Z outputs = self.bert( 2025-08-14T21:52:22.6085787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6085860Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6086155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6086228Z layer_outputs = layer_module( 2025-08-14T21:52:22.6086457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6086534Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6086825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6086935Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6087223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6087351Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6087643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6087728Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6087731Z 2025-08-14T21:52:22.6087840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6088055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6088127Z return mod(**inputs) 2025-08-14T21:52:22.6088419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6088484Z outputs = self.bert( 2025-08-14T21:52:22.6088771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6088844Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6089122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6089202Z layer_outputs = layer_module( 2025-08-14T21:52:22.6089415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6089491Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6089779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6089863Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6090128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6090205Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6090522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6090634Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6090936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6091025Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6091028Z 2025-08-14T21:52:22.6091127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6091340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6091413Z return mod(**inputs) 2025-08-14T21:52:22.6091697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6091763Z outputs = self.bert( 2025-08-14T21:52:22.6092051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6092126Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6092432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6092506Z layer_outputs = layer_module( 2025-08-14T21:52:22.6092736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6092826Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6093145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6093239Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6093511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6093592Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6093927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6094036Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6094351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6094480Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6094711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6094791Z return self.act(input) 2025-08-14T21:52:22.6094794Z 2025-08-14T21:52:22.6094894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6095088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6095161Z return mod(**inputs) 2025-08-14T21:52:22.6095444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6095518Z outputs = self.bert( 2025-08-14T21:52:22.6095802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6095877Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6096165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6096235Z layer_outputs = layer_module( 2025-08-14T21:52:22.6096453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6096545Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6096842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6096939Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6097235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6097316Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6097660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6097822Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6098130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6098216Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6098220Z 2025-08-14T21:52:22.6098335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6098537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6098603Z return mod(**inputs) 2025-08-14T21:52:22.6098887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6098960Z outputs = self.bert( 2025-08-14T21:52:22.6099245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6099327Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6099633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6099709Z layer_outputs = layer_module( 2025-08-14T21:52:22.6099946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6100027Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6100344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6100431Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6100729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6100823Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6101152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6101294Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6101593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.6101673Z return input_tensor + hidden_states 2025-08-14T21:52:22.6101677Z 2025-08-14T21:52:22.6101790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6101999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6102067Z return mod(**inputs) 2025-08-14T21:52:22.6102386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6102456Z outputs = self.bert( 2025-08-14T21:52:22.6102761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6102840Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6103145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6103234Z layer_outputs = layer_module( 2025-08-14T21:52:22.6103470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6103583Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6103897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6103987Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6104322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6104399Z self_outputs = self.self( 2025-08-14T21:52:22.6104663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6104750Z return func(*args, **kwargs) 2025-08-14T21:52:22.6105059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6105155Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6105160Z 2025-08-14T21:52:22.6105268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6105480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6105570Z return mod(**inputs) 2025-08-14T21:52:22.6105872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6105943Z outputs = self.bert( 2025-08-14T21:52:22.6106265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6106343Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6106650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6106724Z layer_outputs = layer_module( 2025-08-14T21:52:22.6106953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6107042Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6107359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6107453Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6107752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6107827Z self_outputs = self.self( 2025-08-14T21:52:22.6108088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6108162Z return func(*args, **kwargs) 2025-08-14T21:52:22.6108473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6108565Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6108569Z 2025-08-14T21:52:22.6108860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6109086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6109157Z return mod(**inputs) 2025-08-14T21:52:22.6109465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6109542Z outputs = self.bert( 2025-08-14T21:52:22.6109846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6109934Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6110242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6110379Z layer_outputs = layer_module( 2025-08-14T21:52:22.6110623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6110706Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6111017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6111144Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6111452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6111538Z self_outputs = self.self( 2025-08-14T21:52:22.6111796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6111874Z return func(*args, **kwargs) 2025-08-14T21:52:22.6112191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6112280Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6112284Z 2025-08-14T21:52:22.6112382Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6112469Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6112580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6112802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6112873Z return mod(**inputs) 2025-08-14T21:52:22.6113215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6113298Z outputs = self.bert( 2025-08-14T21:52:22.6113611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6113697Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6114012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6114118Z layer_outputs = layer_module( 2025-08-14T21:52:22.6114368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6114455Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6114766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6114862Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6115172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6115321Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6115632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6115790Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6115798Z 2025-08-14T21:52:22.6115922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6116137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6116217Z return mod(**inputs) 2025-08-14T21:52:22.6116536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6116607Z outputs = self.bert( 2025-08-14T21:52:22.6116930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6117006Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6117311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6117418Z layer_outputs = layer_module( 2025-08-14T21:52:22.6117655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6117748Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6118071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6118163Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6118449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6118530Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6118873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6118984Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6119287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6119382Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6119387Z 2025-08-14T21:52:22.6119492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6119710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6119802Z return mod(**inputs) 2025-08-14T21:52:22.6120104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6120175Z outputs = self.bert( 2025-08-14T21:52:22.6120450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6120522Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6120824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6120896Z layer_outputs = layer_module( 2025-08-14T21:52:22.6121121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6121201Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6121488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6121575Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6121823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6121903Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6122211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6122310Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6122597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6122710Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6122918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6122997Z return self.act(input) 2025-08-14T21:52:22.6123001Z 2025-08-14T21:52:22.6123106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6123319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6123387Z return mod(**inputs) 2025-08-14T21:52:22.6123736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6123813Z outputs = self.bert( 2025-08-14T21:52:22.6124115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6124219Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6124528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6124599Z layer_outputs = layer_module( 2025-08-14T21:52:22.6124826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6124902Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6125189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6125279Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6125541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6125624Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6125947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6126100Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6126394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6126473Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6126477Z 2025-08-14T21:52:22.6126584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6126779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6126845Z return mod(**inputs) 2025-08-14T21:52:22.6127168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6127238Z outputs = self.bert( 2025-08-14T21:52:22.6127522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6127603Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6127882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6127961Z layer_outputs = layer_module( 2025-08-14T21:52:22.6128176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6128252Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6128543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6128625Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6128913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6128982Z self_outputs = self.self( 2025-08-14T21:52:22.6129225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6129302Z return func(*args, **kwargs) 2025-08-14T21:52:22.6129588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6129666Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6129677Z 2025-08-14T21:52:22.6129774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6129982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6130056Z return mod(**inputs) 2025-08-14T21:52:22.6130343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6130427Z outputs = self.bert( 2025-08-14T21:52:22.6130729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6130803Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6131101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6131171Z layer_outputs = layer_module( 2025-08-14T21:52:22.6131392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6131479Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6131771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6131861Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6132150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6132217Z self_outputs = self.self( 2025-08-14T21:52:22.6132478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6132551Z return func(*args, **kwargs) 2025-08-14T21:52:22.6132840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6132925Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6132930Z 2025-08-14T21:52:22.6133033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6133245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6133327Z return mod(**inputs) 2025-08-14T21:52:22.6133612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6133689Z outputs = self.bert( 2025-08-14T21:52:22.6133975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6134052Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6134353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6134426Z layer_outputs = layer_module( 2025-08-14T21:52:22.6134654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6134735Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6135022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6135115Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6135398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6135477Z self_outputs = self.self( 2025-08-14T21:52:22.6135718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6135791Z return func(*args, **kwargs) 2025-08-14T21:52:22.6136083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6136215Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6136218Z 2025-08-14T21:52:22.6136298Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6136386Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6136487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6136724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6136790Z return mod(**inputs) 2025-08-14T21:52:22.6137078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6137153Z outputs = self.bert( 2025-08-14T21:52:22.6137437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6137510Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6137801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6137872Z layer_outputs = layer_module( 2025-08-14T21:52:22.6138099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6138176Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6138460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6138563Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6138849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6138981Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6139261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6139343Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6139346Z 2025-08-14T21:52:22.6139470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6139669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6139746Z return mod(**inputs) 2025-08-14T21:52:22.6140031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6140096Z outputs = self.bert( 2025-08-14T21:52:22.6140383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6140455Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6140734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6140815Z layer_outputs = layer_module( 2025-08-14T21:52:22.6141033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6141116Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6141399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6141480Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6141745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6141820Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6142137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6142246Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6142548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6142637Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6142641Z 2025-08-14T21:52:22.6142759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6142956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6143031Z return mod(**inputs) 2025-08-14T21:52:22.6143344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6143419Z outputs = self.bert( 2025-08-14T21:52:22.6143718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6143793Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6144102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6144178Z layer_outputs = layer_module( 2025-08-14T21:52:22.6144415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6144498Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6144815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6144918Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6145173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6145249Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6145577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6145681Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6145985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6146104Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6146314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6146395Z return self.act(input) 2025-08-14T21:52:22.6146399Z 2025-08-14T21:52:22.6146499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6146702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6146768Z return mod(**inputs) 2025-08-14T21:52:22.6147055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6147130Z outputs = self.bert( 2025-08-14T21:52:22.6147416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6147488Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6147780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6147853Z layer_outputs = layer_module( 2025-08-14T21:52:22.6148078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6148154Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6148438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6148547Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6148802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6148886Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6149195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6149354Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6149644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6149725Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6149728Z 2025-08-14T21:52:22.6149827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6150028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6150092Z return mod(**inputs) 2025-08-14T21:52:22.6150390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6150454Z outputs = self.bert( 2025-08-14T21:52:22.6150725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6150804Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6151094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6151172Z layer_outputs = layer_module( 2025-08-14T21:52:22.6151380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6151458Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6151741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6151822Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6152086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6152169Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6152477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6152609Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6152890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.6152966Z return input_tensor + hidden_states 2025-08-14T21:52:22.6152969Z 2025-08-14T21:52:22.6153078Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6153272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6153343Z return mod(**inputs) 2025-08-14T21:52:22.6153632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6153698Z outputs = self.bert( 2025-08-14T21:52:22.6154008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6154084Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6154391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6154472Z layer_outputs = layer_module( 2025-08-14T21:52:22.6154708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6154814Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6155121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6155206Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6155541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6155614Z self_outputs = self.self( 2025-08-14T21:52:22.6155965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6156047Z return func(*args, **kwargs) 2025-08-14T21:52:22.6156355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6156450Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6156456Z 2025-08-14T21:52:22.6156568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6156796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6156877Z return mod(**inputs) 2025-08-14T21:52:22.6157195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6157280Z outputs = self.bert( 2025-08-14T21:52:22.6157623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6157703Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6158025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6158093Z layer_outputs = layer_module( 2025-08-14T21:52:22.6158312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6158388Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6158679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6158768Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6159048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6159115Z self_outputs = self.self( 2025-08-14T21:52:22.6159359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6159428Z return func(*args, **kwargs) 2025-08-14T21:52:22.6159748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6159824Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6159828Z 2025-08-14T21:52:22.6159927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6160128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6160193Z return mod(**inputs) 2025-08-14T21:52:22.6160482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6160547Z outputs = self.bert( 2025-08-14T21:52:22.6160827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6160907Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6161182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6161270Z layer_outputs = layer_module( 2025-08-14T21:52:22.6161499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6161575Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6161869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6161967Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6162249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6162325Z self_outputs = self.self( 2025-08-14T21:52:22.6162563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6162640Z return func(*args, **kwargs) 2025-08-14T21:52:22.6162922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6163001Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6163004Z 2025-08-14T21:52:22.6163092Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6163169Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6163270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6163469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6163550Z return mod(**inputs) 2025-08-14T21:52:22.6163838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6163901Z outputs = self.bert( 2025-08-14T21:52:22.6164177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6164255Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6164528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6164612Z layer_outputs = layer_module( 2025-08-14T21:52:22.6164834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6164910Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6165190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6165268Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6165541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6165671Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6165946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6166033Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6166038Z 2025-08-14T21:52:22.6166137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6166325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6166395Z return mod(**inputs) 2025-08-14T21:52:22.6166675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6166739Z outputs = self.bert( 2025-08-14T21:52:22.6167021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6167091Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6167386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6167455Z layer_outputs = layer_module( 2025-08-14T21:52:22.6167672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6167775Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6168058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6168160Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6168409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6168482Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6168801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6168902Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6169185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6169264Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6169269Z 2025-08-14T21:52:22.6169367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6169581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6169646Z return mod(**inputs) 2025-08-14T21:52:22.6169922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6169995Z outputs = self.bert( 2025-08-14T21:52:22.6170272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6170350Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6170640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6170711Z layer_outputs = layer_module( 2025-08-14T21:52:22.6170932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6171005Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6171285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6171370Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6171622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6171712Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6172012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6172110Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6172392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6172503Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6172711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6172778Z return self.act(input) 2025-08-14T21:52:22.6172781Z 2025-08-14T21:52:22.6172877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6173068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6173129Z return mod(**inputs) 2025-08-14T21:52:22.6173427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6173489Z outputs = self.bert( 2025-08-14T21:52:22.6173754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6173846Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6174113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6174179Z layer_outputs = layer_module( 2025-08-14T21:52:22.6174389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6174463Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6174735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6174815Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6175057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6175137Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6175434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6175581Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6175851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6175927Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6175930Z 2025-08-14T21:52:22.6176033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6176221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6176283Z return mod(**inputs) 2025-08-14T21:52:22.6176574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6176639Z outputs = self.bert( 2025-08-14T21:52:22.6176913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6176985Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6177265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6177345Z layer_outputs = layer_module( 2025-08-14T21:52:22.6177564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6177650Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6177931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6178015Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6178306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6178376Z self_outputs = self.self( 2025-08-14T21:52:22.6178618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6178696Z return func(*args, **kwargs) 2025-08-14T21:52:22.6178989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6179077Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6179081Z 2025-08-14T21:52:22.6179198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6179385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6179458Z return mod(**inputs) 2025-08-14T21:52:22.6179737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6179827Z outputs = self.bert( 2025-08-14T21:52:22.6180106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6180175Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6180460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6180529Z layer_outputs = layer_module( 2025-08-14T21:52:22.6180740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6180825Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6181101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6181188Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6181467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6181553Z self_outputs = self.self( 2025-08-14T21:52:22.6181795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6181864Z return func(*args, **kwargs) 2025-08-14T21:52:22.6182147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6182225Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6182228Z 2025-08-14T21:52:22.6182325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6182538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6182603Z return mod(**inputs) 2025-08-14T21:52:22.6182881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6182951Z outputs = self.bert( 2025-08-14T21:52:22.6183226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6183302Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6183573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6183640Z layer_outputs = layer_module( 2025-08-14T21:52:22.6183860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6183937Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6184222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6184311Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6184601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6184676Z self_outputs = self.self( 2025-08-14T21:52:22.6184909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6184978Z return func(*args, **kwargs) 2025-08-14T21:52:22.6185265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6185356Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6185360Z 2025-08-14T21:52:22.6185447Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6185526Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6185627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6185858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6185927Z return mod(**inputs) 2025-08-14T21:52:22.6186238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6186318Z outputs = self.bert( 2025-08-14T21:52:22.6186623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6186709Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6187017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6187094Z layer_outputs = layer_module( 2025-08-14T21:52:22.6187340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6187425Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6187753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6187840Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6188139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6188285Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6188580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6188666Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6188677Z 2025-08-14T21:52:22.6188801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6188997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6189072Z return mod(**inputs) 2025-08-14T21:52:22.6189359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6189423Z outputs = self.bert( 2025-08-14T21:52:22.6189712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6189784Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6190077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6190152Z layer_outputs = layer_module( 2025-08-14T21:52:22.6190381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6190466Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6190768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6190858Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6191137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6191219Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6191559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6191687Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6191987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6192080Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6192101Z 2025-08-14T21:52:22.6192210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6192425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6192496Z return mod(**inputs) 2025-08-14T21:52:22.6192799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6192876Z outputs = self.bert( 2025-08-14T21:52:22.6193182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6193262Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6193567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6193644Z layer_outputs = layer_module( 2025-08-14T21:52:22.6193881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6193962Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6194274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6194373Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6194645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6194734Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6195065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6195175Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6195507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6195631Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6195931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6196019Z return self.act(input) 2025-08-14T21:52:22.6196024Z 2025-08-14T21:52:22.6196131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6196346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6196423Z return mod(**inputs) 2025-08-14T21:52:22.6196727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6196808Z outputs = self.bert( 2025-08-14T21:52:22.6197111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6197200Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6197496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6197574Z layer_outputs = layer_module( 2025-08-14T21:52:22.6197816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6197897Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6198193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6198312Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6198583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6198672Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6199005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6199167Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6199478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6199565Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6199568Z 2025-08-14T21:52:22.6199681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6199888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6199960Z return mod(**inputs) 2025-08-14T21:52:22.6200275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6200346Z outputs = self.bert( 2025-08-14T21:52:22.6200653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6200729Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6201048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6201131Z layer_outputs = layer_module( 2025-08-14T21:52:22.6201360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6201442Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6201750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6201836Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6202130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6202213Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6202550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6202694Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6203000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.6203089Z return input_tensor + hidden_states 2025-08-14T21:52:22.6203093Z 2025-08-14T21:52:22.6203201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6203404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6203480Z return mod(**inputs) 2025-08-14T21:52:22.6203784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6203855Z outputs = self.bert( 2025-08-14T21:52:22.6204162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6204240Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6204553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6204628Z layer_outputs = layer_module( 2025-08-14T21:52:22.6204872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6204982Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6205287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6205382Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6205700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6205774Z self_outputs = self.self( 2025-08-14T21:52:22.6206125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6206444Z return func(*args, **kwargs) 2025-08-14T21:52:22.6206779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6206966Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6206971Z 2025-08-14T21:52:22.6207104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6207338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6207450Z return mod(**inputs) 2025-08-14T21:52:22.6207818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6207957Z outputs = self.bert( 2025-08-14T21:52:22.6208295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6208399Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6208877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6208970Z layer_outputs = layer_module( 2025-08-14T21:52:22.6209308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6209420Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6209794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6209940Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6210264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6210349Z self_outputs = self.self( 2025-08-14T21:52:22.6210698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6210796Z return func(*args, **kwargs) 2025-08-14T21:52:22.6211150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6211257Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6211262Z 2025-08-14T21:52:22.6211397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6211686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6211794Z return mod(**inputs) 2025-08-14T21:52:22.6212176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6212272Z outputs = self.bert( 2025-08-14T21:52:22.6212598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6212719Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6213070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6213239Z layer_outputs = layer_module( 2025-08-14T21:52:22.6213496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6213603Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6213992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6214090Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6214436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6214574Z self_outputs = self.self( 2025-08-14T21:52:22.6214852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6214988Z return func(*args, **kwargs) 2025-08-14T21:52:22.6215321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6215410Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6215415Z 2025-08-14T21:52:22.6215573Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6215674Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6215821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6216041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6216155Z return mod(**inputs) 2025-08-14T21:52:22.6216526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6216626Z outputs = self.bert( 2025-08-14T21:52:22.6216924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6217053Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6218239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6218374Z layer_outputs = layer_module( 2025-08-14T21:52:22.6218661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6218772Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6219118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6219226Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6219550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6219684Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6220003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6220151Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6220155Z 2025-08-14T21:52:22.6220276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6220526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6220614Z return mod(**inputs) 2025-08-14T21:52:22.6237581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6237833Z outputs = self.bert( 2025-08-14T21:52:22.6238195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6238280Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6238694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6238775Z layer_outputs = layer_module( 2025-08-14T21:52:22.6239027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6239151Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6239445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6239546Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6239816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6239906Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6240227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6240342Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6240642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6240731Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6240737Z 2025-08-14T21:52:22.6240859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6241117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6241191Z return mod(**inputs) 2025-08-14T21:52:22.6241489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6241558Z outputs = self.bert( 2025-08-14T21:52:22.6241840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6241929Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6242243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6242332Z layer_outputs = layer_module( 2025-08-14T21:52:22.6242561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6242642Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6242955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6243046Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6243328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6243421Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6243766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6243889Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6244202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6244319Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6244545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6244617Z return self.act(input) 2025-08-14T21:52:22.6244622Z 2025-08-14T21:52:22.6244741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6244951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6245038Z return mod(**inputs) 2025-08-14T21:52:22.6245335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6245404Z outputs = self.bert( 2025-08-14T21:52:22.6245688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6245788Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6246072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6246152Z layer_outputs = layer_module( 2025-08-14T21:52:22.6246379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6246461Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6246748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6246834Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6247096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6247174Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6247485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6247648Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6247932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6248021Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6248025Z 2025-08-14T21:52:22.6248131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6248334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6248408Z return mod(**inputs) 2025-08-14T21:52:22.6248709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6248779Z outputs = self.bert( 2025-08-14T21:52:22.6249072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6249151Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6249443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6249515Z layer_outputs = layer_module( 2025-08-14T21:52:22.6249735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6249824Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6250110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6250204Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6250491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6250564Z self_outputs = self.self( 2025-08-14T21:52:22.6250823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6250896Z return func(*args, **kwargs) 2025-08-14T21:52:22.6251180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6251269Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6251291Z 2025-08-14T21:52:22.6251397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6251602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6251669Z return mod(**inputs) 2025-08-14T21:52:22.6251952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6252045Z outputs = self.bert( 2025-08-14T21:52:22.6252334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6252418Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6252715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6252785Z layer_outputs = layer_module( 2025-08-14T21:52:22.6253008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6253087Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6253366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6253457Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6253735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6253828Z self_outputs = self.self( 2025-08-14T21:52:22.6254066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6254136Z return func(*args, **kwargs) 2025-08-14T21:52:22.6254418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6254497Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6254501Z 2025-08-14T21:52:22.6254607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6254841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6254907Z return mod(**inputs) 2025-08-14T21:52:22.6255202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6255268Z outputs = self.bert( 2025-08-14T21:52:22.6255555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6255638Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6255929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6256008Z layer_outputs = layer_module( 2025-08-14T21:52:22.6256223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6256300Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6256600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6256690Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6256991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6257082Z self_outputs = self.self( 2025-08-14T21:52:22.6257324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6257403Z return func(*args, **kwargs) 2025-08-14T21:52:22.6257689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6257803Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6257806Z 2025-08-14T21:52:22.6257900Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6257981Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6258110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6258310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6258376Z return mod(**inputs) 2025-08-14T21:52:22.6258676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6258743Z outputs = self.bert( 2025-08-14T21:52:22.6259026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6259109Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6259392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6259472Z layer_outputs = layer_module( 2025-08-14T21:52:22.6259691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6259768Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6260090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6260172Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6260466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6260593Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6260869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6260955Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6260976Z 2025-08-14T21:52:22.6261075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6261266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6261338Z return mod(**inputs) 2025-08-14T21:52:22.6261618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6261689Z outputs = self.bert( 2025-08-14T21:52:22.6261966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6262037Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6262319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6262390Z layer_outputs = layer_module( 2025-08-14T21:52:22.6262610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6262687Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6262972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6263067Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6263326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6263413Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6263729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6263848Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6264132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6264213Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6264232Z 2025-08-14T21:52:22.6264333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6264531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6264599Z return mod(**inputs) 2025-08-14T21:52:22.6264895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6264961Z outputs = self.bert( 2025-08-14T21:52:22.6265241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6265325Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6265612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6265682Z layer_outputs = layer_module( 2025-08-14T21:52:22.6265910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6265987Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6266290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6266375Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6266641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6266726Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6267033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6267157Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6267439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6267552Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6267768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6267838Z return self.act(input) 2025-08-14T21:52:22.6267842Z 2025-08-14T21:52:22.6267941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6268143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6268207Z return mod(**inputs) 2025-08-14T21:52:22.6268497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6268560Z outputs = self.bert( 2025-08-14T21:52:22.6268849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6268925Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6269221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6269294Z layer_outputs = layer_module( 2025-08-14T21:52:22.6269519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6269597Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6269885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6269995Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6270258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6270337Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6270680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6270815Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6271108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6271190Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6271193Z 2025-08-14T21:52:22.6271297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6271506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6271573Z return mod(**inputs) 2025-08-14T21:52:22.6271869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6271938Z outputs = self.bert( 2025-08-14T21:52:22.6272244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6272330Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6272660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6272744Z layer_outputs = layer_module( 2025-08-14T21:52:22.6272974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6273057Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6273365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6273468Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6273749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6273838Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6274175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6274321Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6274621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.6274703Z return input_tensor + hidden_states 2025-08-14T21:52:22.6274708Z 2025-08-14T21:52:22.6274824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6275035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6275115Z return mod(**inputs) 2025-08-14T21:52:22.6275417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6275490Z outputs = self.bert( 2025-08-14T21:52:22.6275913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6275997Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6276297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6276383Z layer_outputs = layer_module( 2025-08-14T21:52:22.6276619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6276736Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6277046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6277153Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6277474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6277553Z self_outputs = self.self( 2025-08-14T21:52:22.6277821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6277894Z return func(*args, **kwargs) 2025-08-14T21:52:22.6278180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6278272Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6278278Z 2025-08-14T21:52:22.6278380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6278589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6278667Z return mod(**inputs) 2025-08-14T21:52:22.6278951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6279044Z outputs = self.bert( 2025-08-14T21:52:22.6279330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6279405Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6279711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6279788Z layer_outputs = layer_module( 2025-08-14T21:52:22.6280023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6280119Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6280424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6280518Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6280819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6280891Z self_outputs = self.self( 2025-08-14T21:52:22.6281138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6281208Z return func(*args, **kwargs) 2025-08-14T21:52:22.6281499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6281580Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6281583Z 2025-08-14T21:52:22.6281686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6281887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6281955Z return mod(**inputs) 2025-08-14T21:52:22.6282245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6282309Z outputs = self.bert( 2025-08-14T21:52:22.6282590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6282670Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6282953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6283042Z layer_outputs = layer_module( 2025-08-14T21:52:22.6283283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6283363Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6283698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6283785Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6284094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6284176Z self_outputs = self.self( 2025-08-14T21:52:22.6284443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6284527Z return func(*args, **kwargs) 2025-08-14T21:52:22.6284833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6284922Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6284925Z 2025-08-14T21:52:22.6285025Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6285122Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6285231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6285467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6285540Z return mod(**inputs) 2025-08-14T21:52:22.6285853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6285923Z outputs = self.bert( 2025-08-14T21:52:22.6286225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6286312Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6286630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6286707Z layer_outputs = layer_module( 2025-08-14T21:52:22.6286948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6287029Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6287337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6287422Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6287722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6287870Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6288171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6288266Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6288270Z 2025-08-14T21:52:22.6288378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6288584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6288663Z return mod(**inputs) 2025-08-14T21:52:22.6288964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6289033Z outputs = self.bert( 2025-08-14T21:52:22.6289344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6289445Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6289757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6289832Z layer_outputs = layer_module( 2025-08-14T21:52:22.6290066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6290174Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6290477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6290573Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6290844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6290924Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6291269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6291381Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6291684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6291779Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6291782Z 2025-08-14T21:52:22.6291889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6292117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6292189Z return mod(**inputs) 2025-08-14T21:52:22.6292489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6292566Z outputs = self.bert( 2025-08-14T21:52:22.6292879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6292967Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6293282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6293359Z layer_outputs = layer_module( 2025-08-14T21:52:22.6293602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6293685Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6294001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6294087Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6294357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6294446Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6294781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6294896Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6295199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6295320Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6295551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6295624Z return self.act(input) 2025-08-14T21:52:22.6295628Z 2025-08-14T21:52:22.6295734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6295948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6296035Z return mod(**inputs) 2025-08-14T21:52:22.6296348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6296419Z outputs = self.bert( 2025-08-14T21:52:22.6296722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6296829Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6297128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6297209Z layer_outputs = layer_module( 2025-08-14T21:52:22.6297438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6297519Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6297823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6297912Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6298182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6298270Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6298622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6298771Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6299068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6299154Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6299158Z 2025-08-14T21:52:22.6299278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6299485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6299561Z return mod(**inputs) 2025-08-14T21:52:22.6299934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6300006Z outputs = self.bert( 2025-08-14T21:52:22.6300313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6300390Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6300688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6300770Z layer_outputs = layer_module( 2025-08-14T21:52:22.6300998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6301087Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6301386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6301472Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6301776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6301849Z self_outputs = self.self( 2025-08-14T21:52:22.6302113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6302189Z return func(*args, **kwargs) 2025-08-14T21:52:22.6302484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:52:22.6302576Z query_layer = self.query(hidden_states) 2025-08-14T21:52:22.6302597Z 2025-08-14T21:52:22.6302707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6302913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6302989Z return mod(**inputs) 2025-08-14T21:52:22.6303314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6303389Z outputs = self.bert( 2025-08-14T21:52:22.6303687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6303763Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6304067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6304142Z layer_outputs = layer_module( 2025-08-14T21:52:22.6304380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6304462Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6304763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6304859Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6305176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6305251Z self_outputs = self.self( 2025-08-14T21:52:22.6305511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6305585Z return func(*args, **kwargs) 2025-08-14T21:52:22.6305890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:52:22.6305973Z key_layer = self.key(current_states) 2025-08-14T21:52:22.6305977Z 2025-08-14T21:52:22.6306083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6306315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6306389Z return mod(**inputs) 2025-08-14T21:52:22.6306698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6306769Z outputs = self.bert( 2025-08-14T21:52:22.6307068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6307146Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6307428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6307500Z layer_outputs = layer_module( 2025-08-14T21:52:22.6307727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6307804Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6308095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6308176Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6308466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:52:22.6308544Z self_outputs = self.self( 2025-08-14T21:52:22.6309002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:52:22.6309086Z return func(*args, **kwargs) 2025-08-14T21:52:22.6310400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:52:22.6311118Z value_layer = self.value(current_states) 2025-08-14T21:52:22.6311127Z 2025-08-14T21:52:22.6311261Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6311348Z cudagraph partition due to non gpu ops 2025-08-14T21:52:22.6311537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6311786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6311944Z return mod(**inputs) 2025-08-14T21:52:22.6312309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6312385Z outputs = self.bert( 2025-08-14T21:52:22.6312696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6312792Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6313112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6313198Z layer_outputs = layer_module( 2025-08-14T21:52:22.6313458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6313553Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6313937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:52:22.6314031Z self_attention_outputs = self.attention( 2025-08-14T21:52:22.6314350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:52:22.6314506Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:52:22.6314830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:52:22.6314983Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6314989Z 2025-08-14T21:52:22.6315111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6315347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6315433Z return mod(**inputs) 2025-08-14T21:52:22.6316078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6316159Z outputs = self.bert( 2025-08-14T21:52:22.6316490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6316574Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6316897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6316977Z layer_outputs = layer_module( 2025-08-14T21:52:22.6317222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6317317Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6317627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6317725Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6317999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6318081Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6318421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6318583Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6318893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:52:22.6319009Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6319013Z 2025-08-14T21:52:22.6319123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6319348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6319419Z return mod(**inputs) 2025-08-14T21:52:22.6319729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6319800Z outputs = self.bert( 2025-08-14T21:52:22.6320066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6320147Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6320414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6320481Z layer_outputs = layer_module( 2025-08-14T21:52:22.6320697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6320772Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6321064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6321150Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6321392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6321471Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6321768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:52:22.6321883Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:52:22.6322165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:52:22.6322276Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:22.6322489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:52:22.6322557Z return self.act(input) 2025-08-14T21:52:22.6322560Z 2025-08-14T21:52:22.6322662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6322859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6322922Z return mod(**inputs) 2025-08-14T21:52:22.6323204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6323268Z outputs = self.bert( 2025-08-14T21:52:22.6323547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6323626Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6323897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6323966Z layer_outputs = layer_module( 2025-08-14T21:52:22.6324183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6324257Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6324538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6324644Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6324894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6324975Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6325303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6325446Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6325730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:52:22.6325807Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:22.6325810Z 2025-08-14T21:52:22.6325914Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6326103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6326166Z return mod(**inputs) 2025-08-14T21:52:22.6326446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:52:22.6326509Z outputs = self.bert( 2025-08-14T21:52:22.6326785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:52:22.6326880Z encoder_outputs = self.encoder( 2025-08-14T21:52:22.6327153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:52:22.6327229Z layer_outputs = layer_module( 2025-08-14T21:52:22.6327437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:22.6327523Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:22.6327788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:52:22.6327883Z layer_output = apply_chunking_to_forward( 2025-08-14T21:52:22.6328134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:52:22.6328208Z return forward_fn(*input_tensors) 2025-08-14T21:52:22.6328506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:52:22.6328639Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:52:22.6328908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:52:22.6328990Z return input_tensor + hidden_states 2025-08-14T21:52:22.6328996Z 2025-08-14T21:52:22.6329092Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6329280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6329351Z return mod(**inputs) 2025-08-14T21:52:22.6329630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1611, in forward 2025-08-14T21:52:22.6329719Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:52:22.6329723Z 2025-08-14T21:52:22.6329825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6330016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6330090Z return mod(**inputs) 2025-08-14T21:52:22.6330370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1629, in forward 2025-08-14T21:52:22.6330496Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:52:22.6330507Z 2025-08-14T21:52:22.6330604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:22.6330795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:22.6330886Z return mod(**inputs) 2025-08-14T21:52:22.6331167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1630, in forward 2025-08-14T21:52:22.6331259Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:52:22.6331263Z 2025-08-14T21:52:32.6493015Z Compilation time (from dynamo_timed): 22.990534169 2025-08-14T21:52:32.6493378Z pass 2025-08-14T21:52:32.6493725Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:32.6494561Z TIMING: _recursive_pre_grad_passes:0.01152 _recursive_joint_graph_passes:1.15566 _recursive_post_grad_passes:0.14789 async_compile.wait:0.00324 code_gen:8.71354 inductor_compile:10.99666 backend_compile:17.46539 gc:0.00106 entire_frame_compile:22.99053 total_wall_time:22.99053 2025-08-14T21:52:32.6495663Z STATS: call_* op count: 724 | FakeTensorMode.__torch_dispatch__:28476 | FakeTensor.__torch_dispatch__:8921 | ProxyTorchDispatchMode.__torch_dispatch__:10973 2025-08-14T21:52:32.6496200Z Dynamo produced 1 graphs covering 724 ops with 0 graph breaks (0 unique) 2025-08-14T21:52:38.3991237Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:52:38.3992211Z from pkg_resources import resource_filename 2025-08-14T21:52:38.9764664Z 2025-08-14T21:52:39.6880910Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:52:39.6887181Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:52:39.6953537Z cpu eval MobileBertForMaskedLM 2025-08-14T21:52:40.0123423Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:40.1765294Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:40.3369136Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:07.5284362Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5288030Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5291159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5294261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5294757Z return mod(**inputs) 2025-08-14T21:53:07.5295221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5295705Z outputs = self.mobilebert( 2025-08-14T21:53:07.5296164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:53:07.5296610Z embedding_output = self.embeddings( 2025-08-14T21:53:07.5297110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-08-14T21:53:07.5297552Z inputs_embeds = torch.cat( 2025-08-14T21:53:07.5297677Z 2025-08-14T21:53:07.5297810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5298236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5298598Z return mod(**inputs) 2025-08-14T21:53:07.5299027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-14T21:53:07.5299981Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:53:07.5300448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-14T21:53:07.5300942Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:53:07.5301521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-08-14T21:53:07.5302123Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-08-14T21:53:07.5302406Z 2025-08-14T21:53:07.5302521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5302949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5303313Z return mod(**inputs) 2025-08-14T21:53:07.5303728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5304189Z outputs = self.mobilebert( 2025-08-14T21:53:07.5304626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:53:07.5305081Z embedding_output = self.embeddings( 2025-08-14T21:53:07.5305550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-08-14T21:53:07.5306095Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-08-14T21:53:07.5306286Z 2025-08-14T21:53:07.5306405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5306801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5307146Z return mod(**inputs) 2025-08-14T21:53:07.5307565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5308017Z outputs = self.mobilebert( 2025-08-14T21:53:07.5308540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:53:07.5309143Z embedding_output = self.embeddings( 2025-08-14T21:53:07.5309596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-08-14T21:53:07.5310068Z embeddings = self.LayerNorm(embeddings) 2025-08-14T21:53:07.5310539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5311024Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5311190Z 2025-08-14T21:53:07.5311312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5311726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5312089Z return mod(**inputs) 2025-08-14T21:53:07.5312516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5312991Z outputs = self.mobilebert( 2025-08-14T21:53:07.5313471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5313935Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5314391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5314852Z layer_outputs = layer_module( 2025-08-14T21:53:07.5315294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5315989Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5316597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5317092Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5317568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5318066Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5318219Z 2025-08-14T21:53:07.5318342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5318727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5319100Z return mod(**inputs) 2025-08-14T21:53:07.5319574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5320032Z outputs = self.mobilebert( 2025-08-14T21:53:07.5320488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5320969Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5321423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5321886Z layer_outputs = layer_module( 2025-08-14T21:53:07.5322360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5322823Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5323289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5323742Z self_outputs = self.self( 2025-08-14T21:53:07.5324194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.5324644Z self.value(value_tensor) 2025-08-14T21:53:07.5324772Z 2025-08-14T21:53:07.5324929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5325314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5325667Z return mod(**inputs) 2025-08-14T21:53:07.5326092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5326540Z outputs = self.mobilebert( 2025-08-14T21:53:07.5326972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5327418Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5327852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5328278Z layer_outputs = layer_module( 2025-08-14T21:53:07.5328713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5329249Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5329799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.5330291Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.5330841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5331296Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5331438Z 2025-08-14T21:53:07.5331553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5331922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5332285Z return mod(**inputs) 2025-08-14T21:53:07.5332695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5333123Z outputs = self.mobilebert( 2025-08-14T21:53:07.5333557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5334013Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5334459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5334893Z layer_outputs = layer_module( 2025-08-14T21:53:07.5335323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5335851Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5336381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5336867Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5337340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.5337799Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.5338266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5338724Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5338892Z 2025-08-14T21:53:07.5339003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5339382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5339795Z return mod(**inputs) 2025-08-14T21:53:07.5340257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5340695Z outputs = self.mobilebert( 2025-08-14T21:53:07.5341112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5341546Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5341979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5342412Z layer_outputs = layer_module( 2025-08-14T21:53:07.5342837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5343288Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5343736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5344169Z self_outputs = self.self( 2025-08-14T21:53:07.5344582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.5345015Z self.query(query_tensor) 2025-08-14T21:53:07.5345137Z 2025-08-14T21:53:07.5345256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5345632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5345971Z return mod(**inputs) 2025-08-14T21:53:07.5346379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5346808Z outputs = self.mobilebert( 2025-08-14T21:53:07.5347215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5347672Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5348102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5348549Z layer_outputs = layer_module( 2025-08-14T21:53:07.5348964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5349414Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5349861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5350295Z self_outputs = self.self( 2025-08-14T21:53:07.5350719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.5351177Z self.key(key_tensor) 2025-08-14T21:53:07.5351289Z 2025-08-14T21:53:07.5351390Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5351620Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5351883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5352269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5352620Z return mod(**inputs) 2025-08-14T21:53:07.5353078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5353532Z outputs = self.mobilebert( 2025-08-14T21:53:07.5353965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5354408Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5354862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5355311Z layer_outputs = layer_module( 2025-08-14T21:53:07.5355968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5356455Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5356922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5357426Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5357912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.5358393Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5358554Z 2025-08-14T21:53:07.5358668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5359055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5359398Z return mod(**inputs) 2025-08-14T21:53:07.5359812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5360255Z outputs = self.mobilebert( 2025-08-14T21:53:07.5360689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5361132Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5361572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5362015Z layer_outputs = layer_module( 2025-08-14T21:53:07.5362440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5362898Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5363405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5363911Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5364413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.5364952Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5365454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5365941Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5366097Z 2025-08-14T21:53:07.5366207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5366580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5366924Z return mod(**inputs) 2025-08-14T21:53:07.5367356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5367804Z outputs = self.mobilebert( 2025-08-14T21:53:07.5368221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5368660Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5369128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5369565Z layer_outputs = layer_module( 2025-08-14T21:53:07.5370018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5370499Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5370961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5371476Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5372015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5372477Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5372625Z 2025-08-14T21:53:07.5372735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5373116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5373457Z return mod(**inputs) 2025-08-14T21:53:07.5373882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5374342Z outputs = self.mobilebert( 2025-08-14T21:53:07.5374789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5375222Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5375671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5376108Z layer_outputs = layer_module( 2025-08-14T21:53:07.5376527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5377013Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5377463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5377936Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5378408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5378899Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5379080Z 2025-08-14T21:53:07.5379195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5379568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5380942Z return mod(**inputs) 2025-08-14T21:53:07.5381348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5381792Z outputs = self.mobilebert( 2025-08-14T21:53:07.5382239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5382677Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5383098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5383529Z layer_outputs = layer_module( 2025-08-14T21:53:07.5383952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5384499Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5384973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5385490Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5385982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5386423Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5386581Z 2025-08-14T21:53:07.5386692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5387078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5387425Z return mod(**inputs) 2025-08-14T21:53:07.5387849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5388284Z outputs = self.mobilebert( 2025-08-14T21:53:07.5388707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5389137Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5389573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5390011Z layer_outputs = layer_module( 2025-08-14T21:53:07.5390434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5390894Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5391370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5391875Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5392383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5392881Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5393383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5393849Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5394010Z 2025-08-14T21:53:07.5394133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5394512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5394893Z return mod(**inputs) 2025-08-14T21:53:07.5395324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5395834Z outputs = self.mobilebert( 2025-08-14T21:53:07.5396285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5396774Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5397224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5397672Z layer_outputs = layer_module( 2025-08-14T21:53:07.5398109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5398581Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5399042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5399531Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5400019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5400483Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5400636Z 2025-08-14T21:53:07.5400751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5401162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5401525Z return mod(**inputs) 2025-08-14T21:53:07.5401943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5402378Z outputs = self.mobilebert( 2025-08-14T21:53:07.5402809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5403268Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5403723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5404178Z layer_outputs = layer_module( 2025-08-14T21:53:07.5404622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5405103Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5405574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5406067Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5406554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5407050Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5407230Z 2025-08-14T21:53:07.5407346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5407738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5408094Z return mod(**inputs) 2025-08-14T21:53:07.5408519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5409128Z outputs = self.mobilebert( 2025-08-14T21:53:07.5409566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5410096Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5410546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5411047Z layer_outputs = layer_module( 2025-08-14T21:53:07.5411483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5411936Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5412438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5412921Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5413410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5413862Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5414016Z 2025-08-14T21:53:07.5414128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5414516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5414869Z return mod(**inputs) 2025-08-14T21:53:07.5415284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5415729Z outputs = self.mobilebert( 2025-08-14T21:53:07.5416150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5416580Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5417036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5417471Z layer_outputs = layer_module( 2025-08-14T21:53:07.5417903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5418355Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5418805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5419392Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5419876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5420362Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5420841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5421300Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5421456Z 2025-08-14T21:53:07.5421574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5421943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5422298Z return mod(**inputs) 2025-08-14T21:53:07.5422717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5423155Z outputs = self.mobilebert( 2025-08-14T21:53:07.5423573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5424011Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5424446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5424879Z layer_outputs = layer_module( 2025-08-14T21:53:07.5425304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5425762Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5426212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5426705Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5427174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5427651Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5427797Z 2025-08-14T21:53:07.5427913Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5428285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5428629Z return mod(**inputs) 2025-08-14T21:53:07.5429034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5429460Z outputs = self.mobilebert( 2025-08-14T21:53:07.5429884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5430328Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5430779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5431223Z layer_outputs = layer_module( 2025-08-14T21:53:07.5431672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5432162Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5432647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5433140Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5433679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5434175Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5434352Z 2025-08-14T21:53:07.5434470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5434872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5435229Z return mod(**inputs) 2025-08-14T21:53:07.5435649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5436268Z outputs = self.mobilebert( 2025-08-14T21:53:07.5436706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5437165Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5437616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5438061Z layer_outputs = layer_module( 2025-08-14T21:53:07.5438499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5438981Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5439440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5439940Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5440444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5440903Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5441057Z 2025-08-14T21:53:07.5441170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5441556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5441934Z return mod(**inputs) 2025-08-14T21:53:07.5442356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5442798Z outputs = self.mobilebert( 2025-08-14T21:53:07.5443265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5443705Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5444134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5444570Z layer_outputs = layer_module( 2025-08-14T21:53:07.5445001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5445465Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5445922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5446418Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5446915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5447408Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5447913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5448385Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5448538Z 2025-08-14T21:53:07.5449657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5450034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5450365Z return mod(**inputs) 2025-08-14T21:53:07.5450771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5451219Z outputs = self.mobilebert( 2025-08-14T21:53:07.5451631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5452063Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5452491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5452924Z layer_outputs = layer_module( 2025-08-14T21:53:07.5453345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5453836Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5454333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5454804Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5454959Z 2025-08-14T21:53:07.5455075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5455482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5455839Z return mod(**inputs) 2025-08-14T21:53:07.5456264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5456720Z outputs = self.mobilebert( 2025-08-14T21:53:07.5457150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5457598Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5458029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5458484Z layer_outputs = layer_module( 2025-08-14T21:53:07.5458914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5459394Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5459890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5460373Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5460546Z 2025-08-14T21:53:07.5460668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5461042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5461384Z return mod(**inputs) 2025-08-14T21:53:07.5461791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5462228Z outputs = self.mobilebert( 2025-08-14T21:53:07.5462643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5463081Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5463512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5463969Z layer_outputs = layer_module( 2025-08-14T21:53:07.5464397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5464940Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5465476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.5465942Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.5466123Z 2025-08-14T21:53:07.5466241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5466654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5467005Z return mod(**inputs) 2025-08-14T21:53:07.5467415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5467860Z outputs = self.mobilebert( 2025-08-14T21:53:07.5468289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5468742Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5469165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5469611Z layer_outputs = layer_module( 2025-08-14T21:53:07.5470044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5470577Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5471111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.5471603Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.5472096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5472558Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5472726Z 2025-08-14T21:53:07.5472838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5473229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5473625Z return mod(**inputs) 2025-08-14T21:53:07.5474065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5474509Z outputs = self.mobilebert( 2025-08-14T21:53:07.5474958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5475422Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5476056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5476520Z layer_outputs = layer_module( 2025-08-14T21:53:07.5476969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5477503Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5478051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5478553Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5479061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.5479517Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5479681Z 2025-08-14T21:53:07.5479816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5480296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5480656Z return mod(**inputs) 2025-08-14T21:53:07.5481079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5481534Z outputs = self.mobilebert( 2025-08-14T21:53:07.5481973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5482459Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5482902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5483352Z layer_outputs = layer_module( 2025-08-14T21:53:07.5483792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5484321Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5484867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5485371Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5485869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.5486366Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5486862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5487331Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5487490Z 2025-08-14T21:53:07.5487616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5487996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5488351Z return mod(**inputs) 2025-08-14T21:53:07.5488781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5489229Z outputs = self.mobilebert( 2025-08-14T21:53:07.5489694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5490147Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5490591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5491055Z layer_outputs = layer_module( 2025-08-14T21:53:07.5491499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5492050Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5492593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5493076Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5493578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5494031Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5494181Z 2025-08-14T21:53:07.5494290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5494664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5495007Z return mod(**inputs) 2025-08-14T21:53:07.5495445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5495876Z outputs = self.mobilebert( 2025-08-14T21:53:07.5496316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5496789Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5497247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5497700Z layer_outputs = layer_module( 2025-08-14T21:53:07.5498151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5498601Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5499056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5499499Z self_outputs = self.self( 2025-08-14T21:53:07.5499922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.5500364Z self.value(value_tensor) 2025-08-14T21:53:07.5500485Z 2025-08-14T21:53:07.5500594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5500965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5501307Z return mod(**inputs) 2025-08-14T21:53:07.5501720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5502138Z outputs = self.mobilebert( 2025-08-14T21:53:07.5502554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5502986Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5503401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5503828Z layer_outputs = layer_module( 2025-08-14T21:53:07.5504247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5504768Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5505312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.5505792Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.5506278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5506727Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5506874Z 2025-08-14T21:53:07.5506986Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5507361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5507705Z return mod(**inputs) 2025-08-14T21:53:07.5508102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5508530Z outputs = self.mobilebert( 2025-08-14T21:53:07.5509133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5509579Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5509997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5510448Z layer_outputs = layer_module( 2025-08-14T21:53:07.5510957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5511485Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5512005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5512478Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5512947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.5513449Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.5513903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5514363Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5514527Z 2025-08-14T21:53:07.5514650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5515030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5515388Z return mod(**inputs) 2025-08-14T21:53:07.5515900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5516384Z outputs = self.mobilebert( 2025-08-14T21:53:07.5516809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5517255Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5517700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5518153Z layer_outputs = layer_module( 2025-08-14T21:53:07.5518591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5519070Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5519526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5519969Z self_outputs = self.self( 2025-08-14T21:53:07.5520412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.5520902Z self.query(query_tensor) 2025-08-14T21:53:07.5521025Z 2025-08-14T21:53:07.5521145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5521532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5521911Z return mod(**inputs) 2025-08-14T21:53:07.5522335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5522789Z outputs = self.mobilebert( 2025-08-14T21:53:07.5523224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5523687Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5524135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5524585Z layer_outputs = layer_module( 2025-08-14T21:53:07.5525036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5525503Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5525968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5526409Z self_outputs = self.self( 2025-08-14T21:53:07.5526900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.5527341Z self.key(key_tensor) 2025-08-14T21:53:07.5527456Z 2025-08-14T21:53:07.5527544Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5527788Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5528050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5528445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5528799Z return mod(**inputs) 2025-08-14T21:53:07.5529238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5529675Z outputs = self.mobilebert( 2025-08-14T21:53:07.5530087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5530527Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5530953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5531381Z layer_outputs = layer_module( 2025-08-14T21:53:07.5531798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5532246Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5532687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5533170Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5533651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.5534102Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5534249Z 2025-08-14T21:53:07.5534364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5534727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5535064Z return mod(**inputs) 2025-08-14T21:53:07.5535464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5535913Z outputs = self.mobilebert( 2025-08-14T21:53:07.5536329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5536776Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5537202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5537654Z layer_outputs = layer_module( 2025-08-14T21:53:07.5538081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5538526Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5538970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5539457Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5539948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.5540444Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5540932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5541385Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5541548Z 2025-08-14T21:53:07.5541675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5542056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5542399Z return mod(**inputs) 2025-08-14T21:53:07.5542812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5543307Z outputs = self.mobilebert( 2025-08-14T21:53:07.5543725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5544161Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5544606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5545046Z layer_outputs = layer_module( 2025-08-14T21:53:07.5545473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5545920Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5546384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5546855Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5547328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5547769Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5547925Z 2025-08-14T21:53:07.5548035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5548412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5548747Z return mod(**inputs) 2025-08-14T21:53:07.5549165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5549606Z outputs = self.mobilebert( 2025-08-14T21:53:07.5550040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5550473Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5550918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5551369Z layer_outputs = layer_module( 2025-08-14T21:53:07.5551802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5552260Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5552740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5553224Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5553697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5554181Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5554364Z 2025-08-14T21:53:07.5554476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5554863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5555206Z return mod(**inputs) 2025-08-14T21:53:07.5555623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5556152Z outputs = self.mobilebert( 2025-08-14T21:53:07.5556587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5557022Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5557485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5557937Z layer_outputs = layer_module( 2025-08-14T21:53:07.5558374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5558850Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5559333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5559867Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5560369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5560828Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5560981Z 2025-08-14T21:53:07.5561104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5561489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5561832Z return mod(**inputs) 2025-08-14T21:53:07.5562264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5562709Z outputs = self.mobilebert( 2025-08-14T21:53:07.5563124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5563539Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5563939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5564345Z layer_outputs = layer_module( 2025-08-14T21:53:07.5564742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5565192Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5565634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5566131Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5566642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5567129Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5567609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5568049Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5568205Z 2025-08-14T21:53:07.5568309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5568675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5568999Z return mod(**inputs) 2025-08-14T21:53:07.5569373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5569783Z outputs = self.mobilebert( 2025-08-14T21:53:07.5570177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5570584Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5570983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5571389Z layer_outputs = layer_module( 2025-08-14T21:53:07.5571801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5572221Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5572648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5573094Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5573533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5573947Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5574092Z 2025-08-14T21:53:07.5574211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5574567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5574887Z return mod(**inputs) 2025-08-14T21:53:07.5575261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5575669Z outputs = self.mobilebert( 2025-08-14T21:53:07.5576061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5576463Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5576868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5577278Z layer_outputs = layer_module( 2025-08-14T21:53:07.5577686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5578107Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5578536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5578987Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5579431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5579879Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5580050Z 2025-08-14T21:53:07.5580152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5580508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5580849Z return mod(**inputs) 2025-08-14T21:53:07.5581260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5581688Z outputs = self.mobilebert( 2025-08-14T21:53:07.5582140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5582533Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5582927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5583324Z layer_outputs = layer_module( 2025-08-14T21:53:07.5583717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5584150Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5584578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5585036Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5585482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5585904Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5586049Z 2025-08-14T21:53:07.5586180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5586534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5586845Z return mod(**inputs) 2025-08-14T21:53:07.5587253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5587685Z outputs = self.mobilebert( 2025-08-14T21:53:07.5588106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5588565Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5588997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5589407Z layer_outputs = layer_module( 2025-08-14T21:53:07.5589802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5590235Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5590683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5591174Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5591658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5592148Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5592638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5593099Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5593251Z 2025-08-14T21:53:07.5593360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5593736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5594092Z return mod(**inputs) 2025-08-14T21:53:07.5594502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5594940Z outputs = self.mobilebert( 2025-08-14T21:53:07.5595393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5595922Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5596364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5596828Z layer_outputs = layer_module( 2025-08-14T21:53:07.5597261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5597728Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5598182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5598662Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5599145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5599589Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5599743Z 2025-08-14T21:53:07.5599854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5600229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5600572Z return mod(**inputs) 2025-08-14T21:53:07.5601001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5601437Z outputs = self.mobilebert( 2025-08-14T21:53:07.5601858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5602284Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5602721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5603162Z layer_outputs = layer_module( 2025-08-14T21:53:07.5603614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5604059Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5604508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5604989Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5605464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5605926Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5606108Z 2025-08-14T21:53:07.5606218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5606592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5606936Z return mod(**inputs) 2025-08-14T21:53:07.5607348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5607785Z outputs = self.mobilebert( 2025-08-14T21:53:07.5608202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5608610Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5609156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5609574Z layer_outputs = layer_module( 2025-08-14T21:53:07.5609982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5610407Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5610945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5611405Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5611860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5612313Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5612462Z 2025-08-14T21:53:07.5612568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5612936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5613260Z return mod(**inputs) 2025-08-14T21:53:07.5613654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5614069Z outputs = self.mobilebert( 2025-08-14T21:53:07.5614475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5614891Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5615304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5615723Z layer_outputs = layer_module( 2025-08-14T21:53:07.5616154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5616589Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5617018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5617477Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5617929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5618389Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5618872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5619309Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5619461Z 2025-08-14T21:53:07.5619566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5619933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5620256Z return mod(**inputs) 2025-08-14T21:53:07.5620634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5621042Z outputs = self.mobilebert( 2025-08-14T21:53:07.5621436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5621845Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5622244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5622652Z layer_outputs = layer_module( 2025-08-14T21:53:07.5623051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5623505Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5623951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5624378Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5624511Z 2025-08-14T21:53:07.5624616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5624970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5625267Z return mod(**inputs) 2025-08-14T21:53:07.5625634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5626049Z outputs = self.mobilebert( 2025-08-14T21:53:07.5626426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5626880Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5627285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5627692Z layer_outputs = layer_module( 2025-08-14T21:53:07.5628081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5628537Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5628989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5629429Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5629600Z 2025-08-14T21:53:07.5629704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5630063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5630422Z return mod(**inputs) 2025-08-14T21:53:07.5630822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5631232Z outputs = self.mobilebert( 2025-08-14T21:53:07.5631627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5632035Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5632443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5632894Z layer_outputs = layer_module( 2025-08-14T21:53:07.5633321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5633829Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5634353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.5634804Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.5634961Z 2025-08-14T21:53:07.5635077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5635445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5635852Z return mod(**inputs) 2025-08-14T21:53:07.5636276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5636720Z outputs = self.mobilebert( 2025-08-14T21:53:07.5637147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5637608Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5638019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5638431Z layer_outputs = layer_module( 2025-08-14T21:53:07.5638827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5639310Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5639833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.5640287Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.5640743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5641204Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5641349Z 2025-08-14T21:53:07.5641460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5641808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5642139Z return mod(**inputs) 2025-08-14T21:53:07.5642522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5642922Z outputs = self.mobilebert( 2025-08-14T21:53:07.5643315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5643723Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5644130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5644527Z layer_outputs = layer_module( 2025-08-14T21:53:07.5644943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5645435Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5645928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5646377Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5646842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.5647313Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5647465Z 2025-08-14T21:53:07.5647580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5647946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5648289Z return mod(**inputs) 2025-08-14T21:53:07.5648675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5649077Z outputs = self.mobilebert( 2025-08-14T21:53:07.5649473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5649882Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5650282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5650680Z layer_outputs = layer_module( 2025-08-14T21:53:07.5651081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5651572Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5652067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5652524Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5652981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.5653438Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5653911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5654346Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5654504Z 2025-08-14T21:53:07.5654610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5654987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5655308Z return mod(**inputs) 2025-08-14T21:53:07.5655702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5656120Z outputs = self.mobilebert( 2025-08-14T21:53:07.5656525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5656935Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5657344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5657764Z layer_outputs = layer_module( 2025-08-14T21:53:07.5658168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5658675Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5659197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5659643Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5660075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5660497Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5660643Z 2025-08-14T21:53:07.5660747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5661102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5661416Z return mod(**inputs) 2025-08-14T21:53:07.5661819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5662232Z outputs = self.mobilebert( 2025-08-14T21:53:07.5662633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5663062Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5663466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5663883Z layer_outputs = layer_module( 2025-08-14T21:53:07.5664285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5664721Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5665155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5665571Z self_outputs = self.self( 2025-08-14T21:53:07.5665969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.5666384Z self.value(value_tensor) 2025-08-14T21:53:07.5666501Z 2025-08-14T21:53:07.5666612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5666974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5667310Z return mod(**inputs) 2025-08-14T21:53:07.5667695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5668116Z outputs = self.mobilebert( 2025-08-14T21:53:07.5668492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5668901Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5669307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5669722Z layer_outputs = layer_module( 2025-08-14T21:53:07.5670123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5670616Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5671138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.5671621Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.5672115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5672581Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5672731Z 2025-08-14T21:53:07.5672860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5673234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5673598Z return mod(**inputs) 2025-08-14T21:53:07.5674046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5674504Z outputs = self.mobilebert( 2025-08-14T21:53:07.5674941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5675381Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5675893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5676340Z layer_outputs = layer_module( 2025-08-14T21:53:07.5676803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5677405Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5677943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5678394Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5678875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.5679338Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.5679798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5680258Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5680420Z 2025-08-14T21:53:07.5680532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5680916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5681264Z return mod(**inputs) 2025-08-14T21:53:07.5681687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5682118Z outputs = self.mobilebert( 2025-08-14T21:53:07.5682542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5683026Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5683467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5683936Z layer_outputs = layer_module( 2025-08-14T21:53:07.5684357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5684835Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5685291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5685737Z self_outputs = self.self( 2025-08-14T21:53:07.5686153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.5686602Z self.query(query_tensor) 2025-08-14T21:53:07.5686732Z 2025-08-14T21:53:07.5686842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5687221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5687559Z return mod(**inputs) 2025-08-14T21:53:07.5687973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5688413Z outputs = self.mobilebert( 2025-08-14T21:53:07.5688831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5689266Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5689726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5690157Z layer_outputs = layer_module( 2025-08-14T21:53:07.5690573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5691031Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5691477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5691915Z self_outputs = self.self( 2025-08-14T21:53:07.5692339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.5692776Z self.key(key_tensor) 2025-08-14T21:53:07.5692886Z 2025-08-14T21:53:07.5692982Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5693208Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5693458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5693836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5694177Z return mod(**inputs) 2025-08-14T21:53:07.5694577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5695017Z outputs = self.mobilebert( 2025-08-14T21:53:07.5695439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5695869Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5696307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5696743Z layer_outputs = layer_module( 2025-08-14T21:53:07.5697173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5697619Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5698062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5698546Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5699063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.5699504Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5699661Z 2025-08-14T21:53:07.5699770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5700165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5700498Z return mod(**inputs) 2025-08-14T21:53:07.5700904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5701337Z outputs = self.mobilebert( 2025-08-14T21:53:07.5701768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5702206Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5702650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5703082Z layer_outputs = layer_module( 2025-08-14T21:53:07.5703505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5703957Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5704428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5704925Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5705415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.5705921Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5706426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5706902Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5707064Z 2025-08-14T21:53:07.5707199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5707595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5707945Z return mod(**inputs) 2025-08-14T21:53:07.5708355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5709020Z outputs = self.mobilebert( 2025-08-14T21:53:07.5709458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5709904Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5710344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5710794Z layer_outputs = layer_module( 2025-08-14T21:53:07.5711239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5711727Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5712191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5712684Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5713172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5713635Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5713790Z 2025-08-14T21:53:07.5713903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5714348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5714696Z return mod(**inputs) 2025-08-14T21:53:07.5715107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5715586Z outputs = self.mobilebert( 2025-08-14T21:53:07.5716073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5716521Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5716955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5717400Z layer_outputs = layer_module( 2025-08-14T21:53:07.5717841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5718308Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5718776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5719265Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5719726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5720176Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5720384Z 2025-08-14T21:53:07.5720488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5720844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5721155Z return mod(**inputs) 2025-08-14T21:53:07.5721519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5721920Z outputs = self.mobilebert( 2025-08-14T21:53:07.5722316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5722741Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5723146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5723556Z layer_outputs = layer_module( 2025-08-14T21:53:07.5723964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5724388Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5724825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5725293Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5725752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5726174Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5726329Z 2025-08-14T21:53:07.5726429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5726776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5727084Z return mod(**inputs) 2025-08-14T21:53:07.5727462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5727856Z outputs = self.mobilebert( 2025-08-14T21:53:07.5728241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5728639Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5729074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5729486Z layer_outputs = layer_module( 2025-08-14T21:53:07.5729875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5730304Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5730726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5731182Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5731629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5732119Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5732601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5733034Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5733180Z 2025-08-14T21:53:07.5733286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5733490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5733558Z return mod(**inputs) 2025-08-14T21:53:07.5733863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5733937Z outputs = self.mobilebert( 2025-08-14T21:53:07.5734212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5734293Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5734572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5734643Z layer_outputs = layer_module( 2025-08-14T21:53:07.5734930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5735023Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5735294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5735403Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5735679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5735775Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5735779Z 2025-08-14T21:53:07.5735881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5736088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5736155Z return mod(**inputs) 2025-08-14T21:53:07.5736435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5736515Z outputs = self.mobilebert( 2025-08-14T21:53:07.5736798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5736881Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5737158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5737229Z layer_outputs = layer_module( 2025-08-14T21:53:07.5737517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5737634Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5737910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5738030Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5738305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5738441Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5738445Z 2025-08-14T21:53:07.5738548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5738740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5738813Z return mod(**inputs) 2025-08-14T21:53:07.5739082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5739160Z outputs = self.mobilebert( 2025-08-14T21:53:07.5739428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5739501Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5739779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5739852Z layer_outputs = layer_module( 2025-08-14T21:53:07.5740153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5740261Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5740534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5740664Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5740944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5741031Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5741061Z 2025-08-14T21:53:07.5741173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5741366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5741439Z return mod(**inputs) 2025-08-14T21:53:07.5741710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5741781Z outputs = self.mobilebert( 2025-08-14T21:53:07.5742063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5742138Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5742411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5742491Z layer_outputs = layer_module( 2025-08-14T21:53:07.5742763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5742864Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5743134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5743257Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5743535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5743654Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5743938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5744047Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5744051Z 2025-08-14T21:53:07.5744153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5744352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5744467Z return mod(**inputs) 2025-08-14T21:53:07.5744748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5744828Z outputs = self.mobilebert( 2025-08-14T21:53:07.5745122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5745206Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5745511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5745586Z layer_outputs = layer_module( 2025-08-14T21:53:07.5745894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5745991Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5746305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5746438Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5746715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5746805Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5746809Z 2025-08-14T21:53:07.5746908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5747110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5747177Z return mod(**inputs) 2025-08-14T21:53:07.5747469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5747551Z outputs = self.mobilebert( 2025-08-14T21:53:07.5747826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5747898Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5748182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5748252Z layer_outputs = layer_module( 2025-08-14T21:53:07.5748533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5748629Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5748905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5749024Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5749297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5749418Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5749421Z 2025-08-14T21:53:07.5749523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5749717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5749788Z return mod(**inputs) 2025-08-14T21:53:07.5750065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5750153Z outputs = self.mobilebert( 2025-08-14T21:53:07.5750436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5750509Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5750801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5750898Z layer_outputs = layer_module( 2025-08-14T21:53:07.5751190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5751296Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5751586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5751721Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5752012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5752102Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5752108Z 2025-08-14T21:53:07.5752222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5752429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5752499Z return mod(**inputs) 2025-08-14T21:53:07.5752827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5752904Z outputs = self.mobilebert( 2025-08-14T21:53:07.5753201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5753277Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5753581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5753666Z layer_outputs = layer_module( 2025-08-14T21:53:07.5753978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5754087Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5754375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5754504Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5754800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5754924Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5755211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5755317Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5755321Z 2025-08-14T21:53:07.5755427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5755641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5755792Z return mod(**inputs) 2025-08-14T21:53:07.5756094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5756182Z outputs = self.mobilebert( 2025-08-14T21:53:07.5756470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5756557Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5756854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5756959Z layer_outputs = layer_module( 2025-08-14T21:53:07.5757261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5757389Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5757700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5757800Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5757804Z 2025-08-14T21:53:07.5757910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5758124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5758194Z return mod(**inputs) 2025-08-14T21:53:07.5758489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5758574Z outputs = self.mobilebert( 2025-08-14T21:53:07.5758863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5758946Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5759235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5759308Z layer_outputs = layer_module( 2025-08-14T21:53:07.5759619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5759746Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5760030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5760158Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5760161Z 2025-08-14T21:53:07.5760267Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5760496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5760567Z return mod(**inputs) 2025-08-14T21:53:07.5760857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5760939Z outputs = self.mobilebert( 2025-08-14T21:53:07.5761227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5761308Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5761595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5761670Z layer_outputs = layer_module( 2025-08-14T21:53:07.5761965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5762134Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5762422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.5762538Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.5762542Z 2025-08-14T21:53:07.5762642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5762837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5762899Z return mod(**inputs) 2025-08-14T21:53:07.5763165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5763263Z outputs = self.mobilebert( 2025-08-14T21:53:07.5763540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5763619Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5763897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5763986Z layer_outputs = layer_module( 2025-08-14T21:53:07.5764271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5764425Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5764710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.5764829Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.5765106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5765203Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5765208Z 2025-08-14T21:53:07.5765308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5765505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5765582Z return mod(**inputs) 2025-08-14T21:53:07.5765882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5765962Z outputs = self.mobilebert( 2025-08-14T21:53:07.5766237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5766309Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5766591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5766661Z layer_outputs = layer_module( 2025-08-14T21:53:07.5766959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5767125Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5767394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5767520Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5767784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.5767866Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5767877Z 2025-08-14T21:53:07.5767976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5768163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5768234Z return mod(**inputs) 2025-08-14T21:53:07.5768499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5768569Z outputs = self.mobilebert( 2025-08-14T21:53:07.5768844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5768914Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5769184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5769252Z layer_outputs = layer_module( 2025-08-14T21:53:07.5769516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5769696Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5769963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5770101Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5770373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.5770490Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5770763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5770851Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5770855Z 2025-08-14T21:53:07.5770952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5771149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5771211Z return mod(**inputs) 2025-08-14T21:53:07.5771484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5771555Z outputs = self.mobilebert( 2025-08-14T21:53:07.5771837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5771916Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5772180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5772254Z layer_outputs = layer_module( 2025-08-14T21:53:07.5772518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5772673Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5773007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5773119Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5773383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5773471Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5773474Z 2025-08-14T21:53:07.5773571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5773768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5773833Z return mod(**inputs) 2025-08-14T21:53:07.5774104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5774183Z outputs = self.mobilebert( 2025-08-14T21:53:07.5774455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5774538Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5774812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5774882Z layer_outputs = layer_module( 2025-08-14T21:53:07.5775160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5775244Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5775516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5775618Z self_outputs = self.self( 2025-08-14T21:53:07.5775893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.5775973Z self.value(value_tensor) 2025-08-14T21:53:07.5775977Z 2025-08-14T21:53:07.5776079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5776291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5776365Z return mod(**inputs) 2025-08-14T21:53:07.5776641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5776719Z outputs = self.mobilebert( 2025-08-14T21:53:07.5777000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5777070Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5777344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5777411Z layer_outputs = layer_module( 2025-08-14T21:53:07.5777678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5777839Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5778125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.5778243Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.5778513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5778595Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5778601Z 2025-08-14T21:53:07.5778708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5778905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5778991Z return mod(**inputs) 2025-08-14T21:53:07.5779259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5779330Z outputs = self.mobilebert( 2025-08-14T21:53:07.5779607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5779678Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5779949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5780027Z layer_outputs = layer_module( 2025-08-14T21:53:07.5780298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5780460Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5780734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5780844Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5781123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.5781207Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.5781482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5781572Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5781576Z 2025-08-14T21:53:07.5781676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5781909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5781973Z return mod(**inputs) 2025-08-14T21:53:07.5782249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5782346Z outputs = self.mobilebert( 2025-08-14T21:53:07.5782629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5782710Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5782994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5783065Z layer_outputs = layer_module( 2025-08-14T21:53:07.5783355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5783441Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5783731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5783802Z self_outputs = self.self( 2025-08-14T21:53:07.5784083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.5784164Z self.query(query_tensor) 2025-08-14T21:53:07.5784167Z 2025-08-14T21:53:07.5784286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5784482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5784554Z return mod(**inputs) 2025-08-14T21:53:07.5784825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5784906Z outputs = self.mobilebert( 2025-08-14T21:53:07.5785183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5785272Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5785552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5785623Z layer_outputs = layer_module( 2025-08-14T21:53:07.5785902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5785985Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5786256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5786333Z self_outputs = self.self( 2025-08-14T21:53:07.5786606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.5786673Z self.key(key_tensor) 2025-08-14T21:53:07.5786684Z 2025-08-14T21:53:07.5786766Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5786846Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5786957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5787152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5787217Z return mod(**inputs) 2025-08-14T21:53:07.5787504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5787575Z outputs = self.mobilebert( 2025-08-14T21:53:07.5787845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5787924Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5788222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5788302Z layer_outputs = layer_module( 2025-08-14T21:53:07.5788581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5788692Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5788972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5789094Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5789373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.5789456Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5789461Z 2025-08-14T21:53:07.5789562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5789763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5789828Z return mod(**inputs) 2025-08-14T21:53:07.5790097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5790176Z outputs = self.mobilebert( 2025-08-14T21:53:07.5790461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5790542Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5790813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5790885Z layer_outputs = layer_module( 2025-08-14T21:53:07.5791165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5791249Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5791546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5791679Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5791964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.5792103Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5792391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5792496Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5792500Z 2025-08-14T21:53:07.5792607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5792810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5792885Z return mod(**inputs) 2025-08-14T21:53:07.5793172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5793249Z outputs = self.mobilebert( 2025-08-14T21:53:07.5793540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5793617Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5793908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5793982Z layer_outputs = layer_module( 2025-08-14T21:53:07.5794268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5794391Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5794682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5794805Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5795109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5795197Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5795202Z 2025-08-14T21:53:07.5795316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5795522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5795590Z return mod(**inputs) 2025-08-14T21:53:07.5795964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5796052Z outputs = self.mobilebert( 2025-08-14T21:53:07.5796360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5796439Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5796737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5796823Z layer_outputs = layer_module( 2025-08-14T21:53:07.5797143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5797254Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5797550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5797670Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5797977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5798114Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5798118Z 2025-08-14T21:53:07.5798230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5798448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5798522Z return mod(**inputs) 2025-08-14T21:53:07.5798829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5798912Z outputs = self.mobilebert( 2025-08-14T21:53:07.5799206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5799294Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5799592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5799679Z layer_outputs = layer_module( 2025-08-14T21:53:07.5799975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5800078Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5800381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5800517Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5800811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5800911Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5800914Z 2025-08-14T21:53:07.5801039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5801250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5801319Z return mod(**inputs) 2025-08-14T21:53:07.5801605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5801713Z outputs = self.mobilebert( 2025-08-14T21:53:07.5802003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5802085Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5802374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5802449Z layer_outputs = layer_module( 2025-08-14T21:53:07.5802747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5802846Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5803136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5803274Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5803563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5803715Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5804004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5804101Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5804105Z 2025-08-14T21:53:07.5804219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5804425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5804499Z return mod(**inputs) 2025-08-14T21:53:07.5804803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5804880Z outputs = self.mobilebert( 2025-08-14T21:53:07.5805181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5805258Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5805551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5805631Z layer_outputs = layer_module( 2025-08-14T21:53:07.5805924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5806030Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5806324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5806439Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5806737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5806824Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5806830Z 2025-08-14T21:53:07.5806943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5807149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5807219Z return mod(**inputs) 2025-08-14T21:53:07.5807518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5807611Z outputs = self.mobilebert( 2025-08-14T21:53:07.5807901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5807986Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5808278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5808381Z layer_outputs = layer_module( 2025-08-14T21:53:07.5808793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5808902Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5809206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5809324Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5809624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5809744Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5809748Z 2025-08-14T21:53:07.5809856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5810070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5810141Z return mod(**inputs) 2025-08-14T21:53:07.5810477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5810555Z outputs = self.mobilebert( 2025-08-14T21:53:07.5810844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5810926Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5811218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5811291Z layer_outputs = layer_module( 2025-08-14T21:53:07.5811616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5811716Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5812010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5812139Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5812425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5812521Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5812525Z 2025-08-14T21:53:07.5812634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5812846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5812915Z return mod(**inputs) 2025-08-14T21:53:07.5813204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5813284Z outputs = self.mobilebert( 2025-08-14T21:53:07.5813558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5813631Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5813911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5813982Z layer_outputs = layer_module( 2025-08-14T21:53:07.5814263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5814392Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5814670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5814799Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5815101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5815228Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5815501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5815593Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5815597Z 2025-08-14T21:53:07.5815706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5815903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5815968Z return mod(**inputs) 2025-08-14T21:53:07.5816258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5816334Z outputs = self.mobilebert( 2025-08-14T21:53:07.5816626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5816720Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5817016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5817099Z layer_outputs = layer_module( 2025-08-14T21:53:07.5817394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5817509Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5817788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5817915Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5818203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5818287Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5818292Z 2025-08-14T21:53:07.5818400Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5818595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5818661Z return mod(**inputs) 2025-08-14T21:53:07.5818945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5819018Z outputs = self.mobilebert( 2025-08-14T21:53:07.5819293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5819380Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5819668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5819752Z layer_outputs = layer_module( 2025-08-14T21:53:07.5820041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5820138Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5820435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5820551Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5820860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5820984Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5820989Z 2025-08-14T21:53:07.5821095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5821338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5821402Z return mod(**inputs) 2025-08-14T21:53:07.5821675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5821752Z outputs = self.mobilebert( 2025-08-14T21:53:07.5822024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5822103Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5822376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5822446Z layer_outputs = layer_module( 2025-08-14T21:53:07.5822728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5822821Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5823116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5823254Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5823541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5823644Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5823647Z 2025-08-14T21:53:07.5823748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5823944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5824019Z return mod(**inputs) 2025-08-14T21:53:07.5824312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5824391Z outputs = self.mobilebert( 2025-08-14T21:53:07.5824665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5824738Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5825016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5825086Z layer_outputs = layer_module( 2025-08-14T21:53:07.5825357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5825458Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5825748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5825883Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5826176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5826305Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5826600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5826696Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5826699Z 2025-08-14T21:53:07.5826815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5827040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5827110Z return mod(**inputs) 2025-08-14T21:53:07.5827405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5827500Z outputs = self.mobilebert( 2025-08-14T21:53:07.5827806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5827883Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5828179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5828258Z layer_outputs = layer_module( 2025-08-14T21:53:07.5828552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5828679Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5828984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5829071Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5829075Z 2025-08-14T21:53:07.5829189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5829397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5829482Z return mod(**inputs) 2025-08-14T21:53:07.5829780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5829852Z outputs = self.mobilebert( 2025-08-14T21:53:07.5830154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5830232Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5830536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5830648Z layer_outputs = layer_module( 2025-08-14T21:53:07.5830941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5831066Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5831363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5831480Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5831484Z 2025-08-14T21:53:07.5831596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5831804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5831875Z return mod(**inputs) 2025-08-14T21:53:07.5832180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5832257Z outputs = self.mobilebert( 2025-08-14T21:53:07.5832565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5832646Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5832947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5833032Z layer_outputs = layer_module( 2025-08-14T21:53:07.5833333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5833504Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5834541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.5834647Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.5834653Z 2025-08-14T21:53:07.5834771Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5835019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5835091Z return mod(**inputs) 2025-08-14T21:53:07.5835411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5835488Z outputs = self.mobilebert( 2025-08-14T21:53:07.5835870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5835954Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5836269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5836358Z layer_outputs = layer_module( 2025-08-14T21:53:07.5836669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5836852Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5837199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.5837343Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.5837660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5837760Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5837764Z 2025-08-14T21:53:07.5837875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5838116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5838186Z return mod(**inputs) 2025-08-14T21:53:07.5838637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5838731Z outputs = self.mobilebert( 2025-08-14T21:53:07.5839063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5839148Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5839460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5839540Z layer_outputs = layer_module( 2025-08-14T21:53:07.5839853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5840019Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5840334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5840465Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5840783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.5840880Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5840884Z 2025-08-14T21:53:07.5840992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5841225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5841293Z return mod(**inputs) 2025-08-14T21:53:07.5841586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5841697Z outputs = self.mobilebert( 2025-08-14T21:53:07.5842013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5842121Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5842433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5842509Z layer_outputs = layer_module( 2025-08-14T21:53:07.5842821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5843001Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5843299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5843430Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5843705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.5843829Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5844106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5844219Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5844224Z 2025-08-14T21:53:07.5844334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5844525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5844599Z return mod(**inputs) 2025-08-14T21:53:07.5844871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5844941Z outputs = self.mobilebert( 2025-08-14T21:53:07.5845234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5845309Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5845583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5845663Z layer_outputs = layer_module( 2025-08-14T21:53:07.5845959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5846130Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5846432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5846549Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5846857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5846945Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5846951Z 2025-08-14T21:53:07.5847065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5847284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5847355Z return mod(**inputs) 2025-08-14T21:53:07.5847658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5847728Z outputs = self.mobilebert( 2025-08-14T21:53:07.5848009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5848135Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5848415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5848493Z layer_outputs = layer_module( 2025-08-14T21:53:07.5848779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5848881Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5849167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5849238Z self_outputs = self.self( 2025-08-14T21:53:07.5849523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.5849594Z self.value(value_tensor) 2025-08-14T21:53:07.5849599Z 2025-08-14T21:53:07.5849700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5849901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5849967Z return mod(**inputs) 2025-08-14T21:53:07.5850250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5850322Z outputs = self.mobilebert( 2025-08-14T21:53:07.5850611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5850693Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5850966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5851037Z layer_outputs = layer_module( 2025-08-14T21:53:07.5851319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5851475Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5851777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.5851890Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.5852169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5852259Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5852263Z 2025-08-14T21:53:07.5852365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5852568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5852633Z return mod(**inputs) 2025-08-14T21:53:07.5852908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5852988Z outputs = self.mobilebert( 2025-08-14T21:53:07.5853268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5853342Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5853624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5853696Z layer_outputs = layer_module( 2025-08-14T21:53:07.5853976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5854131Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5854407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5854547Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5854825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.5854934Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.5855212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5855304Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5855307Z 2025-08-14T21:53:07.5855413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5855602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5855665Z return mod(**inputs) 2025-08-14T21:53:07.5855937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5856008Z outputs = self.mobilebert( 2025-08-14T21:53:07.5856296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5856367Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5856645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5856724Z layer_outputs = layer_module( 2025-08-14T21:53:07.5857014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5857109Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5857384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5857458Z self_outputs = self.self( 2025-08-14T21:53:07.5857743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.5857830Z self.query(query_tensor) 2025-08-14T21:53:07.5857834Z 2025-08-14T21:53:07.5857936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5858138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5858204Z return mod(**inputs) 2025-08-14T21:53:07.5858487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5858558Z outputs = self.mobilebert( 2025-08-14T21:53:07.5858831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5858911Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5859185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5859266Z layer_outputs = layer_module( 2025-08-14T21:53:07.5859544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5859628Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5859910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5859980Z self_outputs = self.self( 2025-08-14T21:53:07.5860253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.5860328Z self.key(key_tensor) 2025-08-14T21:53:07.5860331Z 2025-08-14T21:53:07.5860413Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5860519Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5860619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5860814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5860889Z return mod(**inputs) 2025-08-14T21:53:07.5861176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5861270Z outputs = self.mobilebert( 2025-08-14T21:53:07.5861572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5861648Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5861944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5862017Z layer_outputs = layer_module( 2025-08-14T21:53:07.5862318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5862413Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5862707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5862842Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5863155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.5863247Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5863251Z 2025-08-14T21:53:07.5863369Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5863577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5863647Z return mod(**inputs) 2025-08-14T21:53:07.5863959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5864039Z outputs = self.mobilebert( 2025-08-14T21:53:07.5864364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5864442Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5864734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5864816Z layer_outputs = layer_module( 2025-08-14T21:53:07.5865115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5865207Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5865498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5865625Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5865917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.5866050Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5866337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5866445Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5866449Z 2025-08-14T21:53:07.5866555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5866763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5866832Z return mod(**inputs) 2025-08-14T21:53:07.5867117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5867221Z outputs = self.mobilebert( 2025-08-14T21:53:07.5867511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5867594Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5867909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5867983Z layer_outputs = layer_module( 2025-08-14T21:53:07.5868286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5868385Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5868678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5868805Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5869098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5869203Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5869206Z 2025-08-14T21:53:07.5869309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5869503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5869581Z return mod(**inputs) 2025-08-14T21:53:07.5869894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5869976Z outputs = self.mobilebert( 2025-08-14T21:53:07.5870255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5870331Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5870632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5870707Z layer_outputs = layer_module( 2025-08-14T21:53:07.5871016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5871125Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5871413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5871538Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5871824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5871942Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5871947Z 2025-08-14T21:53:07.5872060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5872266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5872345Z return mod(**inputs) 2025-08-14T21:53:07.5872632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5872707Z outputs = self.mobilebert( 2025-08-14T21:53:07.5873004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5873080Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5873373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5873447Z layer_outputs = layer_module( 2025-08-14T21:53:07.5873736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5873865Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5874158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5874316Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5874620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5874724Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5874727Z 2025-08-14T21:53:07.5874841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5875049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5875117Z return mod(**inputs) 2025-08-14T21:53:07.5875421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5875500Z outputs = self.mobilebert( 2025-08-14T21:53:07.5876066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5876154Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5876454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5876562Z layer_outputs = layer_module( 2025-08-14T21:53:07.5876867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5876968Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5877276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5877413Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5877740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5877872Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5878174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5878286Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5878291Z 2025-08-14T21:53:07.5878401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5878620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5878693Z return mod(**inputs) 2025-08-14T21:53:07.5879003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5879090Z outputs = self.mobilebert( 2025-08-14T21:53:07.5879393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5879471Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5879778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5879856Z layer_outputs = layer_module( 2025-08-14T21:53:07.5880171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5880270Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5880569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5880694Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5881019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5881118Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5881121Z 2025-08-14T21:53:07.5881230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5881463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5881542Z return mod(**inputs) 2025-08-14T21:53:07.5881852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5881926Z outputs = self.mobilebert( 2025-08-14T21:53:07.5882234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5882310Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5882616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5882691Z layer_outputs = layer_module( 2025-08-14T21:53:07.5882991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5883100Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5883415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5883546Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5883848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5883971Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5883977Z 2025-08-14T21:53:07.5884098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5884313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5884413Z return mod(**inputs) 2025-08-14T21:53:07.5884714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5884792Z outputs = self.mobilebert( 2025-08-14T21:53:07.5885096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5885175Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5885469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5885552Z layer_outputs = layer_module( 2025-08-14T21:53:07.5885912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5886021Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5886311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5886442Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5886740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5886829Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5886834Z 2025-08-14T21:53:07.5886950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5887154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5887223Z return mod(**inputs) 2025-08-14T21:53:07.5887526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5887630Z outputs = self.mobilebert( 2025-08-14T21:53:07.5887928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5888042Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5888337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5888419Z layer_outputs = layer_module( 2025-08-14T21:53:07.5888718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5888817Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5889120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5889251Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5889551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5889677Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5889974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5890099Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5890103Z 2025-08-14T21:53:07.5890215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5890425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5890505Z return mod(**inputs) 2025-08-14T21:53:07.5890799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5890884Z outputs = self.mobilebert( 2025-08-14T21:53:07.5891195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5891274Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5891574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5891647Z layer_outputs = layer_module( 2025-08-14T21:53:07.5891943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5892041Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5892330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5892457Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5892747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5892834Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5892846Z 2025-08-14T21:53:07.5892955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5893163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5893240Z return mod(**inputs) 2025-08-14T21:53:07.5893529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5893604Z outputs = self.mobilebert( 2025-08-14T21:53:07.5893894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5893969Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5894290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5894364Z layer_outputs = layer_module( 2025-08-14T21:53:07.5894654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5894780Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5895068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5895183Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5895473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5895587Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5895591Z 2025-08-14T21:53:07.5895705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5895910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5895979Z return mod(**inputs) 2025-08-14T21:53:07.5896277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5896351Z outputs = self.mobilebert( 2025-08-14T21:53:07.5896658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5896736Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5897030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5897110Z layer_outputs = layer_module( 2025-08-14T21:53:07.5897395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5897494Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5897810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5897942Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5898234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5898322Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5898325Z 2025-08-14T21:53:07.5898430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5898640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5898706Z return mod(**inputs) 2025-08-14T21:53:07.5899007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5899082Z outputs = self.mobilebert( 2025-08-14T21:53:07.5899367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5899449Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5899736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5899810Z layer_outputs = layer_module( 2025-08-14T21:53:07.5900103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5900199Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5900545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5900688Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5900977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5901109Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5901415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5901518Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5901523Z 2025-08-14T21:53:07.5901630Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5901834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5901910Z return mod(**inputs) 2025-08-14T21:53:07.5902197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5902282Z outputs = self.mobilebert( 2025-08-14T21:53:07.5902572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5902648Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5902944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5903019Z layer_outputs = layer_module( 2025-08-14T21:53:07.5903334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5903471Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5903760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5903854Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5903860Z 2025-08-14T21:53:07.5903965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5904193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5904272Z return mod(**inputs) 2025-08-14T21:53:07.5904571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5904653Z outputs = self.mobilebert( 2025-08-14T21:53:07.5904947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5905023Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5905324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5905398Z layer_outputs = layer_module( 2025-08-14T21:53:07.5905688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5905819Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5906124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5906249Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5906252Z 2025-08-14T21:53:07.5906357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5906563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5906640Z return mod(**inputs) 2025-08-14T21:53:07.5906925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5907002Z outputs = self.mobilebert( 2025-08-14T21:53:07.5907291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5907361Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5907643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5907732Z layer_outputs = layer_module( 2025-08-14T21:53:07.5908005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5908173Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5908447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.5908548Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.5908551Z 2025-08-14T21:53:07.5908784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5908988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5909065Z return mod(**inputs) 2025-08-14T21:53:07.5909343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5909423Z outputs = self.mobilebert( 2025-08-14T21:53:07.5909744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5909820Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5910113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5910187Z layer_outputs = layer_module( 2025-08-14T21:53:07.5910476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5910647Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5910963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.5911100Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.5911391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5911490Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5911494Z 2025-08-14T21:53:07.5911609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5911814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5911890Z return mod(**inputs) 2025-08-14T21:53:07.5912176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5912253Z outputs = self.mobilebert( 2025-08-14T21:53:07.5912552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5912630Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5912919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5913008Z layer_outputs = layer_module( 2025-08-14T21:53:07.5913303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5913480Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5913777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5913950Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5914254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.5914346Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5914377Z 2025-08-14T21:53:07.5914494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5914706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5914776Z return mod(**inputs) 2025-08-14T21:53:07.5915079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5915154Z outputs = self.mobilebert( 2025-08-14T21:53:07.5915455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5915534Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5915886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5915977Z layer_outputs = layer_module( 2025-08-14T21:53:07.5916281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5916470Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5916777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5916904Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5917209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.5917340Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5917648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5917749Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5917754Z 2025-08-14T21:53:07.5917855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5918054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5918122Z return mod(**inputs) 2025-08-14T21:53:07.5918393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5918472Z outputs = self.mobilebert( 2025-08-14T21:53:07.5918746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5918821Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5919101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5919174Z layer_outputs = layer_module( 2025-08-14T21:53:07.5919454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5919615Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5919891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5920007Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5920281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5920391Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5920395Z 2025-08-14T21:53:07.5920496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5920688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5920761Z return mod(**inputs) 2025-08-14T21:53:07.5921052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5921131Z outputs = self.mobilebert( 2025-08-14T21:53:07.5921409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5921481Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5921758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5921829Z layer_outputs = layer_module( 2025-08-14T21:53:07.5922104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5922198Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5922472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5922553Z self_outputs = self.self( 2025-08-14T21:53:07.5922844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.5922917Z self.value(value_tensor) 2025-08-14T21:53:07.5922920Z 2025-08-14T21:53:07.5923028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5923224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5923294Z return mod(**inputs) 2025-08-14T21:53:07.5923572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5923643Z outputs = self.mobilebert( 2025-08-14T21:53:07.5923944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5924019Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5924303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5924382Z layer_outputs = layer_module( 2025-08-14T21:53:07.5924657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5924821Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5925101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.5925212Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.5925498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5925583Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5925587Z 2025-08-14T21:53:07.5925694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5925894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5925959Z return mod(**inputs) 2025-08-14T21:53:07.5926253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5926323Z outputs = self.mobilebert( 2025-08-14T21:53:07.5926587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5926684Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5926953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5927030Z layer_outputs = layer_module( 2025-08-14T21:53:07.5927310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5927471Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5927740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5927841Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5928115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.5928200Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.5928465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5928563Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5928568Z 2025-08-14T21:53:07.5928665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5928856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5928940Z return mod(**inputs) 2025-08-14T21:53:07.5929205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5929280Z outputs = self.mobilebert( 2025-08-14T21:53:07.5929544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5929616Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5929890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5929973Z layer_outputs = layer_module( 2025-08-14T21:53:07.5930253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5930338Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5930606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5930685Z self_outputs = self.self( 2025-08-14T21:53:07.5930952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.5931021Z self.query(query_tensor) 2025-08-14T21:53:07.5931030Z 2025-08-14T21:53:07.5931130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5931316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5931387Z return mod(**inputs) 2025-08-14T21:53:07.5931659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5931726Z outputs = self.mobilebert( 2025-08-14T21:53:07.5931991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5932057Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5932326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5932391Z layer_outputs = layer_module( 2025-08-14T21:53:07.5932652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5932758Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5933029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5933097Z self_outputs = self.self( 2025-08-14T21:53:07.5933390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.5933456Z self.key(key_tensor) 2025-08-14T21:53:07.5933461Z 2025-08-14T21:53:07.5933547Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5933626Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.5933726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5933925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5933989Z return mod(**inputs) 2025-08-14T21:53:07.5934264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5934342Z outputs = self.mobilebert( 2025-08-14T21:53:07.5934616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5934706Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5934987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5935057Z layer_outputs = layer_module( 2025-08-14T21:53:07.5935331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5935413Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5935689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5935809Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5936098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.5936191Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5936195Z 2025-08-14T21:53:07.5936293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5936487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5936559Z return mod(**inputs) 2025-08-14T21:53:07.5936827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5936903Z outputs = self.mobilebert( 2025-08-14T21:53:07.5937176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5937249Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5937528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5937598Z layer_outputs = layer_module( 2025-08-14T21:53:07.5937875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5937955Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5938224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.5938351Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.5938621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.5938759Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5939036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5939125Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5939144Z 2025-08-14T21:53:07.5939248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5939436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5939500Z return mod(**inputs) 2025-08-14T21:53:07.5939775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5939844Z outputs = self.mobilebert( 2025-08-14T21:53:07.5940121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5940192Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5940459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5940537Z layer_outputs = layer_module( 2025-08-14T21:53:07.5940803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5940897Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5941190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5941298Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5941571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5941652Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5941657Z 2025-08-14T21:53:07.5941755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5941948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5942028Z return mod(**inputs) 2025-08-14T21:53:07.5942302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5942373Z outputs = self.mobilebert( 2025-08-14T21:53:07.5942641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5942721Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5942991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5943060Z layer_outputs = layer_module( 2025-08-14T21:53:07.5943344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5943439Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5943719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5943830Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5944102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5944221Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5944225Z 2025-08-14T21:53:07.5944322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5944530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5944593Z return mod(**inputs) 2025-08-14T21:53:07.5944882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5944959Z outputs = self.mobilebert( 2025-08-14T21:53:07.5945224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5945317Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5945580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5945650Z layer_outputs = layer_module( 2025-08-14T21:53:07.5945922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5946013Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5946277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5946404Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5946669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5946756Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5946760Z 2025-08-14T21:53:07.5946857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5947069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5947142Z return mod(**inputs) 2025-08-14T21:53:07.5947405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5947480Z outputs = self.mobilebert( 2025-08-14T21:53:07.5947746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5947817Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5948141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5948211Z layer_outputs = layer_module( 2025-08-14T21:53:07.5948476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5948575Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5948837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5948963Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5949227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5949345Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5949615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5949704Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5949708Z 2025-08-14T21:53:07.5949813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5950003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5950067Z return mod(**inputs) 2025-08-14T21:53:07.5950335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5950404Z outputs = self.mobilebert( 2025-08-14T21:53:07.5950669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5950764Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5951033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5951111Z layer_outputs = layer_module( 2025-08-14T21:53:07.5951387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5951499Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5951786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5951902Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5952208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5952297Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5952302Z 2025-08-14T21:53:07.5952409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5952623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5952693Z return mod(**inputs) 2025-08-14T21:53:07.5952992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5953076Z outputs = self.mobilebert( 2025-08-14T21:53:07.5953397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5953483Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5953784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5953859Z layer_outputs = layer_module( 2025-08-14T21:53:07.5954170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5954268Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5954594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5954716Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5955014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5955140Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5955144Z 2025-08-14T21:53:07.5955248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5955473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5955541Z return mod(**inputs) 2025-08-14T21:53:07.5955924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5956021Z outputs = self.mobilebert( 2025-08-14T21:53:07.5956334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5956414Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5956731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5956810Z layer_outputs = layer_module( 2025-08-14T21:53:07.5957105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5957198Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5957477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5957629Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5957900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5957991Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5958013Z 2025-08-14T21:53:07.5958111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5958304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5958376Z return mod(**inputs) 2025-08-14T21:53:07.5958643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5958712Z outputs = self.mobilebert( 2025-08-14T21:53:07.5958993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5959063Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5959331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5959397Z layer_outputs = layer_module( 2025-08-14T21:53:07.5959661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5959762Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5960061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5960190Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5960458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5960576Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5960847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5960953Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5960957Z 2025-08-14T21:53:07.5961057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5961253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5961315Z return mod(**inputs) 2025-08-14T21:53:07.5961590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5961659Z outputs = self.mobilebert( 2025-08-14T21:53:07.5961926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5962006Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5962270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5962346Z layer_outputs = layer_module( 2025-08-14T21:53:07.5962612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5962703Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5962978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5963085Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5963351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5963438Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5963457Z 2025-08-14T21:53:07.5963557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5963752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5963827Z return mod(**inputs) 2025-08-14T21:53:07.5964086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5964177Z outputs = self.mobilebert( 2025-08-14T21:53:07.5964435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5964510Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5964770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5964837Z layer_outputs = layer_module( 2025-08-14T21:53:07.5965106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5965197Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5965463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.5965580Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.5965844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5965972Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5965976Z 2025-08-14T21:53:07.5966073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5966260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5966330Z return mod(**inputs) 2025-08-14T21:53:07.5966592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5966670Z outputs = self.mobilebert( 2025-08-14T21:53:07.5966951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5967024Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5967299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5967370Z layer_outputs = layer_module( 2025-08-14T21:53:07.5967635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5967733Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5967999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5968126Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5968394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.5968476Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5968481Z 2025-08-14T21:53:07.5968587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5968777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5968848Z return mod(**inputs) 2025-08-14T21:53:07.5969116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5969186Z outputs = self.mobilebert( 2025-08-14T21:53:07.5969461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5969550Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5969815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5969892Z layer_outputs = layer_module( 2025-08-14T21:53:07.5970156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.5970274Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.5970539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.5970656Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.5970927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.5971042Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5971316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5971408Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5971411Z 2025-08-14T21:53:07.5971509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5971703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5971766Z return mod(**inputs) 2025-08-14T21:53:07.5972056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5972125Z outputs = self.mobilebert( 2025-08-14T21:53:07.5972392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5972469Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5972738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5972820Z layer_outputs = layer_module( 2025-08-14T21:53:07.5973097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5973216Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5973490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.5973574Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.5973577Z 2025-08-14T21:53:07.5973674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5973868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5973930Z return mod(**inputs) 2025-08-14T21:53:07.5974207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5974278Z outputs = self.mobilebert( 2025-08-14T21:53:07.5974551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5974632Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5974907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5974976Z layer_outputs = layer_module( 2025-08-14T21:53:07.5975259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.5975377Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.5975655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.5975786Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.5975789Z 2025-08-14T21:53:07.5975890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5976089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5976171Z return mod(**inputs) 2025-08-14T21:53:07.5976461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5976532Z outputs = self.mobilebert( 2025-08-14T21:53:07.5976812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5976893Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5977237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5977308Z layer_outputs = layer_module( 2025-08-14T21:53:07.5977599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5977755Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5978047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.5978158Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.5978162Z 2025-08-14T21:53:07.5978264Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5978464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5978529Z return mod(**inputs) 2025-08-14T21:53:07.5978811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5978881Z outputs = self.mobilebert( 2025-08-14T21:53:07.5979168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5979248Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5979520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5979590Z layer_outputs = layer_module( 2025-08-14T21:53:07.5979868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5980022Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5980302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.5980424Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.5980695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5980795Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5980801Z 2025-08-14T21:53:07.5980903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5981102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5981166Z return mod(**inputs) 2025-08-14T21:53:07.5981436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5981515Z outputs = self.mobilebert( 2025-08-14T21:53:07.5981784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5981878Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5982162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5982231Z layer_outputs = layer_module( 2025-08-14T21:53:07.5982545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5982701Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5982975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5983102Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5983381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.5983474Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.5983477Z 2025-08-14T21:53:07.5983577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5983772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5983849Z return mod(**inputs) 2025-08-14T21:53:07.5984125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5984203Z outputs = self.mobilebert( 2025-08-14T21:53:07.5984497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5984569Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5984851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5984923Z layer_outputs = layer_module( 2025-08-14T21:53:07.5985195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.5985373Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.5985648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.5985775Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.5986047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.5986166Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.5986448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5986542Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5986546Z 2025-08-14T21:53:07.5986650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5986846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5986911Z return mod(**inputs) 2025-08-14T21:53:07.5987192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5987263Z outputs = self.mobilebert( 2025-08-14T21:53:07.5987540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5987620Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5987899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5987975Z layer_outputs = layer_module( 2025-08-14T21:53:07.5988280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5988439Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5988723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5988851Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5989134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5989220Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5989225Z 2025-08-14T21:53:07.5989332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5989537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5989607Z return mod(**inputs) 2025-08-14T21:53:07.5989903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5989984Z outputs = self.mobilebert( 2025-08-14T21:53:07.5990277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5990365Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5990676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5990752Z layer_outputs = layer_module( 2025-08-14T21:53:07.5991048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5991136Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.5991442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.5991518Z self_outputs = self.self( 2025-08-14T21:53:07.5991823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.5991909Z self.value(value_tensor) 2025-08-14T21:53:07.5991913Z 2025-08-14T21:53:07.5992018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5992233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5992302Z return mod(**inputs) 2025-08-14T21:53:07.5992594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5992677Z outputs = self.mobilebert( 2025-08-14T21:53:07.5992973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5993049Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5993352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5993425Z layer_outputs = layer_module( 2025-08-14T21:53:07.5993728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5993894Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5994190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.5994312Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.5994607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.5994718Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.5994722Z 2025-08-14T21:53:07.5994827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5995031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5995128Z return mod(**inputs) 2025-08-14T21:53:07.5995416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5995493Z outputs = self.mobilebert( 2025-08-14T21:53:07.5995870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5995950Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5996247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5996325Z layer_outputs = layer_module( 2025-08-14T21:53:07.5996615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.5996788Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.5997087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.5997208Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.5997521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.5997615Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.5997915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.5998014Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.5998018Z 2025-08-14T21:53:07.5998125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.5998357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.5998426Z return mod(**inputs) 2025-08-14T21:53:07.5998733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.5998810Z outputs = self.mobilebert( 2025-08-14T21:53:07.5999103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.5999188Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.5999483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.5999564Z layer_outputs = layer_module( 2025-08-14T21:53:07.5999863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.5999955Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6000257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6000332Z self_outputs = self.self( 2025-08-14T21:53:07.6000624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6000707Z self.query(query_tensor) 2025-08-14T21:53:07.6000710Z 2025-08-14T21:53:07.6000818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6001030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6001098Z return mod(**inputs) 2025-08-14T21:53:07.6001390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6001494Z outputs = self.mobilebert( 2025-08-14T21:53:07.6001785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6001886Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6002177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6002253Z layer_outputs = layer_module( 2025-08-14T21:53:07.6002555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6002644Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6002951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6003035Z self_outputs = self.self( 2025-08-14T21:53:07.6003330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6003410Z self.key(key_tensor) 2025-08-14T21:53:07.6003418Z 2025-08-14T21:53:07.6003506Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6003590Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6003708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6003933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6004004Z return mod(**inputs) 2025-08-14T21:53:07.6004301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6004376Z outputs = self.mobilebert( 2025-08-14T21:53:07.6004678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6004756Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6005060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6005147Z layer_outputs = layer_module( 2025-08-14T21:53:07.6005440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6005539Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6005835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6005965Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6006272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6006363Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6006367Z 2025-08-14T21:53:07.6006477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6006693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6006764Z return mod(**inputs) 2025-08-14T21:53:07.6007060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6007137Z outputs = self.mobilebert( 2025-08-14T21:53:07.6007427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6007511Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6007814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6007928Z layer_outputs = layer_module( 2025-08-14T21:53:07.6008216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6008303Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6008599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6008868Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6009178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6009317Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6009606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6009712Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6009718Z 2025-08-14T21:53:07.6009825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6010031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6010110Z return mod(**inputs) 2025-08-14T21:53:07.6010415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6010496Z outputs = self.mobilebert( 2025-08-14T21:53:07.6010821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6010894Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6011177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6011247Z layer_outputs = layer_module( 2025-08-14T21:53:07.6011516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6011616Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6011908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6012028Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6012301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6012384Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6012387Z 2025-08-14T21:53:07.6012497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6012693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6012767Z return mod(**inputs) 2025-08-14T21:53:07.6013042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6013113Z outputs = self.mobilebert( 2025-08-14T21:53:07.6013396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6013470Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6013743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6013821Z layer_outputs = layer_module( 2025-08-14T21:53:07.6014090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6014188Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6014458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6014600Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6014884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6014996Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6015024Z 2025-08-14T21:53:07.6015133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6015327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6015390Z return mod(**inputs) 2025-08-14T21:53:07.6015670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6015739Z outputs = self.mobilebert( 2025-08-14T21:53:07.6016017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6016091Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6016365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6016444Z layer_outputs = layer_module( 2025-08-14T21:53:07.6016734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6016833Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6017151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6017286Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6017583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6017673Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6017677Z 2025-08-14T21:53:07.6017782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6018015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6018084Z return mod(**inputs) 2025-08-14T21:53:07.6018391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6018466Z outputs = self.mobilebert( 2025-08-14T21:53:07.6018761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6018842Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6019123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6019197Z layer_outputs = layer_module( 2025-08-14T21:53:07.6019486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6019580Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6019867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6019993Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6020274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6020404Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6020685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6020784Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6020808Z 2025-08-14T21:53:07.6020909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6021106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6021181Z return mod(**inputs) 2025-08-14T21:53:07.6021459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6021551Z outputs = self.mobilebert( 2025-08-14T21:53:07.6021839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6021910Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6022197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6022268Z layer_outputs = layer_module( 2025-08-14T21:53:07.6022551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6022650Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6022936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6023055Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6023356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6023442Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6023446Z 2025-08-14T21:53:07.6023553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6023743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6023807Z return mod(**inputs) 2025-08-14T21:53:07.6024089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6024159Z outputs = self.mobilebert( 2025-08-14T21:53:07.6024456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6024532Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6024810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6024890Z layer_outputs = layer_module( 2025-08-14T21:53:07.6025166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6025266Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6025547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6025658Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6025954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6026063Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6026068Z 2025-08-14T21:53:07.6026173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6026366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6026430Z return mod(**inputs) 2025-08-14T21:53:07.6026711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6026781Z outputs = self.mobilebert( 2025-08-14T21:53:07.6027059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6027159Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6027435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6027512Z layer_outputs = layer_module( 2025-08-14T21:53:07.6027803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6027895Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6028180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6028303Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6028590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6028688Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6028691Z 2025-08-14T21:53:07.6028796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6029006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6029073Z return mod(**inputs) 2025-08-14T21:53:07.6029365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6029447Z outputs = self.mobilebert( 2025-08-14T21:53:07.6029767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6029851Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6030140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6030213Z layer_outputs = layer_module( 2025-08-14T21:53:07.6030511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6030634Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6030923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6031059Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6031345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6031477Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6031764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6031861Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6031875Z 2025-08-14T21:53:07.6031983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6032189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6032270Z return mod(**inputs) 2025-08-14T21:53:07.6032561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6032640Z outputs = self.mobilebert( 2025-08-14T21:53:07.6032939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6033018Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6033314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6033393Z layer_outputs = layer_module( 2025-08-14T21:53:07.6033698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6033825Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6034115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6034248Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6034543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6034631Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6034635Z 2025-08-14T21:53:07.6034747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6034952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6035020Z return mod(**inputs) 2025-08-14T21:53:07.6035319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6035394Z outputs = self.mobilebert( 2025-08-14T21:53:07.6035690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6035833Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6036124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6036228Z layer_outputs = layer_module( 2025-08-14T21:53:07.6036515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6036614Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6036917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6037028Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6037326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6037448Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6037453Z 2025-08-14T21:53:07.6037553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6037755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6037819Z return mod(**inputs) 2025-08-14T21:53:07.6038093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6038163Z outputs = self.mobilebert( 2025-08-14T21:53:07.6038586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6038674Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6038940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6039010Z layer_outputs = layer_module( 2025-08-14T21:53:07.6039296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6039387Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6039665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6039783Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6040047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6040161Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6040164Z 2025-08-14T21:53:07.6040262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6040465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6040528Z return mod(**inputs) 2025-08-14T21:53:07.6040811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6040889Z outputs = self.mobilebert( 2025-08-14T21:53:07.6041163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6041233Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6041510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6041577Z layer_outputs = layer_module( 2025-08-14T21:53:07.6041861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6041956Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6042233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6042365Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6042662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6042789Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6043059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6043150Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6043155Z 2025-08-14T21:53:07.6043262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6043454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6043541Z return mod(**inputs) 2025-08-14T21:53:07.6043817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6043888Z outputs = self.mobilebert( 2025-08-14T21:53:07.6044168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6044240Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6044512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6044590Z layer_outputs = layer_module( 2025-08-14T21:53:07.6044862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6044990Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6045265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6045350Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6045353Z 2025-08-14T21:53:07.6045460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6045659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6045730Z return mod(**inputs) 2025-08-14T21:53:07.6046003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6046073Z outputs = self.mobilebert( 2025-08-14T21:53:07.6046350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6046443Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6046715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6046813Z layer_outputs = layer_module( 2025-08-14T21:53:07.6047082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6047207Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6047479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6047589Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6047593Z 2025-08-14T21:53:07.6047701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6047895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6047967Z return mod(**inputs) 2025-08-14T21:53:07.6048244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6048316Z outputs = self.mobilebert( 2025-08-14T21:53:07.6048595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6048724Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6048997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6049076Z layer_outputs = layer_module( 2025-08-14T21:53:07.6049347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6049511Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6049798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6049896Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6049901Z 2025-08-14T21:53:07.6050011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6050207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6050281Z return mod(**inputs) 2025-08-14T21:53:07.6050553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6050624Z outputs = self.mobilebert( 2025-08-14T21:53:07.6050903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6050978Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6051252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6051331Z layer_outputs = layer_module( 2025-08-14T21:53:07.6051603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6051770Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6052046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6052169Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6052449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6052557Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6052561Z 2025-08-14T21:53:07.6052667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6052859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6052924Z return mod(**inputs) 2025-08-14T21:53:07.6053227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6053296Z outputs = self.mobilebert( 2025-08-14T21:53:07.6053575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6053654Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6053928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6054007Z layer_outputs = layer_module( 2025-08-14T21:53:07.6054296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6054453Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6054735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6054858Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6055172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6055258Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6055261Z 2025-08-14T21:53:07.6055361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6055560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6055627Z return mod(**inputs) 2025-08-14T21:53:07.6055905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6055991Z outputs = self.mobilebert( 2025-08-14T21:53:07.6056269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6056351Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6056640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6056715Z layer_outputs = layer_module( 2025-08-14T21:53:07.6057012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6057174Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6057473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6057603Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6057893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6058022Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6058294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6058390Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6058394Z 2025-08-14T21:53:07.6058494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6058686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6058777Z return mod(**inputs) 2025-08-14T21:53:07.6059059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6059132Z outputs = self.mobilebert( 2025-08-14T21:53:07.6059418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6059507Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6059787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6059857Z layer_outputs = layer_module( 2025-08-14T21:53:07.6060136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6060299Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6060570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6060681Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6060956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6061038Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6061042Z 2025-08-14T21:53:07.6061147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6061364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6061434Z return mod(**inputs) 2025-08-14T21:53:07.6061697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6061765Z outputs = self.mobilebert( 2025-08-14T21:53:07.6062042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6062111Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6062393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6062472Z layer_outputs = layer_module( 2025-08-14T21:53:07.6062739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6062831Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6063098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6063172Z self_outputs = self.self( 2025-08-14T21:53:07.6063470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6063549Z self.value(value_tensor) 2025-08-14T21:53:07.6063552Z 2025-08-14T21:53:07.6063663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6063871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6063940Z return mod(**inputs) 2025-08-14T21:53:07.6064238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6064312Z outputs = self.mobilebert( 2025-08-14T21:53:07.6064609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6064692Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6064992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6065099Z layer_outputs = layer_module( 2025-08-14T21:53:07.6065373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6065531Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6065831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6065939Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6066223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6066305Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6066309Z 2025-08-14T21:53:07.6066410Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6066611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6066679Z return mod(**inputs) 2025-08-14T21:53:07.6066961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6067037Z outputs = self.mobilebert( 2025-08-14T21:53:07.6067304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6067382Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6067665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6067736Z layer_outputs = layer_module( 2025-08-14T21:53:07.6068008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6068157Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6068431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6068554Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6068827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6068922Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6069196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6069288Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6069300Z 2025-08-14T21:53:07.6069403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6069598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6069671Z return mod(**inputs) 2025-08-14T21:53:07.6069946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6070017Z outputs = self.mobilebert( 2025-08-14T21:53:07.6070298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6070371Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6070652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6070723Z layer_outputs = layer_module( 2025-08-14T21:53:07.6070998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6071089Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6071366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6071458Z self_outputs = self.self( 2025-08-14T21:53:07.6071742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6071830Z self.query(query_tensor) 2025-08-14T21:53:07.6071834Z 2025-08-14T21:53:07.6071942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6072135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6072201Z return mod(**inputs) 2025-08-14T21:53:07.6072482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6102071Z outputs = self.mobilebert( 2025-08-14T21:53:07.6102614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6102732Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6103056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6103140Z layer_outputs = layer_module( 2025-08-14T21:53:07.6103444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6103551Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6103943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6104040Z self_outputs = self.self( 2025-08-14T21:53:07.6104340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6104416Z self.key(key_tensor) 2025-08-14T21:53:07.6104426Z 2025-08-14T21:53:07.6104528Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6104614Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6104736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6105010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6105090Z return mod(**inputs) 2025-08-14T21:53:07.6105398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6105482Z outputs = self.mobilebert( 2025-08-14T21:53:07.6105777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6105868Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6106169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6106250Z layer_outputs = layer_module( 2025-08-14T21:53:07.6106562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6106655Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6106951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6107085Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6107377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6107478Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6107483Z 2025-08-14T21:53:07.6107597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6107830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6107938Z return mod(**inputs) 2025-08-14T21:53:07.6108224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6108307Z outputs = self.mobilebert( 2025-08-14T21:53:07.6108587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6108862Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6109185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6109263Z layer_outputs = layer_module( 2025-08-14T21:53:07.6109564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6109655Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6109950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6110090Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6110397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6110548Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6110909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6111016Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6111021Z 2025-08-14T21:53:07.6111140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6111351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6111425Z return mod(**inputs) 2025-08-14T21:53:07.6111719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6111794Z outputs = self.mobilebert( 2025-08-14T21:53:07.6112495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6112586Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6112907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6112987Z layer_outputs = layer_module( 2025-08-14T21:53:07.6113284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6113400Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6113700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6113840Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6114151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6114242Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6114247Z 2025-08-14T21:53:07.6114366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6114578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6114649Z return mod(**inputs) 2025-08-14T21:53:07.6114947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6115024Z outputs = self.mobilebert( 2025-08-14T21:53:07.6115331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6115444Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6115818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6115915Z layer_outputs = layer_module( 2025-08-14T21:53:07.6116251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6116365Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6116667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6116790Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6117101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6117228Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6117232Z 2025-08-14T21:53:07.6117343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6117574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6117641Z return mod(**inputs) 2025-08-14T21:53:07.6117926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6118018Z outputs = self.mobilebert( 2025-08-14T21:53:07.6118324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6118414Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6118717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6118806Z layer_outputs = layer_module( 2025-08-14T21:53:07.6119108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6119231Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6119537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6119677Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6119977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6120076Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6120080Z 2025-08-14T21:53:07.6120188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6120407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6120481Z return mod(**inputs) 2025-08-14T21:53:07.6120775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6120862Z outputs = self.mobilebert( 2025-08-14T21:53:07.6121161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6121250Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6121549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6121626Z layer_outputs = layer_module( 2025-08-14T21:53:07.6121932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6122033Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6122402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6122545Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6122846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6123012Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6123309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6123411Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6123415Z 2025-08-14T21:53:07.6123532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6123744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6123821Z return mod(**inputs) 2025-08-14T21:53:07.6124120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6124197Z outputs = self.mobilebert( 2025-08-14T21:53:07.6124502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6124582Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6124894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6124977Z layer_outputs = layer_module( 2025-08-14T21:53:07.6125270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6125369Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6125641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6125753Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6126048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6126135Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6126138Z 2025-08-14T21:53:07.6126246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6126440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6126503Z return mod(**inputs) 2025-08-14T21:53:07.6126785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6126853Z outputs = self.mobilebert( 2025-08-14T21:53:07.6127131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6127204Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6127478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6127555Z layer_outputs = layer_module( 2025-08-14T21:53:07.6127834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6127926Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6128209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6128317Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6128600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6128740Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6128745Z 2025-08-14T21:53:07.6128846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6129051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6129136Z return mod(**inputs) 2025-08-14T21:53:07.6129415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6129488Z outputs = self.mobilebert( 2025-08-14T21:53:07.6129757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6129837Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6130109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6130182Z layer_outputs = layer_module( 2025-08-14T21:53:07.6130460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6130554Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6130834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6130957Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6131244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6131339Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6131342Z 2025-08-14T21:53:07.6131442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6131646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6131713Z return mod(**inputs) 2025-08-14T21:53:07.6131986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6132082Z outputs = self.mobilebert( 2025-08-14T21:53:07.6132354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6132429Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6132708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6132780Z layer_outputs = layer_module( 2025-08-14T21:53:07.6133061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6133154Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6133427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6133556Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6133828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6133958Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6134231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6134324Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6134327Z 2025-08-14T21:53:07.6134434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6134629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6134696Z return mod(**inputs) 2025-08-14T21:53:07.6134995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6135064Z outputs = self.mobilebert( 2025-08-14T21:53:07.6135343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6135432Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6135704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6135781Z layer_outputs = layer_module( 2025-08-14T21:53:07.6136054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6136151Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6136422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6136534Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6136812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6136904Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6136908Z 2025-08-14T21:53:07.6137015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6137223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6137289Z return mod(**inputs) 2025-08-14T21:53:07.6137581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6137654Z outputs = self.mobilebert( 2025-08-14T21:53:07.6137953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6138031Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6138330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6138410Z layer_outputs = layer_module( 2025-08-14T21:53:07.6138684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6138775Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6139055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6139165Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6139446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6139560Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6139564Z 2025-08-14T21:53:07.6139666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6139868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6139937Z return mod(**inputs) 2025-08-14T21:53:07.6140218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6140288Z outputs = self.mobilebert( 2025-08-14T21:53:07.6140562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6140642Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6140916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6141007Z layer_outputs = layer_module( 2025-08-14T21:53:07.6141290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6141383Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6141664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6141803Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6142076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6142166Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6142170Z 2025-08-14T21:53:07.6142268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6142469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6142534Z return mod(**inputs) 2025-08-14T21:53:07.6142802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6142881Z outputs = self.mobilebert( 2025-08-14T21:53:07.6143151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6143223Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6143518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6143589Z layer_outputs = layer_module( 2025-08-14T21:53:07.6143868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6143958Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6144228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6144357Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6144648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6144776Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6145051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6145142Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6145145Z 2025-08-14T21:53:07.6145253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6145447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6145517Z return mod(**inputs) 2025-08-14T21:53:07.6145791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6145862Z outputs = self.mobilebert( 2025-08-14T21:53:07.6146148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6146223Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6146499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6146576Z layer_outputs = layer_module( 2025-08-14T21:53:07.6146848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6146976Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6147253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6147353Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6147357Z 2025-08-14T21:53:07.6147466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6147659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6147748Z return mod(**inputs) 2025-08-14T21:53:07.6148024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6148095Z outputs = self.mobilebert( 2025-08-14T21:53:07.6148376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6148446Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6148723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6148802Z layer_outputs = layer_module( 2025-08-14T21:53:07.6149077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6149201Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6149476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6149602Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6149606Z 2025-08-14T21:53:07.6149714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6149907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6149980Z return mod(**inputs) 2025-08-14T21:53:07.6150248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6150320Z outputs = self.mobilebert( 2025-08-14T21:53:07.6150639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6150717Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6151007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6151088Z layer_outputs = layer_module( 2025-08-14T21:53:07.6151379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6151558Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6151847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6151944Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6151947Z 2025-08-14T21:53:07.6152054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6152250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6152321Z return mod(**inputs) 2025-08-14T21:53:07.6152597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6152666Z outputs = self.mobilebert( 2025-08-14T21:53:07.6152970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6153044Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6153332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6153413Z layer_outputs = layer_module( 2025-08-14T21:53:07.6153731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6153908Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6154200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6154344Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6154641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6154738Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6154742Z 2025-08-14T21:53:07.6154856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6155062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6155132Z return mod(**inputs) 2025-08-14T21:53:07.6155433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6155510Z outputs = self.mobilebert( 2025-08-14T21:53:07.6155900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6155996Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6156319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6156407Z layer_outputs = layer_module( 2025-08-14T21:53:07.6156712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6156894Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6157193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6157343Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6157644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6157735Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6157739Z 2025-08-14T21:53:07.6157848Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6158064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6158145Z return mod(**inputs) 2025-08-14T21:53:07.6158443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6158517Z outputs = self.mobilebert( 2025-08-14T21:53:07.6158806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6158891Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6159182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6159259Z layer_outputs = layer_module( 2025-08-14T21:53:07.6159557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6159718Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6160012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6160141Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6160458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6160594Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6160885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6161007Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6161011Z 2025-08-14T21:53:07.6161114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6161323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6161399Z return mod(**inputs) 2025-08-14T21:53:07.6161686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6161761Z outputs = self.mobilebert( 2025-08-14T21:53:07.6162057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6162132Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6162428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6162503Z layer_outputs = layer_module( 2025-08-14T21:53:07.6162789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6162982Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6163278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6163408Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6163673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6163755Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6163758Z 2025-08-14T21:53:07.6163877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6164069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6164137Z return mod(**inputs) 2025-08-14T21:53:07.6164419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6164489Z outputs = self.mobilebert( 2025-08-14T21:53:07.6164779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6164850Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6165116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6165196Z layer_outputs = layer_module( 2025-08-14T21:53:07.6165467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6165561Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6165837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6165907Z self_outputs = self.self( 2025-08-14T21:53:07.6166193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6166263Z self.value(value_tensor) 2025-08-14T21:53:07.6166266Z 2025-08-14T21:53:07.6166363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6166560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6166641Z return mod(**inputs) 2025-08-14T21:53:07.6166921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6166989Z outputs = self.mobilebert( 2025-08-14T21:53:07.6167279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6167357Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6167616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6167692Z layer_outputs = layer_module( 2025-08-14T21:53:07.6167955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6168109Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6168385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6168493Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6168755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6168842Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6168846Z 2025-08-14T21:53:07.6168961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6169159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6169222Z return mod(**inputs) 2025-08-14T21:53:07.6169487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6169563Z outputs = self.mobilebert( 2025-08-14T21:53:07.6169829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6169924Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6170191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6170261Z layer_outputs = layer_module( 2025-08-14T21:53:07.6170539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6170688Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6170968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6171074Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6171355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6171448Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6171732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6171823Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6171833Z 2025-08-14T21:53:07.6171932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6172123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6172194Z return mod(**inputs) 2025-08-14T21:53:07.6172464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6172533Z outputs = self.mobilebert( 2025-08-14T21:53:07.6172880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6172951Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6173232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6173317Z layer_outputs = layer_module( 2025-08-14T21:53:07.6173591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6173684Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6173960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6174029Z self_outputs = self.self( 2025-08-14T21:53:07.6174314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6174387Z self.query(query_tensor) 2025-08-14T21:53:07.6174390Z 2025-08-14T21:53:07.6174495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6174686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6174750Z return mod(**inputs) 2025-08-14T21:53:07.6175025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6175117Z outputs = self.mobilebert( 2025-08-14T21:53:07.6175391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6175460Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6175722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6175798Z layer_outputs = layer_module( 2025-08-14T21:53:07.6176061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6176157Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6176430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6176499Z self_outputs = self.self( 2025-08-14T21:53:07.6176771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6176835Z self.key(key_tensor) 2025-08-14T21:53:07.6176839Z 2025-08-14T21:53:07.6176917Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6177001Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6177103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6177290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6177361Z return mod(**inputs) 2025-08-14T21:53:07.6177628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6177702Z outputs = self.mobilebert( 2025-08-14T21:53:07.6177966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6178035Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6178306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6178375Z layer_outputs = layer_module( 2025-08-14T21:53:07.6178644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6178724Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6179008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6179134Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6179400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6179500Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6179503Z 2025-08-14T21:53:07.6179609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6179804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6179877Z return mod(**inputs) 2025-08-14T21:53:07.6180159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6180230Z outputs = self.mobilebert( 2025-08-14T21:53:07.6180533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6180603Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6180884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6180956Z layer_outputs = layer_module( 2025-08-14T21:53:07.6181252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6181342Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6181653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6181774Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6182059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6182186Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6182486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6182581Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6182584Z 2025-08-14T21:53:07.6182683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6182886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6182952Z return mod(**inputs) 2025-08-14T21:53:07.6183231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6183302Z outputs = self.mobilebert( 2025-08-14T21:53:07.6183592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6183679Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6183979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6184054Z layer_outputs = layer_module( 2025-08-14T21:53:07.6184348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6184442Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6184724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6184834Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6185108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6185222Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6185226Z 2025-08-14T21:53:07.6185326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6185543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6185632Z return mod(**inputs) 2025-08-14T21:53:07.6185937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6186019Z outputs = self.mobilebert( 2025-08-14T21:53:07.6186324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6186398Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6186712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6186787Z layer_outputs = layer_module( 2025-08-14T21:53:07.6187095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6187194Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6187504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6187629Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6187951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6188086Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6188090Z 2025-08-14T21:53:07.6188191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6188386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6188460Z return mod(**inputs) 2025-08-14T21:53:07.6188750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6188822Z outputs = self.mobilebert( 2025-08-14T21:53:07.6189102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6189174Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6189476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6189551Z layer_outputs = layer_module( 2025-08-14T21:53:07.6189841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6189946Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6190250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6190390Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6190682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6190772Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6190775Z 2025-08-14T21:53:07.6190892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6191097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6191171Z return mod(**inputs) 2025-08-14T21:53:07.6191472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6191549Z outputs = self.mobilebert( 2025-08-14T21:53:07.6191873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6191948Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6192250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6192352Z layer_outputs = layer_module( 2025-08-14T21:53:07.6192644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6192746Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6193037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6193165Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6193460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6193589Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6193899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6193996Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6194000Z 2025-08-14T21:53:07.6194107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6194335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6194404Z return mod(**inputs) 2025-08-14T21:53:07.6194694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6194774Z outputs = self.mobilebert( 2025-08-14T21:53:07.6195076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6195158Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6195475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6195551Z layer_outputs = layer_module( 2025-08-14T21:53:07.6195948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6196054Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6196354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6196474Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6196781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6196883Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6196887Z 2025-08-14T21:53:07.6196998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6197213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6197293Z return mod(**inputs) 2025-08-14T21:53:07.6197596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6197683Z outputs = self.mobilebert( 2025-08-14T21:53:07.6197992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6198069Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6198367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6198470Z layer_outputs = layer_module( 2025-08-14T21:53:07.6198770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6198869Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6199162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6199313Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6199609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6199724Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6199736Z 2025-08-14T21:53:07.6199845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6200051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6200130Z return mod(**inputs) 2025-08-14T21:53:07.6200425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6200499Z outputs = self.mobilebert( 2025-08-14T21:53:07.6200797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6200873Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6201186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6201261Z layer_outputs = layer_module( 2025-08-14T21:53:07.6201550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6201655Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6201944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6202089Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6202385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6202476Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6202480Z 2025-08-14T21:53:07.6202594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6202804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6202872Z return mod(**inputs) 2025-08-14T21:53:07.6203166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6203238Z outputs = self.mobilebert( 2025-08-14T21:53:07.6203533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6203607Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6203894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6203977Z layer_outputs = layer_module( 2025-08-14T21:53:07.6204264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6204360Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6204654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6204783Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6205078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6205223Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6205512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6205633Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6205637Z 2025-08-14T21:53:07.6205743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6205953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6206022Z return mod(**inputs) 2025-08-14T21:53:07.6206308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6206390Z outputs = self.mobilebert( 2025-08-14T21:53:07.6206678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6206763Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6207050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6207126Z layer_outputs = layer_module( 2025-08-14T21:53:07.6207420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6207533Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6207820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6207945Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6208231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6208328Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6208332Z 2025-08-14T21:53:07.6208442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6208922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6209012Z return mod(**inputs) 2025-08-14T21:53:07.6209288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6209369Z outputs = self.mobilebert( 2025-08-14T21:53:07.6209642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6209716Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6210001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6210074Z layer_outputs = layer_module( 2025-08-14T21:53:07.6210345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6210446Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6210720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6210840Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6211127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6211244Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6211248Z 2025-08-14T21:53:07.6211363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6211570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6211679Z return mod(**inputs) 2025-08-14T21:53:07.6211985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6212059Z outputs = self.mobilebert( 2025-08-14T21:53:07.6212364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6212470Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6212767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6212850Z layer_outputs = layer_module( 2025-08-14T21:53:07.6213144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6213249Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6213548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6213679Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6213983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6214074Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6214078Z 2025-08-14T21:53:07.6214191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6214429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6214502Z return mod(**inputs) 2025-08-14T21:53:07.6214808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6214883Z outputs = self.mobilebert( 2025-08-14T21:53:07.6215182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6215267Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6215601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6215689Z layer_outputs = layer_module( 2025-08-14T21:53:07.6215988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6216087Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6216384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6216512Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6216817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6216946Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6217239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6217338Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6217342Z 2025-08-14T21:53:07.6217443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6217644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6217710Z return mod(**inputs) 2025-08-14T21:53:07.6217983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6218061Z outputs = self.mobilebert( 2025-08-14T21:53:07.6218337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6218428Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6218718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6218788Z layer_outputs = layer_module( 2025-08-14T21:53:07.6219089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6219210Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6219484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6219576Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6219579Z 2025-08-14T21:53:07.6219678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6219877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6219944Z return mod(**inputs) 2025-08-14T21:53:07.6220214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6220289Z outputs = self.mobilebert( 2025-08-14T21:53:07.6220563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6220633Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6220928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6220999Z layer_outputs = layer_module( 2025-08-14T21:53:07.6221281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6221402Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6221681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6221812Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6221817Z 2025-08-14T21:53:07.6221921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6222121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6222186Z return mod(**inputs) 2025-08-14T21:53:07.6222458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6222533Z outputs = self.mobilebert( 2025-08-14T21:53:07.6222803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6222876Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6223160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6223231Z layer_outputs = layer_module( 2025-08-14T21:53:07.6223515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6223676Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6223956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6224058Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6224062Z 2025-08-14T21:53:07.6224162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6224364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6224466Z return mod(**inputs) 2025-08-14T21:53:07.6224733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6224810Z outputs = self.mobilebert( 2025-08-14T21:53:07.6225075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6225160Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6225438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6225509Z layer_outputs = layer_module( 2025-08-14T21:53:07.6225792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6225958Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6226246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6226385Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6226676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6226780Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6226784Z 2025-08-14T21:53:07.6226904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6227111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6227187Z return mod(**inputs) 2025-08-14T21:53:07.6227474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6227547Z outputs = self.mobilebert( 2025-08-14T21:53:07.6227840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6227915Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6228225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6228302Z layer_outputs = layer_module( 2025-08-14T21:53:07.6228594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6228765Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6229059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6229198Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6229500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6229591Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6229595Z 2025-08-14T21:53:07.6229711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6229923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6230001Z return mod(**inputs) 2025-08-14T21:53:07.6230303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6230380Z outputs = self.mobilebert( 2025-08-14T21:53:07.6230688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6230764Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6231064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6231166Z layer_outputs = layer_module( 2025-08-14T21:53:07.6231463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6231656Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6231964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6232096Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6232398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6232527Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6232841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6232942Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6232945Z 2025-08-14T21:53:07.6233057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6233276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6233349Z return mod(**inputs) 2025-08-14T21:53:07.6233659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6233746Z outputs = self.mobilebert( 2025-08-14T21:53:07.6234043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6234129Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6234436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6234513Z layer_outputs = layer_module( 2025-08-14T21:53:07.6234836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6235010Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6235316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6235435Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6235801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6235909Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6235917Z 2025-08-14T21:53:07.6236025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6236252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6236325Z return mod(**inputs) 2025-08-14T21:53:07.6236635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6236725Z outputs = self.mobilebert( 2025-08-14T21:53:07.6237026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6237115Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6237394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6237463Z layer_outputs = layer_module( 2025-08-14T21:53:07.6237744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6237853Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6238121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6238202Z self_outputs = self.self( 2025-08-14T21:53:07.6238468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6238558Z self.value(value_tensor) 2025-08-14T21:53:07.6238570Z 2025-08-14T21:53:07.6238670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6238858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6238931Z return mod(**inputs) 2025-08-14T21:53:07.6239195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6239262Z outputs = self.mobilebert( 2025-08-14T21:53:07.6239536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6239605Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6239875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6239945Z layer_outputs = layer_module( 2025-08-14T21:53:07.6240230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6240392Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6240657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6240763Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6241047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6241124Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6241144Z 2025-08-14T21:53:07.6241248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6241432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6241494Z return mod(**inputs) 2025-08-14T21:53:07.6241812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6241881Z outputs = self.mobilebert( 2025-08-14T21:53:07.6242154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6242224Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6242487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6242567Z layer_outputs = layer_module( 2025-08-14T21:53:07.6242830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6242983Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6243256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6243362Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6243634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6243717Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6243983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6244099Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6244102Z 2025-08-14T21:53:07.6244201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6244395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6244475Z return mod(**inputs) 2025-08-14T21:53:07.6244744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6244818Z outputs = self.mobilebert( 2025-08-14T21:53:07.6245086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6245163Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6245431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6245502Z layer_outputs = layer_module( 2025-08-14T21:53:07.6245775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6245858Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6246124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6246200Z self_outputs = self.self( 2025-08-14T21:53:07.6246481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6246558Z self.query(query_tensor) 2025-08-14T21:53:07.6246562Z 2025-08-14T21:53:07.6246658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6246848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6246922Z return mod(**inputs) 2025-08-14T21:53:07.6247188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6247288Z outputs = self.mobilebert( 2025-08-14T21:53:07.6247558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6247630Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6247905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6247973Z layer_outputs = layer_module( 2025-08-14T21:53:07.6248240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6248329Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6248601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6248678Z self_outputs = self.self( 2025-08-14T21:53:07.6248947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6249012Z self.key(key_tensor) 2025-08-14T21:53:07.6249015Z 2025-08-14T21:53:07.6249103Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6249180Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6249279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6249474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6249538Z return mod(**inputs) 2025-08-14T21:53:07.6249820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6249904Z outputs = self.mobilebert( 2025-08-14T21:53:07.6250165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6250243Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6250508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6250596Z layer_outputs = layer_module( 2025-08-14T21:53:07.6250855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6250932Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6251197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6251314Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6251573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6251659Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6251663Z 2025-08-14T21:53:07.6251758Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6251948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6252010Z return mod(**inputs) 2025-08-14T21:53:07.6252277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6252355Z outputs = self.mobilebert( 2025-08-14T21:53:07.6252619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6252696Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6252961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6253032Z layer_outputs = layer_module( 2025-08-14T21:53:07.6253327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6253413Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6253687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6253818Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6254089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6254220Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6254497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6254591Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6254594Z 2025-08-14T21:53:07.6254702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6254905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6254977Z return mod(**inputs) 2025-08-14T21:53:07.6255244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6255315Z outputs = self.mobilebert( 2025-08-14T21:53:07.6255588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6255663Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6255962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6256063Z layer_outputs = layer_module( 2025-08-14T21:53:07.6256359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6256461Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6256747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6256860Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6257141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6257224Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6257227Z 2025-08-14T21:53:07.6257336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6257529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6257596Z return mod(**inputs) 2025-08-14T21:53:07.6257876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6257946Z outputs = self.mobilebert( 2025-08-14T21:53:07.6258218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6258295Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6258581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6258663Z layer_outputs = layer_module( 2025-08-14T21:53:07.6258936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6259027Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6259310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6259440Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6259724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6259839Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6259842Z 2025-08-14T21:53:07.6259944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6260145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6260209Z return mod(**inputs) 2025-08-14T21:53:07.6260490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6260566Z outputs = self.mobilebert( 2025-08-14T21:53:07.6260846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6260924Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6261203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6261274Z layer_outputs = layer_module( 2025-08-14T21:53:07.6261559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6261651Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6261933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6262057Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6262338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6262450Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6262453Z 2025-08-14T21:53:07.6262555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6262773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6262839Z return mod(**inputs) 2025-08-14T21:53:07.6263119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6263200Z outputs = self.mobilebert( 2025-08-14T21:53:07.6263486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6263561Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6263856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6263932Z layer_outputs = layer_module( 2025-08-14T21:53:07.6264227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6264326Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6264613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6264765Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6265060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6265187Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6265457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6265549Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6265552Z 2025-08-14T21:53:07.6265677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6265874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6265941Z return mod(**inputs) 2025-08-14T21:53:07.6266221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6266293Z outputs = self.mobilebert( 2025-08-14T21:53:07.6266573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6266644Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6266924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6267003Z layer_outputs = layer_module( 2025-08-14T21:53:07.6267283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6267382Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6267663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6267774Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6268062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6268145Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6268148Z 2025-08-14T21:53:07.6268249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6268461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6268545Z return mod(**inputs) 2025-08-14T21:53:07.6268825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6268893Z outputs = self.mobilebert( 2025-08-14T21:53:07.6269187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6269269Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6269544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6269621Z layer_outputs = layer_module( 2025-08-14T21:53:07.6269895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6269985Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6270271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6270380Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6270657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6270776Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6270780Z 2025-08-14T21:53:07.6270905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6271116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6271185Z return mod(**inputs) 2025-08-14T21:53:07.6271472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6271555Z outputs = self.mobilebert( 2025-08-14T21:53:07.6271845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6271928Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6272265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6272344Z layer_outputs = layer_module( 2025-08-14T21:53:07.6272639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6272734Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6273018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6273152Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6273440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6273536Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6273539Z 2025-08-14T21:53:07.6273646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6273850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6273929Z return mod(**inputs) 2025-08-14T21:53:07.6274214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6274295Z outputs = self.mobilebert( 2025-08-14T21:53:07.6274581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6274656Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6274957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6275056Z layer_outputs = layer_module( 2025-08-14T21:53:07.6275354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6275457Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6275846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6275995Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6276290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6276419Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6276720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6276818Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6276821Z 2025-08-14T21:53:07.6276937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6277146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6277217Z return mod(**inputs) 2025-08-14T21:53:07.6277517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6277611Z outputs = self.mobilebert( 2025-08-14T21:53:07.6277902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6277988Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6278290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6278373Z layer_outputs = layer_module( 2025-08-14T21:53:07.6278692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6278790Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6279092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6279208Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6279515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6279604Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6279607Z 2025-08-14T21:53:07.6279712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6279924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6279996Z return mod(**inputs) 2025-08-14T21:53:07.6280292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6280365Z outputs = self.mobilebert( 2025-08-14T21:53:07.6280655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6280738Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6281026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6281099Z layer_outputs = layer_module( 2025-08-14T21:53:07.6281395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6281492Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6281821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6281940Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6282233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6282375Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6282378Z 2025-08-14T21:53:07.6282483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6282698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6282767Z return mod(**inputs) 2025-08-14T21:53:07.6283062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6283141Z outputs = self.mobilebert( 2025-08-14T21:53:07.6283433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6283509Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6283803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6283879Z layer_outputs = layer_module( 2025-08-14T21:53:07.6284191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6284290Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6284580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6284721Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6285011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6285108Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6285112Z 2025-08-14T21:53:07.6285233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6285443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6285521Z return mod(**inputs) 2025-08-14T21:53:07.6285810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6285882Z outputs = self.mobilebert( 2025-08-14T21:53:07.6286181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6286253Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6286555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6286629Z layer_outputs = layer_module( 2025-08-14T21:53:07.6286927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6287031Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6287328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6287464Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6287758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6287887Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6288192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6288307Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6288310Z 2025-08-14T21:53:07.6288424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6288641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6288723Z return mod(**inputs) 2025-08-14T21:53:07.6289008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6289080Z outputs = self.mobilebert( 2025-08-14T21:53:07.6289364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6289445Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6289726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6289806Z layer_outputs = layer_module( 2025-08-14T21:53:07.6290085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6290208Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6290504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6290587Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6290605Z 2025-08-14T21:53:07.6290714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6290908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6290973Z return mod(**inputs) 2025-08-14T21:53:07.6291254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6291326Z outputs = self.mobilebert( 2025-08-14T21:53:07.6291598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6291692Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6291964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6292042Z layer_outputs = layer_module( 2025-08-14T21:53:07.6292313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6292429Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6292707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6292815Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6292820Z 2025-08-14T21:53:07.6292929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6293135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6293202Z return mod(**inputs) 2025-08-14T21:53:07.6293495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6293570Z outputs = self.mobilebert( 2025-08-14T21:53:07.6293858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6293939Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6294228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6294308Z layer_outputs = layer_module( 2025-08-14T21:53:07.6294594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6294783Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6295078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6295207Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6295211Z 2025-08-14T21:53:07.6295322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6295537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6295602Z return mod(**inputs) 2025-08-14T21:53:07.6295878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6295948Z outputs = self.mobilebert( 2025-08-14T21:53:07.6296219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6296298Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6296570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6296649Z layer_outputs = layer_module( 2025-08-14T21:53:07.6296934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6297092Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6297372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6297492Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6297768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6297861Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6297865Z 2025-08-14T21:53:07.6297985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6298188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6298258Z return mod(**inputs) 2025-08-14T21:53:07.6298531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6298610Z outputs = self.mobilebert( 2025-08-14T21:53:07.6298884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6298962Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6299234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6299306Z layer_outputs = layer_module( 2025-08-14T21:53:07.6299586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6299742Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6300022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6300144Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6300418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6300508Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6300512Z 2025-08-14T21:53:07.6300612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6300834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6300898Z return mod(**inputs) 2025-08-14T21:53:07.6301169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6301264Z outputs = self.mobilebert( 2025-08-14T21:53:07.6301542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6301615Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6301901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6301972Z layer_outputs = layer_module( 2025-08-14T21:53:07.6302256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6302411Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6302691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6302822Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6303100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6303241Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6303528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6303622Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6303626Z 2025-08-14T21:53:07.6303740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6303948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6304017Z return mod(**inputs) 2025-08-14T21:53:07.6304330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6304410Z outputs = self.mobilebert( 2025-08-14T21:53:07.6304707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6304783Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6305071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6305153Z layer_outputs = layer_module( 2025-08-14T21:53:07.6305445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6305624Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6305916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6306033Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6306333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6306419Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6306423Z 2025-08-14T21:53:07.6306537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6306746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6306813Z return mod(**inputs) 2025-08-14T21:53:07.6307114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6307209Z outputs = self.mobilebert( 2025-08-14T21:53:07.6307498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6307579Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6307886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6307968Z layer_outputs = layer_module( 2025-08-14T21:53:07.6308261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6308351Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6308841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6308926Z self_outputs = self.self( 2025-08-14T21:53:07.6309230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6309316Z self.value(value_tensor) 2025-08-14T21:53:07.6309323Z 2025-08-14T21:53:07.6309435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6309658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6309731Z return mod(**inputs) 2025-08-14T21:53:07.6310110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6310197Z outputs = self.mobilebert( 2025-08-14T21:53:07.6310496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6310581Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6310882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6310959Z layer_outputs = layer_module( 2025-08-14T21:53:07.6311294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6311470Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6311771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6311900Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6312198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6312293Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6312297Z 2025-08-14T21:53:07.6312405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6312617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6312698Z return mod(**inputs) 2025-08-14T21:53:07.6312996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6313083Z outputs = self.mobilebert( 2025-08-14T21:53:07.6313382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6313461Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6313769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6313846Z layer_outputs = layer_module( 2025-08-14T21:53:07.6314143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6314352Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6314650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6314800Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6315099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6315192Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6315503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6315601Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6315605Z 2025-08-14T21:53:07.6315771Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6315992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6316065Z return mod(**inputs) 2025-08-14T21:53:07.6316372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6316450Z outputs = self.mobilebert( 2025-08-14T21:53:07.6316752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6316852Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6317153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6317237Z layer_outputs = layer_module( 2025-08-14T21:53:07.6317573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6317665Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6317991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6318082Z self_outputs = self.self( 2025-08-14T21:53:07.6318379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6318455Z self.query(query_tensor) 2025-08-14T21:53:07.6318459Z 2025-08-14T21:53:07.6318565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6318777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6318845Z return mod(**inputs) 2025-08-14T21:53:07.6319159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6319244Z outputs = self.mobilebert( 2025-08-14T21:53:07.6319568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6319653Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6319936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6320005Z layer_outputs = layer_module( 2025-08-14T21:53:07.6320285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6320366Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6320638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6320706Z self_outputs = self.self( 2025-08-14T21:53:07.6320972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6321068Z self.key(key_tensor) 2025-08-14T21:53:07.6321071Z 2025-08-14T21:53:07.6321149Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6321226Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6321332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6321542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6321616Z return mod(**inputs) 2025-08-14T21:53:07.6321896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6321967Z outputs = self.mobilebert( 2025-08-14T21:53:07.6322245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6322317Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6322596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6322675Z layer_outputs = layer_module( 2025-08-14T21:53:07.6322957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6323047Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6323333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6323466Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6323767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6323855Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6323858Z 2025-08-14T21:53:07.6323972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6324179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6324244Z return mod(**inputs) 2025-08-14T21:53:07.6324544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6324617Z outputs = self.mobilebert( 2025-08-14T21:53:07.6324899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6324970Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6325243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6325319Z layer_outputs = layer_module( 2025-08-14T21:53:07.6325594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6325676Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6325958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6326080Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6326361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6326485Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6326759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6326856Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6326859Z 2025-08-14T21:53:07.6326958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6327189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6327254Z return mod(**inputs) 2025-08-14T21:53:07.6327518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6327594Z outputs = self.mobilebert( 2025-08-14T21:53:07.6327879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6327951Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6328227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6328297Z layer_outputs = layer_module( 2025-08-14T21:53:07.6328575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6328669Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6328941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6329059Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6329331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6329422Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6329425Z 2025-08-14T21:53:07.6329545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6329753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6329827Z return mod(**inputs) 2025-08-14T21:53:07.6330126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6330197Z outputs = self.mobilebert( 2025-08-14T21:53:07.6330475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6330561Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6330848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6330920Z layer_outputs = layer_module( 2025-08-14T21:53:07.6331197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6331297Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6331577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6331692Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6331960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6332068Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6332073Z 2025-08-14T21:53:07.6332180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6332371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6332436Z return mod(**inputs) 2025-08-14T21:53:07.6332722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6332792Z outputs = self.mobilebert( 2025-08-14T21:53:07.6333078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6333150Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6333425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6333524Z layer_outputs = layer_module( 2025-08-14T21:53:07.6333800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6333920Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6334192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6334318Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6334597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6334683Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6334686Z 2025-08-14T21:53:07.6334793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6334988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6335054Z return mod(**inputs) 2025-08-14T21:53:07.6335344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6335415Z outputs = self.mobilebert( 2025-08-14T21:53:07.6335677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6335772Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6336048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6336125Z layer_outputs = layer_module( 2025-08-14T21:53:07.6336398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6336491Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6336789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6336915Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6337191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6337321Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6337594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6337691Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6337694Z 2025-08-14T21:53:07.6337796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6337990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6338063Z return mod(**inputs) 2025-08-14T21:53:07.6338336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6338415Z outputs = self.mobilebert( 2025-08-14T21:53:07.6338688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6338760Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6339039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6339109Z layer_outputs = layer_module( 2025-08-14T21:53:07.6339389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6339481Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6339771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6339889Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6340161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6340260Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6340263Z 2025-08-14T21:53:07.6340373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6340566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6340637Z return mod(**inputs) 2025-08-14T21:53:07.6340912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6340983Z outputs = self.mobilebert( 2025-08-14T21:53:07.6341261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6341332Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6341609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6341681Z layer_outputs = layer_module( 2025-08-14T21:53:07.6341968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6342069Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6342341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6342449Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6342734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6342844Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6342874Z 2025-08-14T21:53:07.6342984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6343177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6343242Z return mod(**inputs) 2025-08-14T21:53:07.6343520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6343589Z outputs = self.mobilebert( 2025-08-14T21:53:07.6343866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6343936Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6344211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6344291Z layer_outputs = layer_module( 2025-08-14T21:53:07.6344569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6344661Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6344941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6345065Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6345342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6345426Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6345430Z 2025-08-14T21:53:07.6345529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6345750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6345814Z return mod(**inputs) 2025-08-14T21:53:07.6346100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6346189Z outputs = self.mobilebert( 2025-08-14T21:53:07.6346463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6346545Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6346818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6346889Z layer_outputs = layer_module( 2025-08-14T21:53:07.6347171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6347262Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6347545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6347668Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6347945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6348092Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6348367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6348468Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6348471Z 2025-08-14T21:53:07.6348571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6348764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6348840Z return mod(**inputs) 2025-08-14T21:53:07.6349138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6349213Z outputs = self.mobilebert( 2025-08-14T21:53:07.6349496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6349566Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6349848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6349919Z layer_outputs = layer_module( 2025-08-14T21:53:07.6350196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6350295Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6350587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6350710Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6351005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6351093Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6351097Z 2025-08-14T21:53:07.6351210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6351414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6351490Z return mod(**inputs) 2025-08-14T21:53:07.6351789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6351861Z outputs = self.mobilebert( 2025-08-14T21:53:07.6352179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6352256Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6352554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6352656Z layer_outputs = layer_module( 2025-08-14T21:53:07.6352955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6353062Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6353365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6353479Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6353779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6353897Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6353901Z 2025-08-14T21:53:07.6354016Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6354223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6354293Z return mod(**inputs) 2025-08-14T21:53:07.6354611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6354687Z outputs = self.mobilebert( 2025-08-14T21:53:07.6354992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6355077Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6355379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6355463Z layer_outputs = layer_module( 2025-08-14T21:53:07.6355858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6355965Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6356269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6356406Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6356724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6356825Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6356829Z 2025-08-14T21:53:07.6356937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6357151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6357219Z return mod(**inputs) 2025-08-14T21:53:07.6357507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6357591Z outputs = self.mobilebert( 2025-08-14T21:53:07.6357880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6357966Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6358254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6358330Z layer_outputs = layer_module( 2025-08-14T21:53:07.6358623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6358743Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6359052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6359181Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6359491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6359627Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6359916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6360011Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6360022Z 2025-08-14T21:53:07.6360128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6360333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6360414Z return mod(**inputs) 2025-08-14T21:53:07.6360719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6360798Z outputs = self.mobilebert( 2025-08-14T21:53:07.6361099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6361179Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6362342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6362426Z layer_outputs = layer_module( 2025-08-14T21:53:07.6362730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6362865Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6363169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6363276Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6363288Z 2025-08-14T21:53:07.6363396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6363602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6363680Z return mod(**inputs) 2025-08-14T21:53:07.6363969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6364053Z outputs = self.mobilebert( 2025-08-14T21:53:07.6364326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6364394Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6364662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6364732Z layer_outputs = layer_module( 2025-08-14T21:53:07.6364996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6365119Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6365390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6365499Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6365510Z 2025-08-14T21:53:07.6365610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6365803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6365878Z return mod(**inputs) 2025-08-14T21:53:07.6366178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6366248Z outputs = self.mobilebert( 2025-08-14T21:53:07.6366534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6366622Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6366906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6366976Z layer_outputs = layer_module( 2025-08-14T21:53:07.6367249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6367413Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6367685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6367787Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6367791Z 2025-08-14T21:53:07.6367893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6368092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6368166Z return mod(**inputs) 2025-08-14T21:53:07.6368467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6368541Z outputs = self.mobilebert( 2025-08-14T21:53:07.6368820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6368891Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6369169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6369240Z layer_outputs = layer_module( 2025-08-14T21:53:07.6369533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6369698Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6369971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6370102Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6370386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6370474Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6370478Z 2025-08-14T21:53:07.6370583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6370771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6370834Z return mod(**inputs) 2025-08-14T21:53:07.6371106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6371175Z outputs = self.mobilebert( 2025-08-14T21:53:07.6371444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6371515Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6371776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6371853Z layer_outputs = layer_module( 2025-08-14T21:53:07.6372117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6372295Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6372560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6372682Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6372985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6373071Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6373075Z 2025-08-14T21:53:07.6373182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6373384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6373447Z return mod(**inputs) 2025-08-14T21:53:07.6373720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6373791Z outputs = self.mobilebert( 2025-08-14T21:53:07.6374056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6374132Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6374402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6374477Z layer_outputs = layer_module( 2025-08-14T21:53:07.6374787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6374937Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6375210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6375329Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6375615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6375733Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6376000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6376093Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6376097Z 2025-08-14T21:53:07.6376193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6376382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6376452Z return mod(**inputs) 2025-08-14T21:53:07.6376716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6376791Z outputs = self.mobilebert( 2025-08-14T21:53:07.6377057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6377125Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6377397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6377464Z layer_outputs = layer_module( 2025-08-14T21:53:07.6377738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6377891Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6378157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6378288Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6378553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6378635Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6378644Z 2025-08-14T21:53:07.6378759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6378946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6379018Z return mod(**inputs) 2025-08-14T21:53:07.6379280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6379349Z outputs = self.mobilebert( 2025-08-14T21:53:07.6379619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6379688Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6379957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6380028Z layer_outputs = layer_module( 2025-08-14T21:53:07.6380290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6380382Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6380665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6380737Z self_outputs = self.self( 2025-08-14T21:53:07.6381013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6381080Z self.value(value_tensor) 2025-08-14T21:53:07.6381084Z 2025-08-14T21:53:07.6381188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6381381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6381443Z return mod(**inputs) 2025-08-14T21:53:07.6381735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6381807Z outputs = self.mobilebert( 2025-08-14T21:53:07.6382076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6382145Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6382421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6382498Z layer_outputs = layer_module( 2025-08-14T21:53:07.6382769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6382937Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6383209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6383316Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6383590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6383669Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6383674Z 2025-08-14T21:53:07.6383772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6383968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6384034Z return mod(**inputs) 2025-08-14T21:53:07.6384304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6384390Z outputs = self.mobilebert( 2025-08-14T21:53:07.6384654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6384731Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6385013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6385085Z layer_outputs = layer_module( 2025-08-14T21:53:07.6385359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6385510Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6385788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6385893Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6386166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6386258Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6386528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6386624Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6386643Z 2025-08-14T21:53:07.6386743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6386931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6387002Z return mod(**inputs) 2025-08-14T21:53:07.6387268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6387347Z outputs = self.mobilebert( 2025-08-14T21:53:07.6387632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6387703Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6387981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6388049Z layer_outputs = layer_module( 2025-08-14T21:53:07.6388317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6388411Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6388682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6388760Z self_outputs = self.self( 2025-08-14T21:53:07.6389033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6389103Z self.query(query_tensor) 2025-08-14T21:53:07.6389107Z 2025-08-14T21:53:07.6389217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6389414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6389480Z return mod(**inputs) 2025-08-14T21:53:07.6389763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6389829Z outputs = self.mobilebert( 2025-08-14T21:53:07.6390104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6390172Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6390437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6390533Z layer_outputs = layer_module( 2025-08-14T21:53:07.6390807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6390932Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6391219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6391294Z self_outputs = self.self( 2025-08-14T21:53:07.6391587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6391656Z self.key(key_tensor) 2025-08-14T21:53:07.6391659Z 2025-08-14T21:53:07.6391745Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6391832Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6391939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6392150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6392219Z return mod(**inputs) 2025-08-14T21:53:07.6392504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6392585Z outputs = self.mobilebert( 2025-08-14T21:53:07.6392888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6392964Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6393257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6393331Z layer_outputs = layer_module( 2025-08-14T21:53:07.6393623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6393711Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6394012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6394150Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6394439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6394538Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6394541Z 2025-08-14T21:53:07.6394647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6394851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6394926Z return mod(**inputs) 2025-08-14T21:53:07.6395215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6395289Z outputs = self.mobilebert( 2025-08-14T21:53:07.6395586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6395662Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6396062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6396144Z layer_outputs = layer_module( 2025-08-14T21:53:07.6396445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6396544Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6396839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6397007Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6397309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6397445Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6397780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6397877Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6397882Z 2025-08-14T21:53:07.6397998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6398204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6398274Z return mod(**inputs) 2025-08-14T21:53:07.6398596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6398673Z outputs = self.mobilebert( 2025-08-14T21:53:07.6398960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6399047Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6399337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6399420Z layer_outputs = layer_module( 2025-08-14T21:53:07.6399721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6399822Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6400120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6400239Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6400539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6400645Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6400650Z 2025-08-14T21:53:07.6400756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6400969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6401038Z return mod(**inputs) 2025-08-14T21:53:07.6401325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6401411Z outputs = self.mobilebert( 2025-08-14T21:53:07.6401697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6401781Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6402070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6402144Z layer_outputs = layer_module( 2025-08-14T21:53:07.6402441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6402541Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6402838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6402954Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6403240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6403367Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6403371Z 2025-08-14T21:53:07.6403475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6403701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6403778Z return mod(**inputs) 2025-08-14T21:53:07.6404097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6404197Z outputs = self.mobilebert( 2025-08-14T21:53:07.6404493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6404572Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6404875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6404951Z layer_outputs = layer_module( 2025-08-14T21:53:07.6405252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6405351Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6405648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6405790Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6406087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6406193Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6406205Z 2025-08-14T21:53:07.6406313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6406518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6406594Z return mod(**inputs) 2025-08-14T21:53:07.6406883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6406958Z outputs = self.mobilebert( 2025-08-14T21:53:07.6407267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6407344Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6407641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6407715Z layer_outputs = layer_module( 2025-08-14T21:53:07.6408039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6408145Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6408448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6408581Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6409072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6409209Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6409515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6409615Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6409619Z 2025-08-14T21:53:07.6409730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6409953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6410024Z return mod(**inputs) 2025-08-14T21:53:07.6410339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6410464Z outputs = self.mobilebert( 2025-08-14T21:53:07.6410784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6410872Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6411185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6411287Z layer_outputs = layer_module( 2025-08-14T21:53:07.6411602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6411702Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6412013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6412134Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6412442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6412541Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6412546Z 2025-08-14T21:53:07.6412656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6412876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6412947Z return mod(**inputs) 2025-08-14T21:53:07.6413284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6413372Z outputs = self.mobilebert( 2025-08-14T21:53:07.6413686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6413763Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6414079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6414157Z layer_outputs = layer_module( 2025-08-14T21:53:07.6414498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6414600Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6414904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6415032Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6415339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6415467Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6415471Z 2025-08-14T21:53:07.6415578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6415808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6415886Z return mod(**inputs) 2025-08-14T21:53:07.6416194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6416280Z outputs = self.mobilebert( 2025-08-14T21:53:07.6416589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6416668Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6416973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6417049Z layer_outputs = layer_module( 2025-08-14T21:53:07.6417415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6417538Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6417806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6417930Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6418222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6418306Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6418309Z 2025-08-14T21:53:07.6418414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6418604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6418672Z return mod(**inputs) 2025-08-14T21:53:07.6418949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6419019Z outputs = self.mobilebert( 2025-08-14T21:53:07.6419306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6419377Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6419654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6419731Z layer_outputs = layer_module( 2025-08-14T21:53:07.6420024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6420125Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6420399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6420520Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6420804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6420936Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6421211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6421299Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6421302Z 2025-08-14T21:53:07.6421401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6421601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6421663Z return mod(**inputs) 2025-08-14T21:53:07.6421929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6422006Z outputs = self.mobilebert( 2025-08-14T21:53:07.6422274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6422354Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6422622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6422691Z layer_outputs = layer_module( 2025-08-14T21:53:07.6422967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6423056Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6423337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6423448Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6423744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6423837Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6423842Z 2025-08-14T21:53:07.6423941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6424162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6424230Z return mod(**inputs) 2025-08-14T21:53:07.6424507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6424588Z outputs = self.mobilebert( 2025-08-14T21:53:07.6424864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6424939Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6425226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6425302Z layer_outputs = layer_module( 2025-08-14T21:53:07.6425594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6425687Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6425954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6426085Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6426351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6426458Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6426469Z 2025-08-14T21:53:07.6426567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6426756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6426825Z return mod(**inputs) 2025-08-14T21:53:07.6427106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6427176Z outputs = self.mobilebert( 2025-08-14T21:53:07.6427446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6427514Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6427785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6427853Z layer_outputs = layer_module( 2025-08-14T21:53:07.6428114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6428212Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6428476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6428601Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6428874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6428957Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6428961Z 2025-08-14T21:53:07.6429067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6429260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6429325Z return mod(**inputs) 2025-08-14T21:53:07.6429605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6429697Z outputs = self.mobilebert( 2025-08-14T21:53:07.6429977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6430049Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6430340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6430418Z layer_outputs = layer_module( 2025-08-14T21:53:07.6430691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6430791Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6431065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6431187Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6431468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6431589Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6431864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6431963Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6431966Z 2025-08-14T21:53:07.6432083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6432290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6432359Z return mod(**inputs) 2025-08-14T21:53:07.6432648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6432733Z outputs = self.mobilebert( 2025-08-14T21:53:07.6433020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6433120Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6433412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6433488Z layer_outputs = layer_module( 2025-08-14T21:53:07.6433786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6433912Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6434202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6434299Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6434304Z 2025-08-14T21:53:07.6434411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6434623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6434695Z return mod(**inputs) 2025-08-14T21:53:07.6434985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6435069Z outputs = self.mobilebert( 2025-08-14T21:53:07.6435360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6435441Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6435785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6435868Z layer_outputs = layer_module( 2025-08-14T21:53:07.6436166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6436316Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6436612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6436770Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6436774Z 2025-08-14T21:53:07.6436882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6437099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6437168Z return mod(**inputs) 2025-08-14T21:53:07.6437457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6437541Z outputs = self.mobilebert( 2025-08-14T21:53:07.6437832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6437919Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6438207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6438285Z layer_outputs = layer_module( 2025-08-14T21:53:07.6438582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6438770Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6439062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6439170Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6439175Z 2025-08-14T21:53:07.6439282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6439495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6439564Z return mod(**inputs) 2025-08-14T21:53:07.6439884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6439970Z outputs = self.mobilebert( 2025-08-14T21:53:07.6440266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6440352Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6440642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6440716Z layer_outputs = layer_module( 2025-08-14T21:53:07.6441011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6441179Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6441478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6441607Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6441898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6442002Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6442006Z 2025-08-14T21:53:07.6442113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6442319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6442393Z return mod(**inputs) 2025-08-14T21:53:07.6442681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6442781Z outputs = self.mobilebert( 2025-08-14T21:53:07.6443078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6443152Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6443467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6443537Z layer_outputs = layer_module( 2025-08-14T21:53:07.6443815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6443971Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6444245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6444374Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6444645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6444730Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6444741Z 2025-08-14T21:53:07.6444839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6445028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6445115Z return mod(**inputs) 2025-08-14T21:53:07.6445392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6445462Z outputs = self.mobilebert( 2025-08-14T21:53:07.6445742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6445812Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6446095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6446183Z layer_outputs = layer_module( 2025-08-14T21:53:07.6446458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6446621Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6446892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6447014Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6447291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6447414Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6447692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6447783Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6447788Z 2025-08-14T21:53:07.6447891Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6448092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6448158Z return mod(**inputs) 2025-08-14T21:53:07.6448436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6448507Z outputs = self.mobilebert( 2025-08-14T21:53:07.6448778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6448878Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6449153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6449231Z layer_outputs = layer_module( 2025-08-14T21:53:07.6449505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6449681Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6449961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6450070Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6450343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6450433Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6450438Z 2025-08-14T21:53:07.6450539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6450739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6450806Z return mod(**inputs) 2025-08-14T21:53:07.6451076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6451155Z outputs = self.mobilebert( 2025-08-14T21:53:07.6451443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6451524Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6451801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6451872Z layer_outputs = layer_module( 2025-08-14T21:53:07.6452158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6452243Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6452538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6452619Z self_outputs = self.self( 2025-08-14T21:53:07.6452895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6452976Z self.value(value_tensor) 2025-08-14T21:53:07.6452979Z 2025-08-14T21:53:07.6453079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6453271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6453345Z return mod(**inputs) 2025-08-14T21:53:07.6453618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6453700Z outputs = self.mobilebert( 2025-08-14T21:53:07.6453977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6454050Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6454333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6454406Z layer_outputs = layer_module( 2025-08-14T21:53:07.6454678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6454842Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6455118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6455251Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6455522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6455603Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6455623Z 2025-08-14T21:53:07.6455733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6455927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6456000Z return mod(**inputs) 2025-08-14T21:53:07.6456273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6456344Z outputs = self.mobilebert( 2025-08-14T21:53:07.6456621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6456694Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6456970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6457050Z layer_outputs = layer_module( 2025-08-14T21:53:07.6457323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6457487Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6457774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6457883Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6458166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6458252Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6458523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6458628Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6458633Z 2025-08-14T21:53:07.6458730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6458922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6458986Z return mod(**inputs) 2025-08-14T21:53:07.6459250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6459326Z outputs = self.mobilebert( 2025-08-14T21:53:07.6459589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6459665Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6459928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6459998Z layer_outputs = layer_module( 2025-08-14T21:53:07.6460268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6460352Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6460624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6460692Z self_outputs = self.self( 2025-08-14T21:53:07.6460958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6461034Z self.query(query_tensor) 2025-08-14T21:53:07.6461037Z 2025-08-14T21:53:07.6461136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6461348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6461420Z return mod(**inputs) 2025-08-14T21:53:07.6461695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6461790Z outputs = self.mobilebert( 2025-08-14T21:53:07.6462063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6462136Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6462415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6462486Z layer_outputs = layer_module( 2025-08-14T21:53:07.6462767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6462854Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6463132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6463211Z self_outputs = self.self( 2025-08-14T21:53:07.6463486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6463551Z self.key(key_tensor) 2025-08-14T21:53:07.6463562Z 2025-08-14T21:53:07.6463671Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6463753Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6463863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6464059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6464124Z return mod(**inputs) 2025-08-14T21:53:07.6464410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6464484Z outputs = self.mobilebert( 2025-08-14T21:53:07.6464785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6464866Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6465128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6465206Z layer_outputs = layer_module( 2025-08-14T21:53:07.6465472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6465554Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6465828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6465946Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6466218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6466299Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6466304Z 2025-08-14T21:53:07.6466402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6466598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6466663Z return mod(**inputs) 2025-08-14T21:53:07.6466929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6467009Z outputs = self.mobilebert( 2025-08-14T21:53:07.6467284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6467386Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6467664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6467736Z layer_outputs = layer_module( 2025-08-14T21:53:07.6468014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6468113Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6468393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6468514Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6468786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6468920Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6469193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6469293Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6469296Z 2025-08-14T21:53:07.6469396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6469593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6469664Z return mod(**inputs) 2025-08-14T21:53:07.6469955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6470028Z outputs = self.mobilebert( 2025-08-14T21:53:07.6470310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6470382Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6470668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6470737Z layer_outputs = layer_module( 2025-08-14T21:53:07.6471025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6471130Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6471407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6471527Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6471803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6471887Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6471890Z 2025-08-14T21:53:07.6472005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6472208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6472277Z return mod(**inputs) 2025-08-14T21:53:07.6472584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6472659Z outputs = self.mobilebert( 2025-08-14T21:53:07.6472965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6473042Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6473341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6473423Z layer_outputs = layer_module( 2025-08-14T21:53:07.6473767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6473891Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6474192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6474308Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6474658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6474775Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6474779Z 2025-08-14T21:53:07.6474882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6475092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6475159Z return mod(**inputs) 2025-08-14T21:53:07.6475461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6475538Z outputs = self.mobilebert( 2025-08-14T21:53:07.6475914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6476006Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6476316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6476423Z layer_outputs = layer_module( 2025-08-14T21:53:07.6476731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6476831Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6477144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6477282Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6477615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6477714Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6477719Z 2025-08-14T21:53:07.6477826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6478055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6478127Z return mod(**inputs) 2025-08-14T21:53:07.6478427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6478509Z outputs = self.mobilebert( 2025-08-14T21:53:07.6478804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6478889Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6479185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6479262Z layer_outputs = layer_module( 2025-08-14T21:53:07.6479566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6479663Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6479960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6480099Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6480395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6480529Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6480851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6480948Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6480952Z 2025-08-14T21:53:07.6481067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6481291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6481371Z return mod(**inputs) 2025-08-14T21:53:07.6481673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6481750Z outputs = self.mobilebert( 2025-08-14T21:53:07.6482054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6482135Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6482437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6482524Z layer_outputs = layer_module( 2025-08-14T21:53:07.6482829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6482939Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6483249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6483367Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6483661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6483748Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6483751Z 2025-08-14T21:53:07.6483864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6484070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6484139Z return mod(**inputs) 2025-08-14T21:53:07.6484458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6484531Z outputs = self.mobilebert( 2025-08-14T21:53:07.6484803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6484883Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6485156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6485232Z layer_outputs = layer_module( 2025-08-14T21:53:07.6485510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6485605Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6485892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6486003Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6486284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6486397Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6486400Z 2025-08-14T21:53:07.6486501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6486702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6486767Z return mod(**inputs) 2025-08-14T21:53:07.6487048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6487135Z outputs = self.mobilebert( 2025-08-14T21:53:07.6487410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6487488Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6487787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6487857Z layer_outputs = layer_module( 2025-08-14T21:53:07.6488142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6488234Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6488516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6488640Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6488918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6489011Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6489016Z 2025-08-14T21:53:07.6489115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6489317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6489398Z return mod(**inputs) 2025-08-14T21:53:07.6489670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6489747Z outputs = self.mobilebert( 2025-08-14T21:53:07.6490026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6490097Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6490375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6490460Z layer_outputs = layer_module( 2025-08-14T21:53:07.6490743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6490836Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6491117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6491249Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6491522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6491649Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6491926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6492017Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6492021Z 2025-08-14T21:53:07.6492128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6492324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6492388Z return mod(**inputs) 2025-08-14T21:53:07.6492669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6492739Z outputs = self.mobilebert( 2025-08-14T21:53:07.6493020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6493092Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6493384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6493461Z layer_outputs = layer_module( 2025-08-14T21:53:07.6493741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6493864Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6494156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6494275Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6494575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6494665Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6494669Z 2025-08-14T21:53:07.6494786Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6495006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6495074Z return mod(**inputs) 2025-08-14T21:53:07.6495355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6495429Z outputs = self.mobilebert( 2025-08-14T21:53:07.6495717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6495798Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6496072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6496149Z layer_outputs = layer_module( 2025-08-14T21:53:07.6496435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6496534Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6496848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6496966Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6497254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6497380Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6497384Z 2025-08-14T21:53:07.6497490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6497700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6497769Z return mod(**inputs) 2025-08-14T21:53:07.6498056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6498139Z outputs = self.mobilebert( 2025-08-14T21:53:07.6498430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6498513Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6498806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6498882Z layer_outputs = layer_module( 2025-08-14T21:53:07.6499178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6499273Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6499561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6499719Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6500008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6500106Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6500109Z 2025-08-14T21:53:07.6500235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6500439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6500518Z return mod(**inputs) 2025-08-14T21:53:07.6500805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6500886Z outputs = self.mobilebert( 2025-08-14T21:53:07.6501171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6501248Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6501544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6501619Z layer_outputs = layer_module( 2025-08-14T21:53:07.6501909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6502016Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6502323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6502462Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6502745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6502872Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6503167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6503280Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6503285Z 2025-08-14T21:53:07.6503399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6503608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6503677Z return mod(**inputs) 2025-08-14T21:53:07.6503973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6504047Z outputs = self.mobilebert( 2025-08-14T21:53:07.6504343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6504420Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6504724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6504805Z layer_outputs = layer_module( 2025-08-14T21:53:07.6505100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6505228Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6505528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6505615Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6505619Z 2025-08-14T21:53:07.6505732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6505938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6506007Z return mod(**inputs) 2025-08-14T21:53:07.6506327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6506400Z outputs = self.mobilebert( 2025-08-14T21:53:07.6506710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6506805Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6507094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6507177Z layer_outputs = layer_module( 2025-08-14T21:53:07.6507468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6507592Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6507902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6508021Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6508025Z 2025-08-14T21:53:07.6508140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6508347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6508419Z return mod(**inputs) 2025-08-14T21:53:07.6508888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6509020Z outputs = self.mobilebert( 2025-08-14T21:53:07.6509322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6509398Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6509687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6509772Z layer_outputs = layer_module( 2025-08-14T21:53:07.6510090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6510259Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6510560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6510661Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6510665Z 2025-08-14T21:53:07.6510780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6510984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6511053Z return mod(**inputs) 2025-08-14T21:53:07.6511353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6511427Z outputs = self.mobilebert( 2025-08-14T21:53:07.6511724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6511801Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6512093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6512182Z layer_outputs = layer_module( 2025-08-14T21:53:07.6512480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6512650Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6512953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6513124Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6513431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6513529Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6513557Z 2025-08-14T21:53:07.6513667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6513887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6513959Z return mod(**inputs) 2025-08-14T21:53:07.6514262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6514338Z outputs = self.mobilebert( 2025-08-14T21:53:07.6514637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6514724Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6515021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6515106Z layer_outputs = layer_module( 2025-08-14T21:53:07.6515402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6515573Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6516145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6516288Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6516603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6516706Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6516712Z 2025-08-14T21:53:07.6516822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6517066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6517140Z return mod(**inputs) 2025-08-14T21:53:07.6517439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6517526Z outputs = self.mobilebert( 2025-08-14T21:53:07.6517825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6517914Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6518210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6518286Z layer_outputs = layer_module( 2025-08-14T21:53:07.6518596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6518763Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6519063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6519206Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6519509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6519646Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6519940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6520038Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6520061Z 2025-08-14T21:53:07.6520175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6520384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6520463Z return mod(**inputs) 2025-08-14T21:53:07.6520756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6520852Z outputs = self.mobilebert( 2025-08-14T21:53:07.6521161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6521238Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6521537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6521619Z layer_outputs = layer_module( 2025-08-14T21:53:07.6521920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6522103Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6522408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6522527Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6522854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6522945Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6522950Z 2025-08-14T21:53:07.6523066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6523278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6523350Z return mod(**inputs) 2025-08-14T21:53:07.6523661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6523735Z outputs = self.mobilebert( 2025-08-14T21:53:07.6524057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6524137Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6524436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6524520Z layer_outputs = layer_module( 2025-08-14T21:53:07.6524816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6524909Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6525216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6525293Z self_outputs = self.self( 2025-08-14T21:53:07.6525602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6525674Z self.value(value_tensor) 2025-08-14T21:53:07.6525679Z 2025-08-14T21:53:07.6525778Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6525981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6526048Z return mod(**inputs) 2025-08-14T21:53:07.6526329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6526398Z outputs = self.mobilebert( 2025-08-14T21:53:07.6526671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6526771Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6527050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6527122Z layer_outputs = layer_module( 2025-08-14T21:53:07.6527403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6527579Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6527858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6527965Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6528234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6528323Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6528327Z 2025-08-14T21:53:07.6528424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6528623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6528689Z return mod(**inputs) 2025-08-14T21:53:07.6528959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6529038Z outputs = self.mobilebert( 2025-08-14T21:53:07.6529326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6529399Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6529684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6529753Z layer_outputs = layer_module( 2025-08-14T21:53:07.6530038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6530212Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6530485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6530601Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6530874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6530968Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6531240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6531332Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6531337Z 2025-08-14T21:53:07.6531444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6531638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6531706Z return mod(**inputs) 2025-08-14T21:53:07.6531987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6532060Z outputs = self.mobilebert( 2025-08-14T21:53:07.6532340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6532412Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6532683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6532761Z layer_outputs = layer_module( 2025-08-14T21:53:07.6533034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6533161Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6533440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6533532Z self_outputs = self.self( 2025-08-14T21:53:07.6533817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6533890Z self.query(query_tensor) 2025-08-14T21:53:07.6533894Z 2025-08-14T21:53:07.6533994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6534194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6534259Z return mod(**inputs) 2025-08-14T21:53:07.6534543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6534614Z outputs = self.mobilebert( 2025-08-14T21:53:07.6534890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6534970Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6535246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6535331Z layer_outputs = layer_module( 2025-08-14T21:53:07.6535629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6535714Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6535991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6536062Z self_outputs = self.self( 2025-08-14T21:53:07.6536336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6536434Z self.key(key_tensor) 2025-08-14T21:53:07.6536439Z 2025-08-14T21:53:07.6536523Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6536609Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6536711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6536908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6536980Z return mod(**inputs) 2025-08-14T21:53:07.6537251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6537320Z outputs = self.mobilebert( 2025-08-14T21:53:07.6537600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6537675Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6537958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6538028Z layer_outputs = layer_module( 2025-08-14T21:53:07.6538308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6538398Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6538674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6538804Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6539079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6539181Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6539184Z 2025-08-14T21:53:07.6539292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6539487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6539552Z return mod(**inputs) 2025-08-14T21:53:07.6539849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6539920Z outputs = self.mobilebert( 2025-08-14T21:53:07.6540201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6540273Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6540544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6540623Z layer_outputs = layer_module( 2025-08-14T21:53:07.6540899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6540989Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6541261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6541382Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6541681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6541805Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6542076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6542172Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6542178Z 2025-08-14T21:53:07.6542277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6542480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6542563Z return mod(**inputs) 2025-08-14T21:53:07.6542837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6542916Z outputs = self.mobilebert( 2025-08-14T21:53:07.6543187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6543266Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6543540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6543609Z layer_outputs = layer_module( 2025-08-14T21:53:07.6543886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6543981Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6544253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6544371Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6544656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6544752Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6544756Z 2025-08-14T21:53:07.6544861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6545065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6545141Z return mod(**inputs) 2025-08-14T21:53:07.6545425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6545525Z outputs = self.mobilebert( 2025-08-14T21:53:07.6545824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6545918Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6546200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6546272Z layer_outputs = layer_module( 2025-08-14T21:53:07.6546542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6546641Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6546915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6547039Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6547329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6547446Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6547452Z 2025-08-14T21:53:07.6547564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6547766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6547860Z return mod(**inputs) 2025-08-14T21:53:07.6548154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6548228Z outputs = self.mobilebert( 2025-08-14T21:53:07.6548526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6548602Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6548903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6548994Z layer_outputs = layer_module( 2025-08-14T21:53:07.6549286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6549393Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6549686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6549815Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6550112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6550200Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6550205Z 2025-08-14T21:53:07.6550318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6550529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6550598Z return mod(**inputs) 2025-08-14T21:53:07.6550899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6550971Z outputs = self.mobilebert( 2025-08-14T21:53:07.6551266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6551342Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6551634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6551714Z layer_outputs = layer_module( 2025-08-14T21:53:07.6552024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6552123Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6552418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6552568Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6552860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6552986Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6553275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6553379Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6553385Z 2025-08-14T21:53:07.6553491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6553700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6553772Z return mod(**inputs) 2025-08-14T21:53:07.6554059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6554141Z outputs = self.mobilebert( 2025-08-14T21:53:07.6554444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6554523Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6554818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6554891Z layer_outputs = layer_module( 2025-08-14T21:53:07.6555190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6555288Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6555597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6555802Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6556103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6556202Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6556206Z 2025-08-14T21:53:07.6556314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6556521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6556599Z return mod(**inputs) 2025-08-14T21:53:07.6556889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6556966Z outputs = self.mobilebert( 2025-08-14T21:53:07.6557264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6557342Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6557642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6557720Z layer_outputs = layer_module( 2025-08-14T21:53:07.6558009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6558116Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6558406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6558555Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6558845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6558960Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6558984Z 2025-08-14T21:53:07.6559099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6559302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6559377Z return mod(**inputs) 2025-08-14T21:53:07.6559662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6559732Z outputs = self.mobilebert( 2025-08-14T21:53:07.6560012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6560086Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6560359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6560437Z layer_outputs = layer_module( 2025-08-14T21:53:07.6560712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6560811Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6561111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6561234Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6561512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6561595Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6561601Z 2025-08-14T21:53:07.6561707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6561913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6561979Z return mod(**inputs) 2025-08-14T21:53:07.6562257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6562328Z outputs = self.mobilebert( 2025-08-14T21:53:07.6562600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6562680Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6562951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6563027Z layer_outputs = layer_module( 2025-08-14T21:53:07.6563307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6563400Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6563685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6563807Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6564088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6564208Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6564479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6564578Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6564581Z 2025-08-14T21:53:07.6564701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6564898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6564973Z return mod(**inputs) 2025-08-14T21:53:07.6565248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6565343Z outputs = self.mobilebert( 2025-08-14T21:53:07.6565615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6565687Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6565966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6566037Z layer_outputs = layer_module( 2025-08-14T21:53:07.6566315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6566408Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6566681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6566800Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6567073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6567174Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6567185Z 2025-08-14T21:53:07.6567286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6567477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6567549Z return mod(**inputs) 2025-08-14T21:53:07.6567820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6567891Z outputs = self.mobilebert( 2025-08-14T21:53:07.6568186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6568259Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6568540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6568610Z layer_outputs = layer_module( 2025-08-14T21:53:07.6568883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6568981Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6569253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6569364Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6569649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6569761Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6569766Z 2025-08-14T21:53:07.6569872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6570065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6570130Z return mod(**inputs) 2025-08-14T21:53:07.6570410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6570480Z outputs = self.mobilebert( 2025-08-14T21:53:07.6570765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6570852Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6571126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6571206Z layer_outputs = layer_module( 2025-08-14T21:53:07.6571482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6571594Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6571877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6571998Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6572278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6572362Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6572367Z 2025-08-14T21:53:07.6572469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6572675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6572740Z return mod(**inputs) 2025-08-14T21:53:07.6573020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6573093Z outputs = self.mobilebert( 2025-08-14T21:53:07.6573383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6573464Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6573735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6573807Z layer_outputs = layer_module( 2025-08-14T21:53:07.6574089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6574181Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6574480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6574615Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6574880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6575002Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6575263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6575359Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6575362Z 2025-08-14T21:53:07.6575460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6575651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6575721Z return mod(**inputs) 2025-08-14T21:53:07.6575988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6576064Z outputs = self.mobilebert( 2025-08-14T21:53:07.6576329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6576400Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6576671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6576739Z layer_outputs = layer_module( 2025-08-14T21:53:07.6577003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6577169Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6577433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6577536Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6577540Z 2025-08-14T21:53:07.6577636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6577826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6577898Z return mod(**inputs) 2025-08-14T21:53:07.6578162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6578237Z outputs = self.mobilebert( 2025-08-14T21:53:07.6578502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6578573Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6578845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6578913Z layer_outputs = layer_module( 2025-08-14T21:53:07.6579176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6579314Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6579583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6579698Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6579701Z 2025-08-14T21:53:07.6579800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6579991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6580066Z return mod(**inputs) 2025-08-14T21:53:07.6580353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6580432Z outputs = self.mobilebert( 2025-08-14T21:53:07.6580708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6580778Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6581060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6581131Z layer_outputs = layer_module( 2025-08-14T21:53:07.6581398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6581561Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6581832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6581934Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6581938Z 2025-08-14T21:53:07.6582037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6582230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6582304Z return mod(**inputs) 2025-08-14T21:53:07.6582577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6582652Z outputs = self.mobilebert( 2025-08-14T21:53:07.6582919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6583011Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6583288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6583358Z layer_outputs = layer_module( 2025-08-14T21:53:07.6583626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6583809Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6584074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6584199Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6584461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6584549Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6584554Z 2025-08-14T21:53:07.6584657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6584844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6584917Z return mod(**inputs) 2025-08-14T21:53:07.6585181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6585249Z outputs = self.mobilebert( 2025-08-14T21:53:07.6585536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6585609Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6585872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6585946Z layer_outputs = layer_module( 2025-08-14T21:53:07.6586211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6586384Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6586652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6586771Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6587045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6587126Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6587129Z 2025-08-14T21:53:07.6587232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6587416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6587482Z return mod(**inputs) 2025-08-14T21:53:07.6587762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6587831Z outputs = self.mobilebert( 2025-08-14T21:53:07.6588103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6588173Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6588439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6588515Z layer_outputs = layer_module( 2025-08-14T21:53:07.6588784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6588934Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6589237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6589354Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6589627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6589763Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6590026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6590123Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6590126Z 2025-08-14T21:53:07.6590227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6590426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6590491Z return mod(**inputs) 2025-08-14T21:53:07.6590768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6590844Z outputs = self.mobilebert( 2025-08-14T21:53:07.6591121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6591197Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6591495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6591571Z layer_outputs = layer_module( 2025-08-14T21:53:07.6591870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6592039Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6592341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6592467Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6592784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6592881Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6592886Z 2025-08-14T21:53:07.6592992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6593196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6593272Z return mod(**inputs) 2025-08-14T21:53:07.6593559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6593640Z outputs = self.mobilebert( 2025-08-14T21:53:07.6593931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6594008Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6594305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6594380Z layer_outputs = layer_module( 2025-08-14T21:53:07.6594666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6594765Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6595054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6595138Z self_outputs = self.self( 2025-08-14T21:53:07.6595442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6595539Z self.value(value_tensor) 2025-08-14T21:53:07.6595543Z 2025-08-14T21:53:07.6595657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6595968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6596054Z return mod(**inputs) 2025-08-14T21:53:07.6596376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6596452Z outputs = self.mobilebert( 2025-08-14T21:53:07.6596761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6596840Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6597147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6597232Z layer_outputs = layer_module( 2025-08-14T21:53:07.6597538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6597714Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6598009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6598128Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6598449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6598537Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6598541Z 2025-08-14T21:53:07.6598657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6598861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6598934Z return mod(**inputs) 2025-08-14T21:53:07.6599235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6599322Z outputs = self.mobilebert( 2025-08-14T21:53:07.6599595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6599674Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6599946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6600025Z layer_outputs = layer_module( 2025-08-14T21:53:07.6600299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6600461Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6600745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6600855Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6601137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6601223Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6601496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6601593Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6601597Z 2025-08-14T21:53:07.6601696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6601893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6601966Z return mod(**inputs) 2025-08-14T21:53:07.6602270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6602353Z outputs = self.mobilebert( 2025-08-14T21:53:07.6602646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6602739Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6603036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6603110Z layer_outputs = layer_module( 2025-08-14T21:53:07.6603407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6603497Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6603786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6603871Z self_outputs = self.self( 2025-08-14T21:53:07.6604163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6604237Z self.query(query_tensor) 2025-08-14T21:53:07.6604249Z 2025-08-14T21:53:07.6604357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6604559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6604655Z return mod(**inputs) 2025-08-14T21:53:07.6604948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6605021Z outputs = self.mobilebert( 2025-08-14T21:53:07.6605320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6605397Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6605694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6605783Z layer_outputs = layer_module( 2025-08-14T21:53:07.6606076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6606173Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6606466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6606541Z self_outputs = self.self( 2025-08-14T21:53:07.6606836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6606905Z self.key(key_tensor) 2025-08-14T21:53:07.6606908Z 2025-08-14T21:53:07.6607003Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6607087Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6607193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6607406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6607476Z return mod(**inputs) 2025-08-14T21:53:07.6607766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6607848Z outputs = self.mobilebert( 2025-08-14T21:53:07.6608135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6608217Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6608520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6608625Z layer_outputs = layer_module( 2025-08-14T21:53:07.6609068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6609165Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6609460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6609982Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6610271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6610370Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6610374Z 2025-08-14T21:53:07.6610481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6610687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6610766Z return mod(**inputs) 2025-08-14T21:53:07.6611052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6611135Z outputs = self.mobilebert( 2025-08-14T21:53:07.6611421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6611498Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6611826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6611903Z layer_outputs = layer_module( 2025-08-14T21:53:07.6612205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6612292Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6612593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6612729Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6613053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6613187Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6613487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6613583Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6613587Z 2025-08-14T21:53:07.6613701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6613906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6613975Z return mod(**inputs) 2025-08-14T21:53:07.6614275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6614349Z outputs = self.mobilebert( 2025-08-14T21:53:07.6614646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6614725Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6615015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6615100Z layer_outputs = layer_module( 2025-08-14T21:53:07.6615390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6615489Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6615798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6615942Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6616247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6616337Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6616358Z 2025-08-14T21:53:07.6616466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6616680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6616748Z return mod(**inputs) 2025-08-14T21:53:07.6617054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6617128Z outputs = self.mobilebert( 2025-08-14T21:53:07.6617428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6617515Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6617806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6617880Z layer_outputs = layer_module( 2025-08-14T21:53:07.6618175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6618274Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6618585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6618702Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6618992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6619116Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6619120Z 2025-08-14T21:53:07.6619226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6619452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6619521Z return mod(**inputs) 2025-08-14T21:53:07.6619815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6619897Z outputs = self.mobilebert( 2025-08-14T21:53:07.6620197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6620277Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6620552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6620620Z layer_outputs = layer_module( 2025-08-14T21:53:07.6620903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6620995Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6621269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6621400Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6621674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6621763Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6621767Z 2025-08-14T21:53:07.6621866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6622059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6622149Z return mod(**inputs) 2025-08-14T21:53:07.6622431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6622509Z outputs = self.mobilebert( 2025-08-14T21:53:07.6622789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6622878Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6623161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6623232Z layer_outputs = layer_module( 2025-08-14T21:53:07.6623504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6623604Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6623879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6624016Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6624308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6624438Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6624754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6624849Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6624853Z 2025-08-14T21:53:07.6624962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6625167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6625237Z return mod(**inputs) 2025-08-14T21:53:07.6625531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6625606Z outputs = self.mobilebert( 2025-08-14T21:53:07.6625917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6626001Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6626293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6626377Z layer_outputs = layer_module( 2025-08-14T21:53:07.6626664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6626756Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6627035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6627145Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6627428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6627510Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6627516Z 2025-08-14T21:53:07.6627617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6627822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6627887Z return mod(**inputs) 2025-08-14T21:53:07.6628167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6628243Z outputs = self.mobilebert( 2025-08-14T21:53:07.6628519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6628614Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6628890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6628960Z layer_outputs = layer_module( 2025-08-14T21:53:07.6629256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6629347Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6629627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6629736Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6630007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6630131Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6630136Z 2025-08-14T21:53:07.6630244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6630462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6630532Z return mod(**inputs) 2025-08-14T21:53:07.6630821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6630907Z outputs = self.mobilebert( 2025-08-14T21:53:07.6631210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6631288Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6631583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6631656Z layer_outputs = layer_module( 2025-08-14T21:53:07.6631952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6632048Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6632360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6632501Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6632797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6632886Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6632897Z 2025-08-14T21:53:07.6633007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6633217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6633294Z return mod(**inputs) 2025-08-14T21:53:07.6633592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6633667Z outputs = self.mobilebert( 2025-08-14T21:53:07.6633974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6634052Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6634360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6634437Z layer_outputs = layer_module( 2025-08-14T21:53:07.6634734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6634841Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6635138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6635297Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6635598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6635812Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6636128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6636227Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6636231Z 2025-08-14T21:53:07.6636340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6636557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6636640Z return mod(**inputs) 2025-08-14T21:53:07.6636934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6637012Z outputs = self.mobilebert( 2025-08-14T21:53:07.6637302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6637391Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6637675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6637775Z layer_outputs = layer_module( 2025-08-14T21:53:07.6638059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6638161Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6638468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6638588Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6638903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6639005Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6639010Z 2025-08-14T21:53:07.6639119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6639336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6639408Z return mod(**inputs) 2025-08-14T21:53:07.6639702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6639786Z outputs = self.mobilebert( 2025-08-14T21:53:07.6640083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6640167Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6640472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6640551Z layer_outputs = layer_module( 2025-08-14T21:53:07.6640854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6640953Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6641253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6641380Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6641673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6641802Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6641825Z 2025-08-14T21:53:07.6641933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6642144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6642221Z return mod(**inputs) 2025-08-14T21:53:07.6642544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6642627Z outputs = self.mobilebert( 2025-08-14T21:53:07.6642929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6643005Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6643310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6643385Z layer_outputs = layer_module( 2025-08-14T21:53:07.6643687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6643792Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6644094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6644235Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6644551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6644644Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6644647Z 2025-08-14T21:53:07.6644772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6644978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6645053Z return mod(**inputs) 2025-08-14T21:53:07.6645342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6645416Z outputs = self.mobilebert( 2025-08-14T21:53:07.6645725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6645802Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6646099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6646178Z layer_outputs = layer_module( 2025-08-14T21:53:07.6646457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6646557Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6646850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6646981Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6647288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6647423Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6647710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6647801Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6647805Z 2025-08-14T21:53:07.6647906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6648112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6648178Z return mod(**inputs) 2025-08-14T21:53:07.6648466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6648557Z outputs = self.mobilebert( 2025-08-14T21:53:07.6648835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6648930Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6649203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6649274Z layer_outputs = layer_module( 2025-08-14T21:53:07.6649555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6649676Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6649957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6650040Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6650043Z 2025-08-14T21:53:07.6650144Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6650347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6650426Z return mod(**inputs) 2025-08-14T21:53:07.6650698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6650787Z outputs = self.mobilebert( 2025-08-14T21:53:07.6651055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6651133Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6651395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6651465Z layer_outputs = layer_module( 2025-08-14T21:53:07.6651747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6651886Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6652181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6652310Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6652313Z 2025-08-14T21:53:07.6652420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6652626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6652695Z return mod(**inputs) 2025-08-14T21:53:07.6652982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6653059Z outputs = self.mobilebert( 2025-08-14T21:53:07.6653337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6653421Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6653706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6653779Z layer_outputs = layer_module( 2025-08-14T21:53:07.6654055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6654215Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6654497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6654595Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6654616Z 2025-08-14T21:53:07.6654718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6654921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6654985Z return mod(**inputs) 2025-08-14T21:53:07.6655265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6655358Z outputs = self.mobilebert( 2025-08-14T21:53:07.6655654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6655738Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6656042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6656113Z layer_outputs = layer_module( 2025-08-14T21:53:07.6656404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6656561Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6656850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6656975Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6657282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6657388Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6657392Z 2025-08-14T21:53:07.6657497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6657705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6657776Z return mod(**inputs) 2025-08-14T21:53:07.6658060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6658142Z outputs = self.mobilebert( 2025-08-14T21:53:07.6658446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6658524Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6658821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6658892Z layer_outputs = layer_module( 2025-08-14T21:53:07.6659173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6659325Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6659598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6659730Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6660008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6660098Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6660102Z 2025-08-14T21:53:07.6660201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6660394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6660468Z return mod(**inputs) 2025-08-14T21:53:07.6660743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6660821Z outputs = self.mobilebert( 2025-08-14T21:53:07.6661097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6661186Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6661468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6661556Z layer_outputs = layer_module( 2025-08-14T21:53:07.6661826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6661987Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6662256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6662383Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6662653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6662775Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6663056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6663148Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6663152Z 2025-08-14T21:53:07.6663261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6663473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6663541Z return mod(**inputs) 2025-08-14T21:53:07.6663822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6663893Z outputs = self.mobilebert( 2025-08-14T21:53:07.6664166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6664246Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6664535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6664617Z layer_outputs = layer_module( 2025-08-14T21:53:07.6664895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6665056Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6665338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6665449Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6665734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6665818Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6665821Z 2025-08-14T21:53:07.6665922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6666122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6666189Z return mod(**inputs) 2025-08-14T21:53:07.6666473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6666544Z outputs = self.mobilebert( 2025-08-14T21:53:07.6666816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6666895Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6667166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6667254Z layer_outputs = layer_module( 2025-08-14T21:53:07.6667536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6667628Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6667924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6667995Z self_outputs = self.self( 2025-08-14T21:53:07.6668269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6668348Z self.value(value_tensor) 2025-08-14T21:53:07.6668352Z 2025-08-14T21:53:07.6668454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6668655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6668723Z return mod(**inputs) 2025-08-14T21:53:07.6668995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6669072Z outputs = self.mobilebert( 2025-08-14T21:53:07.6669346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6669419Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6669716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6669788Z layer_outputs = layer_module( 2025-08-14T21:53:07.6670068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6670223Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6670495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6670613Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6670911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6671006Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6671010Z 2025-08-14T21:53:07.6671115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6671324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6671399Z return mod(**inputs) 2025-08-14T21:53:07.6671685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6671759Z outputs = self.mobilebert( 2025-08-14T21:53:07.6672055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6672130Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6672425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6672500Z layer_outputs = layer_module( 2025-08-14T21:53:07.6672787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6672960Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6673246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6673364Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6673653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6673762Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6674061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6674175Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6674179Z 2025-08-14T21:53:07.6674285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6674498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6674567Z return mod(**inputs) 2025-08-14T21:53:07.6674862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6674935Z outputs = self.mobilebert( 2025-08-14T21:53:07.6675225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6675307Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6675597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6675680Z layer_outputs = layer_module( 2025-08-14T21:53:07.6676053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6676175Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6676478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6676556Z self_outputs = self.self( 2025-08-14T21:53:07.6676851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6676940Z self.query(query_tensor) 2025-08-14T21:53:07.6676945Z 2025-08-14T21:53:07.6677063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6677332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6677404Z return mod(**inputs) 2025-08-14T21:53:07.6677702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6677786Z outputs = self.mobilebert( 2025-08-14T21:53:07.6678092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6678178Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6678482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6678558Z layer_outputs = layer_module( 2025-08-14T21:53:07.6678870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6678962Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6679262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6679347Z self_outputs = self.self( 2025-08-14T21:53:07.6679655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6679733Z self.key(key_tensor) 2025-08-14T21:53:07.6679736Z 2025-08-14T21:53:07.6679825Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6679913Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6680031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6680244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6680332Z return mod(**inputs) 2025-08-14T21:53:07.6680636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6680712Z outputs = self.mobilebert( 2025-08-14T21:53:07.6681016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6681120Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6681420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6681506Z layer_outputs = layer_module( 2025-08-14T21:53:07.6681801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6681897Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6682200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6682331Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6682635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6682728Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6682732Z 2025-08-14T21:53:07.6682840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6683075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6683146Z return mod(**inputs) 2025-08-14T21:53:07.6683450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6683526Z outputs = self.mobilebert( 2025-08-14T21:53:07.6683827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6683910Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6684226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6684314Z layer_outputs = layer_module( 2025-08-14T21:53:07.6684617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6684710Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6685018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6685148Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6685451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6685595Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6685893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6686000Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6686004Z 2025-08-14T21:53:07.6686111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6686322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6686400Z return mod(**inputs) 2025-08-14T21:53:07.6686699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6686783Z outputs = self.mobilebert( 2025-08-14T21:53:07.6687080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6687187Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6687499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6687576Z layer_outputs = layer_module( 2025-08-14T21:53:07.6687891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6688003Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6688302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6688431Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6688732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6688826Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6688830Z 2025-08-14T21:53:07.6688945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6689162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6689239Z return mod(**inputs) 2025-08-14T21:53:07.6689538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6689614Z outputs = self.mobilebert( 2025-08-14T21:53:07.6689935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6690016Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6690315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6690400Z layer_outputs = layer_module( 2025-08-14T21:53:07.6690706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6690832Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6691142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6691260Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6691559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6691676Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6691680Z 2025-08-14T21:53:07.6691791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6691994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6692064Z return mod(**inputs) 2025-08-14T21:53:07.6692357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6692432Z outputs = self.mobilebert( 2025-08-14T21:53:07.6692722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6692805Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6693096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6693176Z layer_outputs = layer_module( 2025-08-14T21:53:07.6693466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6693563Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6693859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6694011Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6694307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6694414Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6694417Z 2025-08-14T21:53:07.6694521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6694734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6694803Z return mod(**inputs) 2025-08-14T21:53:07.6695099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6695172Z outputs = self.mobilebert( 2025-08-14T21:53:07.6695460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6695546Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6695867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6695943Z layer_outputs = layer_module( 2025-08-14T21:53:07.6696238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6696355Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6696655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6696783Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6697105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6697242Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6697579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6697685Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6697689Z 2025-08-14T21:53:07.6697796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6698003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6698081Z return mod(**inputs) 2025-08-14T21:53:07.6698371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6698446Z outputs = self.mobilebert( 2025-08-14T21:53:07.6698773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6698850Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6699147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6699223Z layer_outputs = layer_module( 2025-08-14T21:53:07.6699515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6699617Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6699938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6700062Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6700352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6700460Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6700464Z 2025-08-14T21:53:07.6700578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6700784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6700852Z return mod(**inputs) 2025-08-14T21:53:07.6701163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6701237Z outputs = self.mobilebert( 2025-08-14T21:53:07.6701532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6701608Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6701896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6701978Z layer_outputs = layer_module( 2025-08-14T21:53:07.6702267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6702373Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6702663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6702781Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6703096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6703214Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6703218Z 2025-08-14T21:53:07.6703330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6703533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6703601Z return mod(**inputs) 2025-08-14T21:53:07.6703898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6703988Z outputs = self.mobilebert( 2025-08-14T21:53:07.6704278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6704364Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6704655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6704739Z layer_outputs = layer_module( 2025-08-14T21:53:07.6705028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6705129Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6705426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6705560Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6705849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6705951Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6705956Z 2025-08-14T21:53:07.6706064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6706281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6706353Z return mod(**inputs) 2025-08-14T21:53:07.6706645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6706731Z outputs = self.mobilebert( 2025-08-14T21:53:07.6707027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6707137Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6707429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6707521Z layer_outputs = layer_module( 2025-08-14T21:53:07.6707815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6707913Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6708198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6708333Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6708619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6708983Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6709282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6709377Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6709383Z 2025-08-14T21:53:07.6709497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6709740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6709820Z return mod(**inputs) 2025-08-14T21:53:07.6710117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6710193Z outputs = self.mobilebert( 2025-08-14T21:53:07.6710504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6710583Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6710922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6710998Z layer_outputs = layer_module( 2025-08-14T21:53:07.6711293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6711400Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6711704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6711823Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6712136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6712226Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6712230Z 2025-08-14T21:53:07.6712349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6712563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6712635Z return mod(**inputs) 2025-08-14T21:53:07.6712947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6713024Z outputs = self.mobilebert( 2025-08-14T21:53:07.6713334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6713411Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6713710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6713794Z layer_outputs = layer_module( 2025-08-14T21:53:07.6714119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6714220Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6714525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6714671Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6714975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6715095Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6715099Z 2025-08-14T21:53:07.6715207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6715426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6715497Z return mod(**inputs) 2025-08-14T21:53:07.6715863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6715948Z outputs = self.mobilebert( 2025-08-14T21:53:07.6716250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6716339Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6716656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6716734Z layer_outputs = layer_module( 2025-08-14T21:53:07.6717042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6717142Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6717450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6717585Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6717907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6718010Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6718014Z 2025-08-14T21:53:07.6718736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6718961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6719033Z return mod(**inputs) 2025-08-14T21:53:07.6719332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6719418Z outputs = self.mobilebert( 2025-08-14T21:53:07.6719717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6719806Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6720109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6720188Z layer_outputs = layer_module( 2025-08-14T21:53:07.6720499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6720600Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6720901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6721044Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6721341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6721502Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6721805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6721903Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6721924Z 2025-08-14T21:53:07.6722043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6722256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6722334Z return mod(**inputs) 2025-08-14T21:53:07.6722634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6722710Z outputs = self.mobilebert( 2025-08-14T21:53:07.6723017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6723096Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6723383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6723462Z layer_outputs = layer_module( 2025-08-14T21:53:07.6723742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6723867Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6724190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6724276Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6724279Z 2025-08-14T21:53:07.6724388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6724582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6724657Z return mod(**inputs) 2025-08-14T21:53:07.6724953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6725025Z outputs = self.mobilebert( 2025-08-14T21:53:07.6725308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6725378Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6725654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6725732Z layer_outputs = layer_module( 2025-08-14T21:53:07.6726006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6726131Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6726409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6726520Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6726524Z 2025-08-14T21:53:07.6726635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6726831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6726901Z return mod(**inputs) 2025-08-14T21:53:07.6727177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6727246Z outputs = self.mobilebert( 2025-08-14T21:53:07.6727526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6727597Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6727893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6727969Z layer_outputs = layer_module( 2025-08-14T21:53:07.6728240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6728423Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6728694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6728787Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6728791Z 2025-08-14T21:53:07.6728900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6729093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6729167Z return mod(**inputs) 2025-08-14T21:53:07.6729439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6729507Z outputs = self.mobilebert( 2025-08-14T21:53:07.6729788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6729861Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6730149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6730228Z layer_outputs = layer_module( 2025-08-14T21:53:07.6730500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6730661Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6730931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6731053Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6731354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6731448Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6731452Z 2025-08-14T21:53:07.6731561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6731757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6731823Z return mod(**inputs) 2025-08-14T21:53:07.6732109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6732183Z outputs = self.mobilebert( 2025-08-14T21:53:07.6732466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6732543Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6732821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6732904Z layer_outputs = layer_module( 2025-08-14T21:53:07.6733181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6733342Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6733627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6733754Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6734041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6734144Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6734148Z 2025-08-14T21:53:07.6734248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6734449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6734531Z return mod(**inputs) 2025-08-14T21:53:07.6734812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6734882Z outputs = self.mobilebert( 2025-08-14T21:53:07.6735158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6735242Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6735515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6735586Z layer_outputs = layer_module( 2025-08-14T21:53:07.6735872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6736023Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6736305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6736440Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6736716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6736844Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6737119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6737218Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6737221Z 2025-08-14T21:53:07.6737338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6737536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6737612Z return mod(**inputs) 2025-08-14T21:53:07.6737888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6737968Z outputs = self.mobilebert( 2025-08-14T21:53:07.6738243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6738317Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6738606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6738677Z layer_outputs = layer_module( 2025-08-14T21:53:07.6738944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6739105Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6739380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6739498Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6739783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6739863Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6739867Z 2025-08-14T21:53:07.6739971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6740159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6740251Z return mod(**inputs) 2025-08-14T21:53:07.6740528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6740596Z outputs = self.mobilebert( 2025-08-14T21:53:07.6740882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6740952Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6741218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6741295Z layer_outputs = layer_module( 2025-08-14T21:53:07.6741561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6741651Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6741921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6741990Z self_outputs = self.self( 2025-08-14T21:53:07.6742263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6742332Z self.value(value_tensor) 2025-08-14T21:53:07.6742336Z 2025-08-14T21:53:07.6742441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6742643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6742707Z return mod(**inputs) 2025-08-14T21:53:07.6742978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6743046Z outputs = self.mobilebert( 2025-08-14T21:53:07.6743314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6743390Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6743673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6743750Z layer_outputs = layer_module( 2025-08-14T21:53:07.6744019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6744172Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6744450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6744557Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6744838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6744920Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6744924Z 2025-08-14T21:53:07.6745023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6745225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6745293Z return mod(**inputs) 2025-08-14T21:53:07.6745576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6745653Z outputs = self.mobilebert( 2025-08-14T21:53:07.6745949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6746032Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6746333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6746428Z layer_outputs = layer_module( 2025-08-14T21:53:07.6746736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6746901Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6747229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6747344Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6747644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6750025Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6750360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6751593Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6751601Z 2025-08-14T21:53:07.6751722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6751963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6752040Z return mod(**inputs) 2025-08-14T21:53:07.6752351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6752427Z outputs = self.mobilebert( 2025-08-14T21:53:07.6752726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6752811Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6753157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6753239Z layer_outputs = layer_module( 2025-08-14T21:53:07.6753563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6753656Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6753963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6754049Z self_outputs = self.self( 2025-08-14T21:53:07.6754345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6754421Z self.query(query_tensor) 2025-08-14T21:53:07.6754426Z 2025-08-14T21:53:07.6754545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6754759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6754839Z return mod(**inputs) 2025-08-14T21:53:07.6755138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6755216Z outputs = self.mobilebert( 2025-08-14T21:53:07.6755523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6755605Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6756018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6756101Z layer_outputs = layer_module( 2025-08-14T21:53:07.6756400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6756504Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6756818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6756930Z self_outputs = self.self( 2025-08-14T21:53:07.6757233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6757323Z self.key(key_tensor) 2025-08-14T21:53:07.6757327Z 2025-08-14T21:53:07.6757420Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6757506Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6757616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6757828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6757897Z return mod(**inputs) 2025-08-14T21:53:07.6758257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6758339Z outputs = self.mobilebert( 2025-08-14T21:53:07.6758650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6758735Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6759027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6759102Z layer_outputs = layer_module( 2025-08-14T21:53:07.6759399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6759487Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6759781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6759914Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6760203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6760302Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6760305Z 2025-08-14T21:53:07.6760408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6760617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6760693Z return mod(**inputs) 2025-08-14T21:53:07.6760993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6761067Z outputs = self.mobilebert( 2025-08-14T21:53:07.6761326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6761395Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6761662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6761733Z layer_outputs = layer_module( 2025-08-14T21:53:07.6762001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6762080Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6762346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6762470Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6762741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6762862Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6763135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6763246Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6763251Z 2025-08-14T21:53:07.6763359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6763555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6763639Z return mod(**inputs) 2025-08-14T21:53:07.6763920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6763991Z outputs = self.mobilebert( 2025-08-14T21:53:07.6764273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6764346Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6764654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6764737Z layer_outputs = layer_module( 2025-08-14T21:53:07.6765030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6765127Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6765423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6765541Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6765846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6765932Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6765936Z 2025-08-14T21:53:07.6766036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6766236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6766304Z return mod(**inputs) 2025-08-14T21:53:07.6766584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6766656Z outputs = self.mobilebert( 2025-08-14T21:53:07.6766929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6767011Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6767282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6767352Z layer_outputs = layer_module( 2025-08-14T21:53:07.6767639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6767733Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6768007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6768116Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6768382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6768505Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6768509Z 2025-08-14T21:53:07.6768607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6768805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6768872Z return mod(**inputs) 2025-08-14T21:53:07.6769143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6769221Z outputs = self.mobilebert( 2025-08-14T21:53:07.6769515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6769586Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6769871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6769959Z layer_outputs = layer_module( 2025-08-14T21:53:07.6770238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6770330Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6770604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6770757Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6771028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6771133Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6771137Z 2025-08-14T21:53:07.6771235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6771423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6771496Z return mod(**inputs) 2025-08-14T21:53:07.6771760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6771835Z outputs = self.mobilebert( 2025-08-14T21:53:07.6772099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6772168Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6772444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6772514Z layer_outputs = layer_module( 2025-08-14T21:53:07.6772778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6772876Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6773149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6773278Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6773549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6773668Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6773949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6774041Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6774046Z 2025-08-14T21:53:07.6774153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6774346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6774412Z return mod(**inputs) 2025-08-14T21:53:07.6774689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6774758Z outputs = self.mobilebert( 2025-08-14T21:53:07.6775030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6775109Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6775382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6775478Z layer_outputs = layer_module( 2025-08-14T21:53:07.6775760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6775852Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6776143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6776273Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6776553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6776638Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6776642Z 2025-08-14T21:53:07.6776743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6776961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6777029Z return mod(**inputs) 2025-08-14T21:53:07.6777323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6777403Z outputs = self.mobilebert( 2025-08-14T21:53:07.6777673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6777751Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6778031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6778099Z layer_outputs = layer_module( 2025-08-14T21:53:07.6778369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6778460Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6778742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6778851Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6779125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6779243Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6779247Z 2025-08-14T21:53:07.6779346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6779537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6779609Z return mod(**inputs) 2025-08-14T21:53:07.6779884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6779960Z outputs = self.mobilebert( 2025-08-14T21:53:07.6780236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6780307Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6780585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6780658Z layer_outputs = layer_module( 2025-08-14T21:53:07.6780938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6781031Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6781303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6781434Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6781703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6781810Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6781820Z 2025-08-14T21:53:07.6781919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6782115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6782219Z return mod(**inputs) 2025-08-14T21:53:07.6782510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6782583Z outputs = self.mobilebert( 2025-08-14T21:53:07.6782882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6782957Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6783303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6783389Z layer_outputs = layer_module( 2025-08-14T21:53:07.6783683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6783784Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6784064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6784191Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6784499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6784625Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6784928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6785026Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6785029Z 2025-08-14T21:53:07.6785135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6785350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6785420Z return mod(**inputs) 2025-08-14T21:53:07.6785716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6785797Z outputs = self.mobilebert( 2025-08-14T21:53:07.6786068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6786149Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6786422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6786499Z layer_outputs = layer_module( 2025-08-14T21:53:07.6786774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6786867Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6787144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6787255Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6787526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6787618Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6787621Z 2025-08-14T21:53:07.6787723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6787926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6788011Z return mod(**inputs) 2025-08-14T21:53:07.6788284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6788363Z outputs = self.mobilebert( 2025-08-14T21:53:07.6788637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6788733Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6789007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6789077Z layer_outputs = layer_module( 2025-08-14T21:53:07.6789357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6789467Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6789746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6789887Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6790162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6790279Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6790282Z 2025-08-14T21:53:07.6790380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6790572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6790643Z return mod(**inputs) 2025-08-14T21:53:07.6790917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6790993Z outputs = self.mobilebert( 2025-08-14T21:53:07.6791265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6791339Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6791618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6791688Z layer_outputs = layer_module( 2025-08-14T21:53:07.6791958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6792059Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6792328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6792456Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6792729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6792814Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6792818Z 2025-08-14T21:53:07.6792925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6793128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6793205Z return mod(**inputs) 2025-08-14T21:53:07.6793493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6793565Z outputs = self.mobilebert( 2025-08-14T21:53:07.6793861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6793937Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6794227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6794330Z layer_outputs = layer_module( 2025-08-14T21:53:07.6794620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6794724Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6795031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6795157Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6795452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6795576Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6795977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6796082Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6796086Z 2025-08-14T21:53:07.6796215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6796439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6796513Z return mod(**inputs) 2025-08-14T21:53:07.6796811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6796896Z outputs = self.mobilebert( 2025-08-14T21:53:07.6797203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6797289Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6797599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6797678Z layer_outputs = layer_module( 2025-08-14T21:53:07.6797982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6798108Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6798406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6798495Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6798499Z 2025-08-14T21:53:07.6798603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6798816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6798886Z return mod(**inputs) 2025-08-14T21:53:07.6799189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6799270Z outputs = self.mobilebert( 2025-08-14T21:53:07.6799563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6799647Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6799936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6800013Z layer_outputs = layer_module( 2025-08-14T21:53:07.6800310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6800435Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6800743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6800859Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6800880Z 2025-08-14T21:53:07.6800988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6801203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6801274Z return mod(**inputs) 2025-08-14T21:53:07.6801577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6801670Z outputs = self.mobilebert( 2025-08-14T21:53:07.6801971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6802054Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6802362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6802452Z layer_outputs = layer_module( 2025-08-14T21:53:07.6802750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6802937Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6803236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6803337Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6803341Z 2025-08-14T21:53:07.6803447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6803666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6803734Z return mod(**inputs) 2025-08-14T21:53:07.6804063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6804137Z outputs = self.mobilebert( 2025-08-14T21:53:07.6804436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6804522Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6804817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6804900Z layer_outputs = layer_module( 2025-08-14T21:53:07.6805198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6805362Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6805674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6805804Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6806098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6806204Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6806209Z 2025-08-14T21:53:07.6806316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6806531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6806600Z return mod(**inputs) 2025-08-14T21:53:07.6806894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6806978Z outputs = self.mobilebert( 2025-08-14T21:53:07.6807269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6807351Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6807647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6807753Z layer_outputs = layer_module( 2025-08-14T21:53:07.6808051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6808213Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6808528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6808792Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6809089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6809188Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6809239Z 2025-08-14T21:53:07.6809349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6809557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6809662Z return mod(**inputs) 2025-08-14T21:53:07.6809954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6810038Z outputs = self.mobilebert( 2025-08-14T21:53:07.6810325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6810401Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6810696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6810769Z layer_outputs = layer_module( 2025-08-14T21:53:07.6811070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6811232Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6811523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6811663Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6811953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6812080Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6812384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6812481Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6812485Z 2025-08-14T21:53:07.6812599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6812805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6812878Z return mod(**inputs) 2025-08-14T21:53:07.6813174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6813247Z outputs = self.mobilebert( 2025-08-14T21:53:07.6813544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6813618Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6813905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6813985Z layer_outputs = layer_module( 2025-08-14T21:53:07.6814275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6814441Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6814768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6814884Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6815207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6815294Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6815298Z 2025-08-14T21:53:07.6815404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6815626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6815691Z return mod(**inputs) 2025-08-14T21:53:07.6815986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6816063Z outputs = self.mobilebert( 2025-08-14T21:53:07.6816354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6816434Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6816710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6816795Z layer_outputs = layer_module( 2025-08-14T21:53:07.6817085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6817174Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6817481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6817558Z self_outputs = self.self( 2025-08-14T21:53:07.6817850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6817939Z self.value(value_tensor) 2025-08-14T21:53:07.6817943Z 2025-08-14T21:53:07.6818049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6818261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6818332Z return mod(**inputs) 2025-08-14T21:53:07.6818624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6818715Z outputs = self.mobilebert( 2025-08-14T21:53:07.6818986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6819059Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6819348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6819420Z layer_outputs = layer_module( 2025-08-14T21:53:07.6819692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6819845Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6820114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6820228Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6820492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6820578Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6820584Z 2025-08-14T21:53:07.6820687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6820901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6820975Z return mod(**inputs) 2025-08-14T21:53:07.6821257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6821348Z outputs = self.mobilebert( 2025-08-14T21:53:07.6821611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6821679Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6821947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6822015Z layer_outputs = layer_module( 2025-08-14T21:53:07.6822296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6822463Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6822752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6822869Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6823145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6823230Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6823514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6823605Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6823609Z 2025-08-14T21:53:07.6823719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6823915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6823982Z return mod(**inputs) 2025-08-14T21:53:07.6824264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6824335Z outputs = self.mobilebert( 2025-08-14T21:53:07.6824608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6824687Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6824961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6825038Z layer_outputs = layer_module( 2025-08-14T21:53:07.6825312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6825397Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6825683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6825755Z self_outputs = self.self( 2025-08-14T21:53:07.6826036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6826112Z self.query(query_tensor) 2025-08-14T21:53:07.6826115Z 2025-08-14T21:53:07.6826217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6826416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6826481Z return mod(**inputs) 2025-08-14T21:53:07.6826755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6826831Z outputs = self.mobilebert( 2025-08-14T21:53:07.6827104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6827201Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6827482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6827569Z layer_outputs = layer_module( 2025-08-14T21:53:07.6827844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6827925Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6828203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6828271Z self_outputs = self.self( 2025-08-14T21:53:07.6828556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6828631Z self.key(key_tensor) 2025-08-14T21:53:07.6828634Z 2025-08-14T21:53:07.6828736Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6828814Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6828919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6829111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6829181Z return mod(**inputs) 2025-08-14T21:53:07.6829458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6829528Z outputs = self.mobilebert( 2025-08-14T21:53:07.6829811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6829884Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6830157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6830239Z layer_outputs = layer_module( 2025-08-14T21:53:07.6830510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6830603Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6830874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6830995Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6831276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6831360Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6831365Z 2025-08-14T21:53:07.6831472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6831667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6831733Z return mod(**inputs) 2025-08-14T21:53:07.6832014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6832085Z outputs = self.mobilebert( 2025-08-14T21:53:07.6832356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6832435Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6832729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6832812Z layer_outputs = layer_module( 2025-08-14T21:53:07.6833110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6833220Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6833519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6833644Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6833974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6834104Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6834401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6834504Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6834508Z 2025-08-14T21:53:07.6834632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6834837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6834916Z return mod(**inputs) 2025-08-14T21:53:07.6835278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6835362Z outputs = self.mobilebert( 2025-08-14T21:53:07.6835674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6835830Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6836155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6836232Z layer_outputs = layer_module( 2025-08-14T21:53:07.6836546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6836660Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6836960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6837086Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6837383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6837470Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6837483Z 2025-08-14T21:53:07.6837584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6837778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6837851Z return mod(**inputs) 2025-08-14T21:53:07.6838126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6838197Z outputs = self.mobilebert( 2025-08-14T21:53:07.6838479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6838552Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6838833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6838906Z layer_outputs = layer_module( 2025-08-14T21:53:07.6839176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6839281Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6839557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6839668Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6839949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6840085Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6840089Z 2025-08-14T21:53:07.6840196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6840389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6840471Z return mod(**inputs) 2025-08-14T21:53:07.6840755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6840825Z outputs = self.mobilebert( 2025-08-14T21:53:07.6841114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6841203Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6841479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6841580Z layer_outputs = layer_module( 2025-08-14T21:53:07.6841854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6841946Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6842224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6842349Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6842632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6842715Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6842720Z 2025-08-14T21:53:07.6842822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6843025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6843090Z return mod(**inputs) 2025-08-14T21:53:07.6843376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6843443Z outputs = self.mobilebert( 2025-08-14T21:53:07.6843699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6843775Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6844034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6844105Z layer_outputs = layer_module( 2025-08-14T21:53:07.6844365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6844453Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6844720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6844834Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6845094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6845212Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6845469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6845561Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6845565Z 2025-08-14T21:53:07.6845661Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6845848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6845940Z return mod(**inputs) 2025-08-14T21:53:07.6846205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6846281Z outputs = self.mobilebert( 2025-08-14T21:53:07.6846563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6846632Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6846901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6846969Z layer_outputs = layer_module( 2025-08-14T21:53:07.6847248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6847345Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6847640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6847752Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6848011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6848091Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6848094Z 2025-08-14T21:53:07.6848199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6848381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6848450Z return mod(**inputs) 2025-08-14T21:53:07.6848709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6848779Z outputs = self.mobilebert( 2025-08-14T21:53:07.6849056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6849126Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6849395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6849472Z layer_outputs = layer_module( 2025-08-14T21:53:07.6849736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6849834Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6850099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6850209Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6850480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6850599Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6850603Z 2025-08-14T21:53:07.6850706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6850888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6850950Z return mod(**inputs) 2025-08-14T21:53:07.6851217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6851284Z outputs = self.mobilebert( 2025-08-14T21:53:07.6851546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6851623Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6851879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6851973Z layer_outputs = layer_module( 2025-08-14T21:53:07.6852232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6852318Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6852601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6852715Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6852981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6853060Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6853063Z 2025-08-14T21:53:07.6853180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6853372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6853452Z return mod(**inputs) 2025-08-14T21:53:07.6853711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6853784Z outputs = self.mobilebert( 2025-08-14T21:53:07.6854044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6854118Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6854375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6854441Z layer_outputs = layer_module( 2025-08-14T21:53:07.6854712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6854802Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6855069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6855185Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6855449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6855577Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6855850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6855946Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6855949Z 2025-08-14T21:53:07.6856063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6856251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6856326Z return mod(**inputs) 2025-08-14T21:53:07.6856595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6856667Z outputs = self.mobilebert( 2025-08-14T21:53:07.6856948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6857021Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6857303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6857370Z layer_outputs = layer_module( 2025-08-14T21:53:07.6857633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6857730Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6858015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6858126Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6858387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6858484Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6858488Z 2025-08-14T21:53:07.6858592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6858779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6858845Z return mod(**inputs) 2025-08-14T21:53:07.6859168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6859236Z outputs = self.mobilebert( 2025-08-14T21:53:07.6859532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6859602Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6859858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6859933Z layer_outputs = layer_module( 2025-08-14T21:53:07.6860193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6860288Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6860551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6860660Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6860929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6861038Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6861041Z 2025-08-14T21:53:07.6861137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6861332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6861397Z return mod(**inputs) 2025-08-14T21:53:07.6861668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6861738Z outputs = self.mobilebert( 2025-08-14T21:53:07.6862003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6862095Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6862352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6862429Z layer_outputs = layer_module( 2025-08-14T21:53:07.6862691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6862781Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6863053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6863172Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6863436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6863524Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6863527Z 2025-08-14T21:53:07.6863626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6863823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6864759Z return mod(**inputs) 2025-08-14T21:53:07.6865026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6865104Z outputs = self.mobilebert( 2025-08-14T21:53:07.6865389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6865469Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6865734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6865804Z layer_outputs = layer_module( 2025-08-14T21:53:07.6866092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6866186Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6866486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6866619Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6866901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6867026Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6867288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6867377Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6867380Z 2025-08-14T21:53:07.6867487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6867675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6867749Z return mod(**inputs) 2025-08-14T21:53:07.6868015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6868083Z outputs = self.mobilebert( 2025-08-14T21:53:07.6868350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6868422Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6868692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6868767Z layer_outputs = layer_module( 2025-08-14T21:53:07.6869042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6869166Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6869444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6869533Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6869536Z 2025-08-14T21:53:07.6869643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6869838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6869910Z return mod(**inputs) 2025-08-14T21:53:07.6870183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6870251Z outputs = self.mobilebert( 2025-08-14T21:53:07.6870530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6870598Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6870878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6870972Z layer_outputs = layer_module( 2025-08-14T21:53:07.6871253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6871395Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6871667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6871775Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6871778Z 2025-08-14T21:53:07.6871888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6872099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6872174Z return mod(**inputs) 2025-08-14T21:53:07.6872449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6872536Z outputs = self.mobilebert( 2025-08-14T21:53:07.6872825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6872898Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6873184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6873253Z layer_outputs = layer_module( 2025-08-14T21:53:07.6873544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6873721Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6874021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6874124Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6874136Z 2025-08-14T21:53:07.6874242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6874471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6874548Z return mod(**inputs) 2025-08-14T21:53:07.6874850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6874925Z outputs = self.mobilebert( 2025-08-14T21:53:07.6875234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6875310Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6875619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6875764Z layer_outputs = layer_module( 2025-08-14T21:53:07.6876084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6876261Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6876572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6876705Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6877022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6877122Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6877128Z 2025-08-14T21:53:07.6877245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6877485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6877551Z return mod(**inputs) 2025-08-14T21:53:07.6877827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6877895Z outputs = self.mobilebert( 2025-08-14T21:53:07.6878199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6878269Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6878541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6878619Z layer_outputs = layer_module( 2025-08-14T21:53:07.6878903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6879054Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6879340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6879460Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6879735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6879817Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6879821Z 2025-08-14T21:53:07.6879919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6880113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6880176Z return mod(**inputs) 2025-08-14T21:53:07.6880451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6880520Z outputs = self.mobilebert( 2025-08-14T21:53:07.6880789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6880867Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6881134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6881212Z layer_outputs = layer_module( 2025-08-14T21:53:07.6881480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6881628Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6881906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6882023Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6882294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6882419Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6882688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6882784Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6882787Z 2025-08-14T21:53:07.6882888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6883083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6883156Z return mod(**inputs) 2025-08-14T21:53:07.6883433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6883530Z outputs = self.mobilebert( 2025-08-14T21:53:07.6883803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6883876Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6884157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6884270Z layer_outputs = layer_module( 2025-08-14T21:53:07.6884541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6884711Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6885005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6885124Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6885414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6885498Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6885501Z 2025-08-14T21:53:07.6885611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6885806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6885879Z return mod(**inputs) 2025-08-14T21:53:07.6886161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6886230Z outputs = self.mobilebert( 2025-08-14T21:53:07.6886505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6886576Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6886861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6886938Z layer_outputs = layer_module( 2025-08-14T21:53:07.6887210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6887307Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6887580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6887653Z self_outputs = self.self( 2025-08-14T21:53:07.6887939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6888010Z self.value(value_tensor) 2025-08-14T21:53:07.6888014Z 2025-08-14T21:53:07.6888122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6888320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6888386Z return mod(**inputs) 2025-08-14T21:53:07.6888666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6888740Z outputs = self.mobilebert( 2025-08-14T21:53:07.6889013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6889091Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6889369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6889447Z layer_outputs = layer_module( 2025-08-14T21:53:07.6889725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6889900Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6890183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6890292Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6890587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6890671Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6890674Z 2025-08-14T21:53:07.6890774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6890972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6891037Z return mod(**inputs) 2025-08-14T21:53:07.6891329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6891402Z outputs = self.mobilebert( 2025-08-14T21:53:07.6891693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6891774Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6892056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6892124Z layer_outputs = layer_module( 2025-08-14T21:53:07.6892402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6892557Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6892839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6892947Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6893233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6893326Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6893603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6893700Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6893704Z 2025-08-14T21:53:07.6893803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6893998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6894067Z return mod(**inputs) 2025-08-14T21:53:07.6894341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6894413Z outputs = self.mobilebert( 2025-08-14T21:53:07.6894696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6894768Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6895047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6895120Z layer_outputs = layer_module( 2025-08-14T21:53:07.6895393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6895485Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6895760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6895838Z self_outputs = self.self( 2025-08-14T21:53:07.6896115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6896205Z self.query(query_tensor) 2025-08-14T21:53:07.6896208Z 2025-08-14T21:53:07.6896318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6896514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6896595Z return mod(**inputs) 2025-08-14T21:53:07.6896875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6896944Z outputs = self.mobilebert( 2025-08-14T21:53:07.6897220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6897307Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6897587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6897669Z layer_outputs = layer_module( 2025-08-14T21:53:07.6897962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6898056Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6898331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6898401Z self_outputs = self.self( 2025-08-14T21:53:07.6898680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6898744Z self.key(key_tensor) 2025-08-14T21:53:07.6898748Z 2025-08-14T21:53:07.6898828Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6898917Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6899020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6899225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6899291Z return mod(**inputs) 2025-08-14T21:53:07.6899562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6899642Z outputs = self.mobilebert( 2025-08-14T21:53:07.6899913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6899985Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6900262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6900332Z layer_outputs = layer_module( 2025-08-14T21:53:07.6900613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6900700Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6900973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6901104Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6901379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6901470Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6901474Z 2025-08-14T21:53:07.6901573Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6901766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6901838Z return mod(**inputs) 2025-08-14T21:53:07.6902110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6902215Z outputs = self.mobilebert( 2025-08-14T21:53:07.6902504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6902575Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6902872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6902943Z layer_outputs = layer_module( 2025-08-14T21:53:07.6903218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6903308Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6903598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6903727Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6904020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6904148Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6904432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6904529Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6904532Z 2025-08-14T21:53:07.6904632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6904835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6904900Z return mod(**inputs) 2025-08-14T21:53:07.6905185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6905258Z outputs = self.mobilebert( 2025-08-14T21:53:07.6905535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6905614Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6905888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6905970Z layer_outputs = layer_module( 2025-08-14T21:53:07.6906257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6906355Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6906660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6906779Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6907073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6907172Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6907176Z 2025-08-14T21:53:07.6907283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6907498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6907566Z return mod(**inputs) 2025-08-14T21:53:07.6907856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6907939Z outputs = self.mobilebert( 2025-08-14T21:53:07.6908227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6908311Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6908603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6908887Z layer_outputs = layer_module( 2025-08-14T21:53:07.6909192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6909340Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6909637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6909763Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6910069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6910200Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6910230Z 2025-08-14T21:53:07.6910342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6910553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6910657Z return mod(**inputs) 2025-08-14T21:53:07.6910956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6911042Z outputs = self.mobilebert( 2025-08-14T21:53:07.6911337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6911415Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6911717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6911794Z layer_outputs = layer_module( 2025-08-14T21:53:07.6912094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6912204Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6912502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6912645Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6912943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6913035Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6913038Z 2025-08-14T21:53:07.6913153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6913364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6913442Z return mod(**inputs) 2025-08-14T21:53:07.6913741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6913821Z outputs = self.mobilebert( 2025-08-14T21:53:07.6914127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6914203Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6914505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6914583Z layer_outputs = layer_module( 2025-08-14T21:53:07.6914879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6914988Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6915284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6915418Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6915808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6915950Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6916263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6916385Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6916389Z 2025-08-14T21:53:07.6916499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6916729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6916799Z return mod(**inputs) 2025-08-14T21:53:07.6917113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6917189Z outputs = self.mobilebert( 2025-08-14T21:53:07.6917500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6917586Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6917883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6917960Z layer_outputs = layer_module( 2025-08-14T21:53:07.6918263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6918362Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6918667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6918786Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6919087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6919188Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6919192Z 2025-08-14T21:53:07.6919300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6919513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6919583Z return mod(**inputs) 2025-08-14T21:53:07.6919878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6919959Z outputs = self.mobilebert( 2025-08-14T21:53:07.6920256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6920333Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6920637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6920715Z layer_outputs = layer_module( 2025-08-14T21:53:07.6921013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6921110Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6921405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6921530Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6921824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6921949Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6921954Z 2025-08-14T21:53:07.6922060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6922283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6922362Z return mod(**inputs) 2025-08-14T21:53:07.6922650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6922743Z outputs = self.mobilebert( 2025-08-14T21:53:07.6923045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6923121Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6923423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6923497Z layer_outputs = layer_module( 2025-08-14T21:53:07.6923824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6923933Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6924253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6924391Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6924679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6924766Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6924770Z 2025-08-14T21:53:07.6924884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6925087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6925155Z return mod(**inputs) 2025-08-14T21:53:07.6925449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6925524Z outputs = self.mobilebert( 2025-08-14T21:53:07.6925819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6925894Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6926182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6926267Z layer_outputs = layer_module( 2025-08-14T21:53:07.6926556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6926659Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6926949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6927076Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6927371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6927496Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6927790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6927886Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6927890Z 2025-08-14T21:53:07.6927997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6928208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6928277Z return mod(**inputs) 2025-08-14T21:53:07.6928566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6928648Z outputs = self.mobilebert( 2025-08-14T21:53:07.6928958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6929041Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6929329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6929422Z layer_outputs = layer_module( 2025-08-14T21:53:07.6929718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6929814Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6930113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6930236Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6930515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6930623Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6930628Z 2025-08-14T21:53:07.6930729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6930923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6930999Z return mod(**inputs) 2025-08-14T21:53:07.6931274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6931349Z outputs = self.mobilebert( 2025-08-14T21:53:07.6931626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6931700Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6931984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6932058Z layer_outputs = layer_module( 2025-08-14T21:53:07.6932341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6932433Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6932710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6932828Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6933106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6933215Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6933228Z 2025-08-14T21:53:07.6933328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6933524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6933601Z return mod(**inputs) 2025-08-14T21:53:07.6933879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6933950Z outputs = self.mobilebert( 2025-08-14T21:53:07.6934233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6934304Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6934588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6934659Z layer_outputs = layer_module( 2025-08-14T21:53:07.6934950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6935076Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6935375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6935512Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6935814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.6935905Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6935909Z 2025-08-14T21:53:07.6936019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6936213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6936278Z return mod(**inputs) 2025-08-14T21:53:07.6936575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6936649Z outputs = self.mobilebert( 2025-08-14T21:53:07.6936947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6937019Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6937293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6937373Z layer_outputs = layer_module( 2025-08-14T21:53:07.6937644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6937735Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6938018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.6938139Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.6938425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.6938546Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6938826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6938926Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6938929Z 2025-08-14T21:53:07.6939030Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6939231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6939296Z return mod(**inputs) 2025-08-14T21:53:07.6939572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6939652Z outputs = self.mobilebert( 2025-08-14T21:53:07.6939930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6940002Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6940287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6940363Z layer_outputs = layer_module( 2025-08-14T21:53:07.6940664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6940790Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6941093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6941189Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6941193Z 2025-08-14T21:53:07.6941297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6941536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6941613Z return mod(**inputs) 2025-08-14T21:53:07.6941883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6941978Z outputs = self.mobilebert( 2025-08-14T21:53:07.6942252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6942323Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6942604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6942675Z layer_outputs = layer_module( 2025-08-14T21:53:07.6942972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.6943108Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.6943381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.6943495Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.6943501Z 2025-08-14T21:53:07.6943601Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6943801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6943863Z return mod(**inputs) 2025-08-14T21:53:07.6944134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6944214Z outputs = self.mobilebert( 2025-08-14T21:53:07.6944483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6944562Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6944833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6944905Z layer_outputs = layer_module( 2025-08-14T21:53:07.6945187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6945353Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6945654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.6945760Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.6945763Z 2025-08-14T21:53:07.6945872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6946082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6946153Z return mod(**inputs) 2025-08-14T21:53:07.6946454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6946535Z outputs = self.mobilebert( 2025-08-14T21:53:07.6946826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6946908Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6947204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6947273Z layer_outputs = layer_module( 2025-08-14T21:53:07.6947558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6947717Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6948050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.6948187Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.6948498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6948601Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6948605Z 2025-08-14T21:53:07.6948710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6948913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6948990Z return mod(**inputs) 2025-08-14T21:53:07.6949317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6949403Z outputs = self.mobilebert( 2025-08-14T21:53:07.6949707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6949785Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6950079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6950156Z layer_outputs = layer_module( 2025-08-14T21:53:07.6950441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6950615Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6950912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6951051Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6951344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.6951433Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6951437Z 2025-08-14T21:53:07.6951555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6951764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6951839Z return mod(**inputs) 2025-08-14T21:53:07.6952128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6952202Z outputs = self.mobilebert( 2025-08-14T21:53:07.6952500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6952576Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6952866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6952949Z layer_outputs = layer_module( 2025-08-14T21:53:07.6953234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.6953404Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.6953690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.6953821Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.6954121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.6954245Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6954572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6954669Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6954673Z 2025-08-14T21:53:07.6954779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6955007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6955075Z return mod(**inputs) 2025-08-14T21:53:07.6955371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6955444Z outputs = self.mobilebert( 2025-08-14T21:53:07.6955835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6955928Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6956254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6956333Z layer_outputs = layer_module( 2025-08-14T21:53:07.6956646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6956832Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6957367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6957521Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6957885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6957999Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6958005Z 2025-08-14T21:53:07.6958138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6958434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6958568Z return mod(**inputs) 2025-08-14T21:53:07.6958927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6959038Z outputs = self.mobilebert( 2025-08-14T21:53:07.6959353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6959483Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6959785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6959907Z layer_outputs = layer_module( 2025-08-14T21:53:07.6960263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6960374Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6960699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6960795Z self_outputs = self.self( 2025-08-14T21:53:07.6961081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.6961241Z self.value(value_tensor) 2025-08-14T21:53:07.6961246Z 2025-08-14T21:53:07.6961379Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6961629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6961718Z return mod(**inputs) 2025-08-14T21:53:07.6962018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6962171Z outputs = self.mobilebert( 2025-08-14T21:53:07.6962485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6962580Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6962928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6963045Z layer_outputs = layer_module( 2025-08-14T21:53:07.6963355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6963558Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6963887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.6964048Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.6964359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.6964498Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.6964503Z 2025-08-14T21:53:07.6964615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6964852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6964982Z return mod(**inputs) 2025-08-14T21:53:07.6965305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6965430Z outputs = self.mobilebert( 2025-08-14T21:53:07.6965754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6965839Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6966207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6966304Z layer_outputs = layer_module( 2025-08-14T21:53:07.6966645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.6966843Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.6967172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.6967354Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.6967692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.6967834Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.6968179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6968310Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6968315Z 2025-08-14T21:53:07.6968457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6968712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6968810Z return mod(**inputs) 2025-08-14T21:53:07.6969147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6969245Z outputs = self.mobilebert( 2025-08-14T21:53:07.6969574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6969662Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6969996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6970133Z layer_outputs = layer_module( 2025-08-14T21:53:07.6970431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6970586Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6970883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6970964Z self_outputs = self.self( 2025-08-14T21:53:07.6971322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.6971421Z self.query(query_tensor) 2025-08-14T21:53:07.6971440Z 2025-08-14T21:53:07.6971594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6971815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6971918Z return mod(**inputs) 2025-08-14T21:53:07.6972262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6972394Z outputs = self.mobilebert( 2025-08-14T21:53:07.6972690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6972814Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6973120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6973237Z layer_outputs = layer_module( 2025-08-14T21:53:07.6973568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6973709Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6974037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.6974130Z self_outputs = self.self( 2025-08-14T21:53:07.6974452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.6974530Z self.key(key_tensor) 2025-08-14T21:53:07.6974534Z 2025-08-14T21:53:07.6974664Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6974810Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.6974933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6975152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6975268Z return mod(**inputs) 2025-08-14T21:53:07.6975551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6975713Z outputs = self.mobilebert( 2025-08-14T21:53:07.6976007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6976102Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6976433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6976524Z layer_outputs = layer_module( 2025-08-14T21:53:07.6976856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6976994Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6977289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6977479Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6977780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.6977910Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.6977914Z 2025-08-14T21:53:07.6978078Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6978304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6978419Z return mod(**inputs) 2025-08-14T21:53:07.6978721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6978823Z outputs = self.mobilebert( 2025-08-14T21:53:07.6979155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6979276Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6979631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6979732Z layer_outputs = layer_module( 2025-08-14T21:53:07.6980029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.6980167Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.6980453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.6980657Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.6980954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.6981108Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.6981439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.6981582Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.6981586Z 2025-08-14T21:53:07.6994986Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6995339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6995420Z return mod(**inputs) 2025-08-14T21:53:07.6995876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6995974Z outputs = self.mobilebert( 2025-08-14T21:53:07.6996307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6996399Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6996706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6996804Z layer_outputs = layer_module( 2025-08-14T21:53:07.6997107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.6997239Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.6997539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.6997666Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.6997966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.6998065Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.6998072Z 2025-08-14T21:53:07.6998193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.6998524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.6998599Z return mod(**inputs) 2025-08-14T21:53:07.6998904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.6999021Z outputs = self.mobilebert( 2025-08-14T21:53:07.6999315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.6999406Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.6999700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.6999788Z layer_outputs = layer_module( 2025-08-14T21:53:07.7000109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7000220Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7000546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7000668Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7000959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7001087Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7001092Z 2025-08-14T21:53:07.7001204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7001424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7001492Z return mod(**inputs) 2025-08-14T21:53:07.7001780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7001862Z outputs = self.mobilebert( 2025-08-14T21:53:07.7002150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7002229Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7002516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7002588Z layer_outputs = layer_module( 2025-08-14T21:53:07.7002875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7002974Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7003276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7003412Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7003701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7003792Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7003796Z 2025-08-14T21:53:07.7003903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7004116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7004186Z return mod(**inputs) 2025-08-14T21:53:07.7004483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7004560Z outputs = self.mobilebert( 2025-08-14T21:53:07.7004859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7004932Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7005248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7005321Z layer_outputs = layer_module( 2025-08-14T21:53:07.7005611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7005727Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7006009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7006141Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7006478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7006603Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7006915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7007013Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7007017Z 2025-08-14T21:53:07.7007126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7007336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7007401Z return mod(**inputs) 2025-08-14T21:53:07.7007701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7007771Z outputs = self.mobilebert( 2025-08-14T21:53:07.7008071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7008147Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7008430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7008508Z layer_outputs = layer_module( 2025-08-14T21:53:07.7008990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7009096Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7009390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7009505Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7009806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7009895Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7009899Z 2025-08-14T21:53:07.7010003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7010212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7010277Z return mod(**inputs) 2025-08-14T21:53:07.7010568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7010643Z outputs = self.mobilebert( 2025-08-14T21:53:07.7010930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7011005Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7011302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7011373Z layer_outputs = layer_module( 2025-08-14T21:53:07.7011674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7011831Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7012129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7012241Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7012558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7012670Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7012674Z 2025-08-14T21:53:07.7012781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7012999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7013071Z return mod(**inputs) 2025-08-14T21:53:07.7013388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7013479Z outputs = self.mobilebert( 2025-08-14T21:53:07.7013801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7013882Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7014181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7014257Z layer_outputs = layer_module( 2025-08-14T21:53:07.7014556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7014654Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7014946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7015079Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7015356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7015449Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7015452Z 2025-08-14T21:53:07.7015564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7015756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7015828Z return mod(**inputs) 2025-08-14T21:53:07.7016104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7016171Z outputs = self.mobilebert( 2025-08-14T21:53:07.7016438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7016509Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7016778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7016845Z layer_outputs = layer_module( 2025-08-14T21:53:07.7017104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7017202Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7017469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7017600Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7017873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7017996Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7018296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7018391Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7018395Z 2025-08-14T21:53:07.7018502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7018722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7018789Z return mod(**inputs) 2025-08-14T21:53:07.7019068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7019140Z outputs = self.mobilebert( 2025-08-14T21:53:07.7019413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7019507Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7019770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7019862Z layer_outputs = layer_module( 2025-08-14T21:53:07.7020122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7020209Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7020475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7020579Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7020838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7020925Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7020929Z 2025-08-14T21:53:07.7021025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7021218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7021282Z return mod(**inputs) 2025-08-14T21:53:07.7021540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7021620Z outputs = self.mobilebert( 2025-08-14T21:53:07.7021876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7021953Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7022207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7022273Z layer_outputs = layer_module( 2025-08-14T21:53:07.7022540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7022629Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7022891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7023005Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7023266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7023381Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7023385Z 2025-08-14T21:53:07.7023481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7023664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7023736Z return mod(**inputs) 2025-08-14T21:53:07.7023991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7024083Z outputs = self.mobilebert( 2025-08-14T21:53:07.7024345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7024411Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7024680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7024765Z layer_outputs = layer_module( 2025-08-14T21:53:07.7025024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7025117Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7025390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7025515Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7025790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7025872Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7025875Z 2025-08-14T21:53:07.7025977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7026162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7026231Z return mod(**inputs) 2025-08-14T21:53:07.7026487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7026554Z outputs = self.mobilebert( 2025-08-14T21:53:07.7026816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7026885Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7027152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7027229Z layer_outputs = layer_module( 2025-08-14T21:53:07.7027500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7027600Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7027867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7027988Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7028263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7028384Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7028661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7028757Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7028760Z 2025-08-14T21:53:07.7028870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7029068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7029134Z return mod(**inputs) 2025-08-14T21:53:07.7029402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7029473Z outputs = self.mobilebert( 2025-08-14T21:53:07.7029732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7029812Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7030073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7030159Z layer_outputs = layer_module( 2025-08-14T21:53:07.7030424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7030567Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7030840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7030920Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7030923Z 2025-08-14T21:53:07.7031020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7031215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7031294Z return mod(**inputs) 2025-08-14T21:53:07.7031568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7031655Z outputs = self.mobilebert( 2025-08-14T21:53:07.7031921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7031997Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7032265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7032333Z layer_outputs = layer_module( 2025-08-14T21:53:07.7032615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7032736Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7033019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7033132Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7033135Z 2025-08-14T21:53:07.7033237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7033438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7033504Z return mod(**inputs) 2025-08-14T21:53:07.7033782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7033852Z outputs = self.mobilebert( 2025-08-14T21:53:07.7034129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7034208Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7034510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7034585Z layer_outputs = layer_module( 2025-08-14T21:53:07.7034880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7035049Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7035348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.7035447Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.7035451Z 2025-08-14T21:53:07.7035556Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7035844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7035918Z return mod(**inputs) 2025-08-14T21:53:07.7036213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7036309Z outputs = self.mobilebert( 2025-08-14T21:53:07.7036600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7036687Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7036976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7037071Z layer_outputs = layer_module( 2025-08-14T21:53:07.7037376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7037533Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7037833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.7037957Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.7038249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7038365Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7038369Z 2025-08-14T21:53:07.7038468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7038668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7038734Z return mod(**inputs) 2025-08-14T21:53:07.7039010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7039090Z outputs = self.mobilebert( 2025-08-14T21:53:07.7039369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7039442Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7039732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7039801Z layer_outputs = layer_module( 2025-08-14T21:53:07.7040076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7040236Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7040512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7040643Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7040920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.7041013Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7041018Z 2025-08-14T21:53:07.7041120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7041318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7041402Z return mod(**inputs) 2025-08-14T21:53:07.7041666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7041744Z outputs = self.mobilebert( 2025-08-14T21:53:07.7042007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7042075Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7042350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7042419Z layer_outputs = layer_module( 2025-08-14T21:53:07.7042687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7042868Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7043140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7043287Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7043560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.7043681Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7043978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7044072Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7044077Z 2025-08-14T21:53:07.7044183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7044392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7044458Z return mod(**inputs) 2025-08-14T21:53:07.7044742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7044817Z outputs = self.mobilebert( 2025-08-14T21:53:07.7045097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7045168Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7045442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7045522Z layer_outputs = layer_module( 2025-08-14T21:53:07.7045796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7045967Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7046243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.7046355Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.7046634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.7046716Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.7046720Z 2025-08-14T21:53:07.7046820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7047025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7047094Z return mod(**inputs) 2025-08-14T21:53:07.7047387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7047464Z outputs = self.mobilebert( 2025-08-14T21:53:07.7047753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7047840Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7048129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7048210Z layer_outputs = layer_module( 2025-08-14T21:53:07.7048497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7048589Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7048884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7048980Z self_outputs = self.self( 2025-08-14T21:53:07.7049268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.7049352Z self.value(value_tensor) 2025-08-14T21:53:07.7049356Z 2025-08-14T21:53:07.7049482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7049695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7049764Z return mod(**inputs) 2025-08-14T21:53:07.7050052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7050134Z outputs = self.mobilebert( 2025-08-14T21:53:07.7050441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7050527Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7050838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7050914Z layer_outputs = layer_module( 2025-08-14T21:53:07.7051213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7051381Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7051668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.7051792Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.7052082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.7052174Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.7052180Z 2025-08-14T21:53:07.7052285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7052489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7052564Z return mod(**inputs) 2025-08-14T21:53:07.7052853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7052934Z outputs = self.mobilebert( 2025-08-14T21:53:07.7053224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7053299Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7053594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7053667Z layer_outputs = layer_module( 2025-08-14T21:53:07.7053952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7054125Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7054413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.7054539Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.7054829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.7054921Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.7055218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7055317Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7055321Z 2025-08-14T21:53:07.7055462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7055667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7055737Z return mod(**inputs) 2025-08-14T21:53:07.7056030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7056124Z outputs = self.mobilebert( 2025-08-14T21:53:07.7056414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7056497Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7056790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7056887Z layer_outputs = layer_module( 2025-08-14T21:53:07.7057175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7057291Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7057593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7057669Z self_outputs = self.self( 2025-08-14T21:53:07.7057971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.7058046Z self.query(query_tensor) 2025-08-14T21:53:07.7058049Z 2025-08-14T21:53:07.7058158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7058375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7058443Z return mod(**inputs) 2025-08-14T21:53:07.7058734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7058818Z outputs = self.mobilebert( 2025-08-14T21:53:07.7059107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7059189Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7059479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7059551Z layer_outputs = layer_module( 2025-08-14T21:53:07.7059850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7059937Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7060235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7060310Z self_outputs = self.self( 2025-08-14T21:53:07.7060600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.7060678Z self.key(key_tensor) 2025-08-14T21:53:07.7060682Z 2025-08-14T21:53:07.7060769Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.7060852Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.7060969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7061173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7061249Z return mod(**inputs) 2025-08-14T21:53:07.7061539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7061611Z outputs = self.mobilebert( 2025-08-14T21:53:07.7061908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7062004Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7062302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7062380Z layer_outputs = layer_module( 2025-08-14T21:53:07.7062656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7062764Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7063045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.7063175Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.7063490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.7063584Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7063590Z 2025-08-14T21:53:07.7063709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7063937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7064009Z return mod(**inputs) 2025-08-14T21:53:07.7064300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7064376Z outputs = self.mobilebert( 2025-08-14T21:53:07.7064661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7064744Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7065042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7065124Z layer_outputs = layer_module( 2025-08-14T21:53:07.7065420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7065513Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7065817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.7065951Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.7066254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.7066388Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7066695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7066806Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7066810Z 2025-08-14T21:53:07.7066927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7067138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7067206Z return mod(**inputs) 2025-08-14T21:53:07.7067491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7067573Z outputs = self.mobilebert( 2025-08-14T21:53:07.7067860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7067935Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7068242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7068316Z layer_outputs = layer_module( 2025-08-14T21:53:07.7068607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7068727Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7069016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7069139Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7069448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7069542Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7069546Z 2025-08-14T21:53:07.7069651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7069856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7069950Z return mod(**inputs) 2025-08-14T21:53:07.7070261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7070338Z outputs = self.mobilebert( 2025-08-14T21:53:07.7070658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7070737Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7071043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7071121Z layer_outputs = layer_module( 2025-08-14T21:53:07.7071430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7071539Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7071849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7071979Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7072280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7072400Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7072404Z 2025-08-14T21:53:07.7072523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7072737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7072806Z return mod(**inputs) 2025-08-14T21:53:07.7073119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7073195Z outputs = self.mobilebert( 2025-08-14T21:53:07.7073516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7073597Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7073897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7073984Z layer_outputs = layer_module( 2025-08-14T21:53:07.7074282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7074392Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7074695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7074830Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7075142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7075235Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7075257Z 2025-08-14T21:53:07.7075366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7075584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7075656Z return mod(**inputs) 2025-08-14T21:53:07.7076039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7076147Z outputs = self.mobilebert( 2025-08-14T21:53:07.7076451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7076539Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7076865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7076968Z layer_outputs = layer_module( 2025-08-14T21:53:07.7077258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7077381Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7077681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7077812Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7078102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7078249Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7078522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7078621Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7078625Z 2025-08-14T21:53:07.7078726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7078933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7079011Z return mod(**inputs) 2025-08-14T21:53:07.7079298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7079382Z outputs = self.mobilebert( 2025-08-14T21:53:07.7079669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7079743Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7080038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7080111Z layer_outputs = layer_module( 2025-08-14T21:53:07.7080402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7080509Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7080797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7080920Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7081212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7081299Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7081303Z 2025-08-14T21:53:07.7081416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7081622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7081700Z return mod(**inputs) 2025-08-14T21:53:07.7081986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7082078Z outputs = self.mobilebert( 2025-08-14T21:53:07.7082386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7082462Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7082770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7082850Z layer_outputs = layer_module( 2025-08-14T21:53:07.7083137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7083239Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7083555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7083672Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7083995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7084117Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7084121Z 2025-08-14T21:53:07.7084241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7084458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7084525Z return mod(**inputs) 2025-08-14T21:53:07.7084818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7084892Z outputs = self.mobilebert( 2025-08-14T21:53:07.7085193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7085270Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7085567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7085660Z layer_outputs = layer_module( 2025-08-14T21:53:07.7085947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7086045Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7086338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7086467Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7086761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7086847Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7086853Z 2025-08-14T21:53:07.7086959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7087170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7087238Z return mod(**inputs) 2025-08-14T21:53:07.7087530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7087607Z outputs = self.mobilebert( 2025-08-14T21:53:07.7087892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7087975Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7088260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7088335Z layer_outputs = layer_module( 2025-08-14T21:53:07.7088626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7088746Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7089046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7089190Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7089478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7089611Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7089898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7090015Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7090020Z 2025-08-14T21:53:07.7090128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7090348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7090429Z return mod(**inputs) 2025-08-14T21:53:07.7090722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7090800Z outputs = self.mobilebert( 2025-08-14T21:53:07.7091101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7091176Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7091475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7091549Z layer_outputs = layer_module( 2025-08-14T21:53:07.7091843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7091951Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7092258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7092388Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7092702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7092791Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7092795Z 2025-08-14T21:53:07.7092908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7093119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7093188Z return mod(**inputs) 2025-08-14T21:53:07.7093487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7093563Z outputs = self.mobilebert( 2025-08-14T21:53:07.7093866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7093942Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7094238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7094320Z layer_outputs = layer_module( 2025-08-14T21:53:07.7094620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7094724Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7095017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7095132Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7095451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7095567Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7095571Z 2025-08-14T21:53:07.7095736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7095941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7096010Z return mod(**inputs) 2025-08-14T21:53:07.7096304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7096379Z outputs = self.mobilebert( 2025-08-14T21:53:07.7096683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7096769Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7097076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7097159Z layer_outputs = layer_module( 2025-08-14T21:53:07.7097449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7097546Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7097847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7097975Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7098276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7098363Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7098368Z 2025-08-14T21:53:07.7098475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7098691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7098761Z return mod(**inputs) 2025-08-14T21:53:07.7099047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7099132Z outputs = self.mobilebert( 2025-08-14T21:53:07.7099426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7099510Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7099798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7099874Z layer_outputs = layer_module( 2025-08-14T21:53:07.7100172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7100273Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7100561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7100701Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7100989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7101124Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7101410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7101509Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7101522Z 2025-08-14T21:53:07.7101627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7101857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7101935Z return mod(**inputs) 2025-08-14T21:53:07.7102224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7102315Z outputs = self.mobilebert( 2025-08-14T21:53:07.7102620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7102696Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7103000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7103072Z layer_outputs = layer_module( 2025-08-14T21:53:07.7103394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7103528Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7103837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7103927Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7103939Z 2025-08-14T21:53:07.7104045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7104247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7104323Z return mod(**inputs) 2025-08-14T21:53:07.7104614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7104687Z outputs = self.mobilebert( 2025-08-14T21:53:07.7104983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7105060Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7105358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7105432Z layer_outputs = layer_module( 2025-08-14T21:53:07.7105722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7105854Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7106144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7106258Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7106268Z 2025-08-14T21:53:07.7106376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7106580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7106658Z return mod(**inputs) 2025-08-14T21:53:07.7106948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7107022Z outputs = self.mobilebert( 2025-08-14T21:53:07.7107319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7107395Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7107697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7107773Z layer_outputs = layer_module( 2025-08-14T21:53:07.7108065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7108240Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7108549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.7108781Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.7108797Z 2025-08-14T21:53:07.7108912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7109161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7109240Z return mod(**inputs) 2025-08-14T21:53:07.7109529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7109606Z outputs = self.mobilebert( 2025-08-14T21:53:07.7109929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7110007Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7110330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7110407Z layer_outputs = layer_module( 2025-08-14T21:53:07.7110695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7110869Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7111158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.7111296Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.7111588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7111685Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7111691Z 2025-08-14T21:53:07.7111807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7112018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7112088Z return mod(**inputs) 2025-08-14T21:53:07.7112392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7112473Z outputs = self.mobilebert( 2025-08-14T21:53:07.7112775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7112862Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7113158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7113244Z layer_outputs = layer_module( 2025-08-14T21:53:07.7113543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7113720Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7114017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7114149Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7114449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.7114540Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7114544Z 2025-08-14T21:53:07.7114652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7114869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7114942Z return mod(**inputs) 2025-08-14T21:53:07.7115286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7115363Z outputs = self.mobilebert( 2025-08-14T21:53:07.7115664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7115827Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7116129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7116215Z layer_outputs = layer_module( 2025-08-14T21:53:07.7116512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7116705Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7117010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7117163Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7117460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.7117599Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7117904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7118013Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7118017Z 2025-08-14T21:53:07.7118128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7118344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7118427Z return mod(**inputs) 2025-08-14T21:53:07.7118732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7118820Z outputs = self.mobilebert( 2025-08-14T21:53:07.7119125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7119206Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7119521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7119598Z layer_outputs = layer_module( 2025-08-14T21:53:07.7119903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7120086Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7120392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.7120523Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.7120830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.7120918Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.7120929Z 2025-08-14T21:53:07.7121037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7121250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7121327Z return mod(**inputs) 2025-08-14T21:53:07.7121630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7121708Z outputs = self.mobilebert( 2025-08-14T21:53:07.7122017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7122113Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7122420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7122495Z layer_outputs = layer_module( 2025-08-14T21:53:07.7122798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7122891Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7123162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7123236Z self_outputs = self.self( 2025-08-14T21:53:07.7123547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.7123624Z self.value(value_tensor) 2025-08-14T21:53:07.7123629Z 2025-08-14T21:53:07.7123743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7123961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7124032Z return mod(**inputs) 2025-08-14T21:53:07.7124328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7124402Z outputs = self.mobilebert( 2025-08-14T21:53:07.7124696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7124774Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7125060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7125150Z layer_outputs = layer_module( 2025-08-14T21:53:07.7125423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7125582Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7125863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.7125973Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.7126250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.7126332Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.7126336Z 2025-08-14T21:53:07.7126436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7126637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7126703Z return mod(**inputs) 2025-08-14T21:53:07.7126986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7127058Z outputs = self.mobilebert( 2025-08-14T21:53:07.7127329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7127410Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7127682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7127755Z layer_outputs = layer_module( 2025-08-14T21:53:07.7128033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7128189Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7128466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.7128595Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.7128869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.7128991Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.7129264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7129362Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7129366Z 2025-08-14T21:53:07.7129466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7129674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7129750Z return mod(**inputs) 2025-08-14T21:53:07.7130028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7130807Z outputs = self.mobilebert( 2025-08-14T21:53:07.7131101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7131174Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7131453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7131523Z layer_outputs = layer_module( 2025-08-14T21:53:07.7131798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7131892Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7132171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7132251Z self_outputs = self.self( 2025-08-14T21:53:07.7132524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.7132594Z self.query(query_tensor) 2025-08-14T21:53:07.7132597Z 2025-08-14T21:53:07.7132710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7132903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7132968Z return mod(**inputs) 2025-08-14T21:53:07.7133249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7133321Z outputs = self.mobilebert( 2025-08-14T21:53:07.7133612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7133683Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7133953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7134032Z layer_outputs = layer_module( 2025-08-14T21:53:07.7134315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7134415Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7134705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7134778Z self_outputs = self.self( 2025-08-14T21:53:07.7135075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.7135146Z self.key(key_tensor) 2025-08-14T21:53:07.7135150Z 2025-08-14T21:53:07.7135237Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.7135350Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.7135459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7135669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7135737Z return mod(**inputs) 2025-08-14T21:53:07.7136055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7136132Z outputs = self.mobilebert( 2025-08-14T21:53:07.7136396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7136466Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7136753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7136825Z layer_outputs = layer_module( 2025-08-14T21:53:07.7137109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7137194Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7137458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.7137586Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.7137849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.7137937Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7137940Z 2025-08-14T21:53:07.7138036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7138222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7138293Z return mod(**inputs) 2025-08-14T21:53:07.7138556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7138625Z outputs = self.mobilebert( 2025-08-14T21:53:07.7138895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7138965Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7139233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7139301Z layer_outputs = layer_module( 2025-08-14T21:53:07.7139562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7139652Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7139916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.7140045Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.7140311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.7140432Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7140705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7140794Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7140797Z 2025-08-14T21:53:07.7140903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7141090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7141157Z return mod(**inputs) 2025-08-14T21:53:07.7141434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7141522Z outputs = self.mobilebert( 2025-08-14T21:53:07.7141788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7141883Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7142151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7142227Z layer_outputs = layer_module( 2025-08-14T21:53:07.7142494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7142587Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7142881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7142996Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7143286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7143371Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7143376Z 2025-08-14T21:53:07.7143474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7143674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7143737Z return mod(**inputs) 2025-08-14T21:53:07.7144012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7144091Z outputs = self.mobilebert( 2025-08-14T21:53:07.7144368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7144447Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7144724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7144795Z layer_outputs = layer_module( 2025-08-14T21:53:07.7145080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7145177Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7145483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7145598Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7145896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7146015Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7146020Z 2025-08-14T21:53:07.7146120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7146319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7146393Z return mod(**inputs) 2025-08-14T21:53:07.7146668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7146746Z outputs = self.mobilebert( 2025-08-14T21:53:07.7147022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7147094Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7147393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7147466Z layer_outputs = layer_module( 2025-08-14T21:53:07.7147773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7147885Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7148161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7148310Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7148582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7148664Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7148675Z 2025-08-14T21:53:07.7148772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7148978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7149056Z return mod(**inputs) 2025-08-14T21:53:07.7149374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7149451Z outputs = self.mobilebert( 2025-08-14T21:53:07.7149750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7149826Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7150133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7150206Z layer_outputs = layer_module( 2025-08-14T21:53:07.7150505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7150611Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7150907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7151041Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7151336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7151464Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7151766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7151864Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7151868Z 2025-08-14T21:53:07.7151973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7152188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7152260Z return mod(**inputs) 2025-08-14T21:53:07.7152554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7152632Z outputs = self.mobilebert( 2025-08-14T21:53:07.7152925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7153010Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7153304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7153377Z layer_outputs = layer_module( 2025-08-14T21:53:07.7153687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7153784Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7154089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7154232Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7154520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7154614Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7154636Z 2025-08-14T21:53:07.7154743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7154952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7155019Z return mod(**inputs) 2025-08-14T21:53:07.7155305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7155385Z outputs = self.mobilebert( 2025-08-14T21:53:07.7155947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7156043Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7156359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7156435Z layer_outputs = layer_module( 2025-08-14T21:53:07.7156739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7156840Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7157136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7157263Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7157571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7157698Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7157704Z 2025-08-14T21:53:07.7157813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7158036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7158112Z return mod(**inputs) 2025-08-14T21:53:07.7158388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7158470Z outputs = self.mobilebert( 2025-08-14T21:53:07.7158740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7158812Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7159099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7159170Z layer_outputs = layer_module( 2025-08-14T21:53:07.7159441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7159544Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7159814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7159946Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7160216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7160299Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7160302Z 2025-08-14T21:53:07.7160413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7160610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7160683Z return mod(**inputs) 2025-08-14T21:53:07.7160976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7161045Z outputs = self.mobilebert( 2025-08-14T21:53:07.7161325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7161413Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7161684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7161764Z layer_outputs = layer_module( 2025-08-14T21:53:07.7162033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7162131Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7162415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7162556Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7162841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7162966Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7163262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7163358Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7163362Z 2025-08-14T21:53:07.7163467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7163682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7163754Z return mod(**inputs) 2025-08-14T21:53:07.7164040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7164125Z outputs = self.mobilebert( 2025-08-14T21:53:07.7164412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7164495Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7164782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7164863Z layer_outputs = layer_module( 2025-08-14T21:53:07.7165141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7165231Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7165512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7165625Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7165898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7165990Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7165994Z 2025-08-14T21:53:07.7166096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7166293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7166366Z return mod(**inputs) 2025-08-14T21:53:07.7166640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7166722Z outputs = self.mobilebert( 2025-08-14T21:53:07.7167009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7167103Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7167401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7167475Z layer_outputs = layer_module( 2025-08-14T21:53:07.7167774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7167889Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7168176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7168303Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7168617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7168728Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7168740Z 2025-08-14T21:53:07.7168839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7169053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7169126Z return mod(**inputs) 2025-08-14T21:53:07.7169404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7169475Z outputs = self.mobilebert( 2025-08-14T21:53:07.7169772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7169847Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7170149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7170224Z layer_outputs = layer_module( 2025-08-14T21:53:07.7170519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7170628Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7170926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7171057Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7171358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7171446Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7171450Z 2025-08-14T21:53:07.7171563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7171770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7171840Z return mod(**inputs) 2025-08-14T21:53:07.7172140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7172219Z outputs = self.mobilebert( 2025-08-14T21:53:07.7172520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7172596Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7172890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7172973Z layer_outputs = layer_module( 2025-08-14T21:53:07.7173274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7173370Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7173680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7173829Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7174127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7174253Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7174563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7174668Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7174672Z 2025-08-14T21:53:07.7174776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7174990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7175075Z return mod(**inputs) 2025-08-14T21:53:07.7175371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7175458Z outputs = self.mobilebert( 2025-08-14T21:53:07.7175774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7175860Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7176154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7176228Z layer_outputs = layer_module( 2025-08-14T21:53:07.7176578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7176705Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7177012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7177109Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7177113Z 2025-08-14T21:53:07.7177221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7177437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7177506Z return mod(**inputs) 2025-08-14T21:53:07.7177805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7177889Z outputs = self.mobilebert( 2025-08-14T21:53:07.7178190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7178272Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7178575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7178650Z layer_outputs = layer_module( 2025-08-14T21:53:07.7178952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7179077Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7179367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7179492Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7179496Z 2025-08-14T21:53:07.7179602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7179823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7179891Z return mod(**inputs) 2025-08-14T21:53:07.7180189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7180280Z outputs = self.mobilebert( 2025-08-14T21:53:07.7180593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7180676Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7180965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7181061Z layer_outputs = layer_module( 2025-08-14T21:53:07.7181355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7181522Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7181832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.7181942Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.7181948Z 2025-08-14T21:53:07.7182055Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7182285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7182355Z return mod(**inputs) 2025-08-14T21:53:07.7182643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7182727Z outputs = self.mobilebert( 2025-08-14T21:53:07.7183018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7183101Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7183390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7183465Z layer_outputs = layer_module( 2025-08-14T21:53:07.7183759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7183926Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7184213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.7184353Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.7184640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7184753Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7184756Z 2025-08-14T21:53:07.7184857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7185055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7185129Z return mod(**inputs) 2025-08-14T21:53:07.7185401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7185479Z outputs = self.mobilebert( 2025-08-14T21:53:07.7185748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7185820Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7186111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7186185Z layer_outputs = layer_module( 2025-08-14T21:53:07.7186470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7186641Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7186929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7187087Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7187375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.7187482Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7187493Z 2025-08-14T21:53:07.7187599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7187804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7187879Z return mod(**inputs) 2025-08-14T21:53:07.7188164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7188275Z outputs = self.mobilebert( 2025-08-14T21:53:07.7188580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7188677Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7188957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7189027Z layer_outputs = layer_module( 2025-08-14T21:53:07.7189303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7189465Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7189743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7189867Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7190152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.7190281Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7190581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7190680Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7190686Z 2025-08-14T21:53:07.7190796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7191014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7191085Z return mod(**inputs) 2025-08-14T21:53:07.7191388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7191469Z outputs = self.mobilebert( 2025-08-14T21:53:07.7191766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7191855Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7192149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7192227Z layer_outputs = layer_module( 2025-08-14T21:53:07.7192529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7192700Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7193001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.7193119Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.7193412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.7193528Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.7193532Z 2025-08-14T21:53:07.7193640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7193854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7193941Z return mod(**inputs) 2025-08-14T21:53:07.7194232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7194316Z outputs = self.mobilebert( 2025-08-14T21:53:07.7194616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7194699Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7195014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7195090Z layer_outputs = layer_module( 2025-08-14T21:53:07.7195402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7195495Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7195876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7195970Z self_outputs = self.self( 2025-08-14T21:53:07.7196262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.7196347Z self.value(value_tensor) 2025-08-14T21:53:07.7196351Z 2025-08-14T21:53:07.7196461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7196674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7196754Z return mod(**inputs) 2025-08-14T21:53:07.7197067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7197152Z outputs = self.mobilebert( 2025-08-14T21:53:07.7197443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7197522Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7197823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7197899Z layer_outputs = layer_module( 2025-08-14T21:53:07.7198202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7198381Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7198672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.7198801Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.7199093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.7199183Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.7199187Z 2025-08-14T21:53:07.7199303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7199506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7199586Z return mod(**inputs) 2025-08-14T21:53:07.7199874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7199950Z outputs = self.mobilebert( 2025-08-14T21:53:07.7200247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7200373Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7200661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7200761Z layer_outputs = layer_module( 2025-08-14T21:53:07.7201049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7201221Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7201510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.7201651Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.7201949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.7202059Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.7202357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7202454Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7202460Z 2025-08-14T21:53:07.7202565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7202780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7202852Z return mod(**inputs) 2025-08-14T21:53:07.7203142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7203225Z outputs = self.mobilebert( 2025-08-14T21:53:07.7203517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7203600Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7203893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7203966Z layer_outputs = layer_module( 2025-08-14T21:53:07.7204270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7204360Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7204657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7204732Z self_outputs = self.self( 2025-08-14T21:53:07.7205021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.7205102Z self.query(query_tensor) 2025-08-14T21:53:07.7205108Z 2025-08-14T21:53:07.7205214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7205418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7205495Z return mod(**inputs) 2025-08-14T21:53:07.7205782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7205863Z outputs = self.mobilebert( 2025-08-14T21:53:07.7206155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7206229Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7206529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7206603Z layer_outputs = layer_module( 2025-08-14T21:53:07.7206897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7207004Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7207290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7207389Z self_outputs = self.self( 2025-08-14T21:53:07.7207676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.7207743Z self.key(key_tensor) 2025-08-14T21:53:07.7207747Z 2025-08-14T21:53:07.7207840Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.7207923Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.7208037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7208257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7208329Z return mod(**inputs) 2025-08-14T21:53:07.7208828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7208918Z outputs = self.mobilebert( 2025-08-14T21:53:07.7209212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7209301Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7209589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7209675Z layer_outputs = layer_module( 2025-08-14T21:53:07.7209961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7210052Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7210350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.7210483Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.7210784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.7210873Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7210877Z 2025-08-14T21:53:07.7210984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7211198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7211267Z return mod(**inputs) 2025-08-14T21:53:07.7211555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7211642Z outputs = self.mobilebert( 2025-08-14T21:53:07.7211930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7212018Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7212312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7212389Z layer_outputs = layer_module( 2025-08-14T21:53:07.7212692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7212780Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7213091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.7213219Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.7213521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.7213693Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7213982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7214081Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7214123Z 2025-08-14T21:53:07.7214229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7214436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7214512Z return mod(**inputs) 2025-08-14T21:53:07.7214814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7214889Z outputs = self.mobilebert( 2025-08-14T21:53:07.7215210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7215291Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7215590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7215663Z layer_outputs = layer_module( 2025-08-14T21:53:07.7215929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7216031Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7216294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7216404Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7216688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7216769Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7216774Z 2025-08-14T21:53:07.7216883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7217076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7217139Z return mod(**inputs) 2025-08-14T21:53:07.7217418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7217488Z outputs = self.mobilebert( 2025-08-14T21:53:07.7217771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7217842Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7218113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7218194Z layer_outputs = layer_module( 2025-08-14T21:53:07.7218468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7218561Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7218844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7218955Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7219247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7219355Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7219359Z 2025-08-14T21:53:07.7219458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7219655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7219717Z return mod(**inputs) 2025-08-14T21:53:07.7220018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7220088Z outputs = self.mobilebert( 2025-08-14T21:53:07.7220356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7220450Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7220726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7220802Z layer_outputs = layer_module( 2025-08-14T21:53:07.7221078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7221183Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7221456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7221597Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7221865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7221954Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7221958Z 2025-08-14T21:53:07.7222054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7222249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7222313Z return mod(**inputs) 2025-08-14T21:53:07.7222583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7222664Z outputs = self.mobilebert( 2025-08-14T21:53:07.7222947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7223033Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7223322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7223398Z layer_outputs = layer_module( 2025-08-14T21:53:07.7223694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7223792Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7224081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7224219Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7224509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7224645Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7224937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7225031Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7225036Z 2025-08-14T21:53:07.7225145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7225342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7225418Z return mod(**inputs) 2025-08-14T21:53:07.7225691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7225765Z outputs = self.mobilebert( 2025-08-14T21:53:07.7226047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7226140Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7226415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7226493Z layer_outputs = layer_module( 2025-08-14T21:53:07.7226772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7226897Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7227166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7227274Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7227575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7227661Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7227666Z 2025-08-14T21:53:07.7227789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7227987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7228054Z return mod(**inputs) 2025-08-14T21:53:07.7228335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7228407Z outputs = self.mobilebert( 2025-08-14T21:53:07.7228675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7228756Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7229028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7229107Z layer_outputs = layer_module( 2025-08-14T21:53:07.7229382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7229473Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7229750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7229863Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7230139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7230248Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7230251Z 2025-08-14T21:53:07.7230349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7230553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7230617Z return mod(**inputs) 2025-08-14T21:53:07.7230888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7230968Z outputs = self.mobilebert( 2025-08-14T21:53:07.7231238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7231318Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7231590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7231659Z layer_outputs = layer_module( 2025-08-14T21:53:07.7232079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7232195Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7232490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7232650Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7232940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7233054Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7233058Z 2025-08-14T21:53:07.7233166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7233379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7233448Z return mod(**inputs) 2025-08-14T21:53:07.7233748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7233848Z outputs = self.mobilebert( 2025-08-14T21:53:07.7234139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7234217Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7234532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7234610Z layer_outputs = layer_module( 2025-08-14T21:53:07.7234909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7235009Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7235299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7235435Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7235818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7236023Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7236318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7236414Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7236420Z 2025-08-14T21:53:07.7236537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7236745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7236817Z return mod(**inputs) 2025-08-14T21:53:07.7237130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7237205Z outputs = self.mobilebert( 2025-08-14T21:53:07.7237507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7237584Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7237875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7237960Z layer_outputs = layer_module( 2025-08-14T21:53:07.7238250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7238364Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7238640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7238749Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7239034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7239118Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7239146Z 2025-08-14T21:53:07.7239249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7239456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7239521Z return mod(**inputs) 2025-08-14T21:53:07.7239801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7239889Z outputs = self.mobilebert( 2025-08-14T21:53:07.7240163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7240242Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7240529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7240608Z layer_outputs = layer_module( 2025-08-14T21:53:07.7240881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7240992Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7241277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7241388Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7241663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7241779Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7241782Z 2025-08-14T21:53:07.7241881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7242096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7242166Z return mod(**inputs) 2025-08-14T21:53:07.7242462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7242545Z outputs = self.mobilebert( 2025-08-14T21:53:07.7242834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7242915Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7243192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7243263Z layer_outputs = layer_module( 2025-08-14T21:53:07.7243544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7243637Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7243924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7244061Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7244357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7244457Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7244461Z 2025-08-14T21:53:07.7244569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7244783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7244861Z return mod(**inputs) 2025-08-14T21:53:07.7245162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7245246Z outputs = self.mobilebert( 2025-08-14T21:53:07.7245548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7245644Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7245955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7246030Z layer_outputs = layer_module( 2025-08-14T21:53:07.7246335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7246448Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7246719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7246847Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7247135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7247259Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7247556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7247652Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7247658Z 2025-08-14T21:53:07.7247780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7247984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7248054Z return mod(**inputs) 2025-08-14T21:53:07.7248351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7248425Z outputs = self.mobilebert( 2025-08-14T21:53:07.7248721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7248798Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7249083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7249163Z layer_outputs = layer_module( 2025-08-14T21:53:07.7249452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7249581Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7249877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7249964Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7249967Z 2025-08-14T21:53:07.7250081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7250287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7250356Z return mod(**inputs) 2025-08-14T21:53:07.7250650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7250726Z outputs = self.mobilebert( 2025-08-14T21:53:07.7251020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7251096Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7251384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7251465Z layer_outputs = layer_module( 2025-08-14T21:53:07.7251754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7251883Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7252203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7252318Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7252322Z 2025-08-14T21:53:07.7252436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7252659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7252727Z return mod(**inputs) 2025-08-14T21:53:07.7253020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7253093Z outputs = self.mobilebert( 2025-08-14T21:53:07.7253386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7253479Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7253770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7253872Z layer_outputs = layer_module( 2025-08-14T21:53:07.7254163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7254332Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7254631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.7254731Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.7254734Z 2025-08-14T21:53:07.7254846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7255057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7255126Z return mod(**inputs) 2025-08-14T21:53:07.7255429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7255502Z outputs = self.mobilebert( 2025-08-14T21:53:07.7255798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7255875Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7256165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7256247Z layer_outputs = layer_module( 2025-08-14T21:53:07.7256534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7256699Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7256996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.7257127Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.7257424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7257523Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7257527Z 2025-08-14T21:53:07.7257631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7257846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7257914Z return mod(**inputs) 2025-08-14T21:53:07.7258221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7258296Z outputs = self.mobilebert( 2025-08-14T21:53:07.7258597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7258704Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7258999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7259074Z layer_outputs = layer_module( 2025-08-14T21:53:07.7259385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7259548Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7259855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7259985Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7260307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.7260407Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7260426Z 2025-08-14T21:53:07.7260535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7260745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7260817Z return mod(**inputs) 2025-08-14T21:53:07.7261107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7261189Z outputs = self.mobilebert( 2025-08-14T21:53:07.7261479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7261566Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7261856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7261931Z layer_outputs = layer_module( 2025-08-14T21:53:07.7262229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7262393Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7262683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7262819Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7263120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.7263255Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7263559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7263658Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7263662Z 2025-08-14T21:53:07.7263779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7263987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7264066Z return mod(**inputs) 2025-08-14T21:53:07.7264353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7264428Z outputs = self.mobilebert( 2025-08-14T21:53:07.7264722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7264800Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7265101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7265202Z layer_outputs = layer_module( 2025-08-14T21:53:07.7265497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7265672Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7265980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.7266095Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.7266394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.7266481Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.7266485Z 2025-08-14T21:53:07.7266618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7266830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7266902Z return mod(**inputs) 2025-08-14T21:53:07.7267225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7267304Z outputs = self.mobilebert( 2025-08-14T21:53:07.7267609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7267687Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7267989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7268072Z layer_outputs = layer_module( 2025-08-14T21:53:07.7268376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7268467Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7268782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7268859Z self_outputs = self.self( 2025-08-14T21:53:07.7269167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:07.7269246Z self.value(value_tensor) 2025-08-14T21:53:07.7269250Z 2025-08-14T21:53:07.7269357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7269579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7269648Z return mod(**inputs) 2025-08-14T21:53:07.7269944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7270031Z outputs = self.mobilebert( 2025-08-14T21:53:07.7270329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7270416Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7270713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7270790Z layer_outputs = layer_module( 2025-08-14T21:53:07.7271094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7271266Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7271576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:07.7271697Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:07.7271993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:07.7272116Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:07.7272120Z 2025-08-14T21:53:07.7272231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7272452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7272542Z return mod(**inputs) 2025-08-14T21:53:07.7272850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7272933Z outputs = self.mobilebert( 2025-08-14T21:53:07.7273240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7273315Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7273649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7273728Z layer_outputs = layer_module( 2025-08-14T21:53:07.7274058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:07.7274232Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:07.7274535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:07.7274659Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:07.7274956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:07.7275056Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:07.7275358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7275459Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7275462Z 2025-08-14T21:53:07.7275579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7275866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7275949Z return mod(**inputs) 2025-08-14T21:53:07.7276259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7276337Z outputs = self.mobilebert( 2025-08-14T21:53:07.7276653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7276732Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7277044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7277133Z layer_outputs = layer_module( 2025-08-14T21:53:07.7277436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7277538Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7277839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7277918Z self_outputs = self.self( 2025-08-14T21:53:07.7278228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:07.7278305Z self.query(query_tensor) 2025-08-14T21:53:07.7278309Z 2025-08-14T21:53:07.7278422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7278642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7278715Z return mod(**inputs) 2025-08-14T21:53:07.7279058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7279135Z outputs = self.mobilebert( 2025-08-14T21:53:07.7279439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7279548Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7279853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7279937Z layer_outputs = layer_module( 2025-08-14T21:53:07.7280238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7280349Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7280654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:07.7280733Z self_outputs = self.self( 2025-08-14T21:53:07.7281049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:07.7281130Z self.key(key_tensor) 2025-08-14T21:53:07.7281136Z 2025-08-14T21:53:07.7281224Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.7281315Z cudagraph partition due to non gpu ops 2025-08-14T21:53:07.7281425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7281636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7281713Z return mod(**inputs) 2025-08-14T21:53:07.7282002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7282073Z outputs = self.mobilebert( 2025-08-14T21:53:07.7282352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7282428Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7282723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7282798Z layer_outputs = layer_module( 2025-08-14T21:53:07.7283085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7283181Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7283469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.7283607Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.7283894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:07.7283987Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7283990Z 2025-08-14T21:53:07.7284103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7284309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7284380Z return mod(**inputs) 2025-08-14T21:53:07.7284672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7284745Z outputs = self.mobilebert( 2025-08-14T21:53:07.7285048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7285123Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7285426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7285529Z layer_outputs = layer_module( 2025-08-14T21:53:07.7285823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:07.7285918Z self_attention_outputs = self.attention( 2025-08-14T21:53:07.7286228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:07.7286355Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:07.7286659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:07.7286790Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7287112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7287222Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7287226Z 2025-08-14T21:53:07.7287353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7287574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7287645Z return mod(**inputs) 2025-08-14T21:53:07.7287944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7288030Z outputs = self.mobilebert( 2025-08-14T21:53:07.7288331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7288413Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7288700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7288773Z layer_outputs = layer_module( 2025-08-14T21:53:07.7289070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7289168Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7289453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7289577Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7289864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7289958Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7289961Z 2025-08-14T21:53:07.7290065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7290270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7290346Z return mod(**inputs) 2025-08-14T21:53:07.7290633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7290715Z outputs = self.mobilebert( 2025-08-14T21:53:07.7290999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7291075Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7291370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7291444Z layer_outputs = layer_module( 2025-08-14T21:53:07.7291727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7291833Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7292117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7292261Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7292546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7292685Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7292689Z 2025-08-14T21:53:07.7292807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7293017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7293093Z return mod(**inputs) 2025-08-14T21:53:07.7293387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7293486Z outputs = self.mobilebert( 2025-08-14T21:53:07.7293792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7293892Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7294189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7294274Z layer_outputs = layer_module( 2025-08-14T21:53:07.7294571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7294679Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7294975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7295122Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7295419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7295508Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7295513Z 2025-08-14T21:53:07.7295626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7295840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7295913Z return mod(**inputs) 2025-08-14T21:53:07.7296216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7296292Z outputs = self.mobilebert( 2025-08-14T21:53:07.7296595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7296672Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7296978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7297061Z layer_outputs = layer_module( 2025-08-14T21:53:07.7297351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7297449Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7297745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7297875Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7298168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7298298Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7298595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7298725Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7298729Z 2025-08-14T21:53:07.7298839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7299058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7299128Z return mod(**inputs) 2025-08-14T21:53:07.7299448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7299535Z outputs = self.mobilebert( 2025-08-14T21:53:07.7299834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7299911Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7300247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7300326Z layer_outputs = layer_module( 2025-08-14T21:53:07.7300680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7300781Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7301078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7301206Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7301502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7301597Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7301601Z 2025-08-14T21:53:07.7301711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7301926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7302006Z return mod(**inputs) 2025-08-14T21:53:07.7302311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7302388Z outputs = self.mobilebert( 2025-08-14T21:53:07.7302695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7302776Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7303089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7303165Z layer_outputs = layer_module( 2025-08-14T21:53:07.7303466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7303574Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7303881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7304013Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7304322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7304444Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7304447Z 2025-08-14T21:53:07.7304567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7304783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7304854Z return mod(**inputs) 2025-08-14T21:53:07.7305164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7305242Z outputs = self.mobilebert( 2025-08-14T21:53:07.7305551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7305651Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7305955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7306059Z layer_outputs = layer_module( 2025-08-14T21:53:07.7306354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7306462Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7306761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7306894Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7307218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7307311Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7307332Z 2025-08-14T21:53:07.7307450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7307661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7307732Z return mod(**inputs) 2025-08-14T21:53:07.7308036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7308113Z outputs = self.mobilebert( 2025-08-14T21:53:07.7308412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7308497Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7309019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7309114Z layer_outputs = layer_module( 2025-08-14T21:53:07.7309413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7309513Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7309817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7309955Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7310260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7310390Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7310688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7310800Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7310804Z 2025-08-14T21:53:07.7310918Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7311130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7311211Z return mod(**inputs) 2025-08-14T21:53:07.7311509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7311594Z outputs = self.mobilebert( 2025-08-14T21:53:07.7311889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7311968Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7312275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7312352Z layer_outputs = layer_module( 2025-08-14T21:53:07.7312708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7312807Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7313107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7313260Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7313557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7313647Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7313658Z 2025-08-14T21:53:07.7313768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7314004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7314086Z return mod(**inputs) 2025-08-14T21:53:07.7314416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7314497Z outputs = self.mobilebert( 2025-08-14T21:53:07.7314809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7314890Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7315202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7315278Z layer_outputs = layer_module( 2025-08-14T21:53:07.7315583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7315693Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7316053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:07.7316179Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:07.7316484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7316606Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7316610Z 2025-08-14T21:53:07.7316726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7316940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7317012Z return mod(**inputs) 2025-08-14T21:53:07.7317321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7317399Z outputs = self.mobilebert( 2025-08-14T21:53:07.7317703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7317783Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7318087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7318168Z layer_outputs = layer_module( 2025-08-14T21:53:07.7318434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7318525Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7318796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7318915Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7319187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:07.7319294Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7319297Z 2025-08-14T21:53:07.7319402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7319605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7319686Z return mod(**inputs) 2025-08-14T21:53:07.7319965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7320034Z outputs = self.mobilebert( 2025-08-14T21:53:07.7320309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7320386Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7320676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7320748Z layer_outputs = layer_module( 2025-08-14T21:53:07.7321038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:07.7321131Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:07.7321404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:07.7321525Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:07.7321789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:07.7321914Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7322181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7322276Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7322281Z 2025-08-14T21:53:07.7322380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7322571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7322642Z return mod(**inputs) 2025-08-14T21:53:07.7322907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7322982Z outputs = self.mobilebert( 2025-08-14T21:53:07.7323245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7323313Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7323589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7323656Z layer_outputs = layer_module( 2025-08-14T21:53:07.7323930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7324054Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7324341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:07.7324435Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7324438Z 2025-08-14T21:53:07.7324542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7324756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7324827Z return mod(**inputs) 2025-08-14T21:53:07.7325106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7325182Z outputs = self.mobilebert( 2025-08-14T21:53:07.7325481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7325553Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7325833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7325930Z layer_outputs = layer_module( 2025-08-14T21:53:07.7326207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:07.7326334Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:07.7326617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:07.7326752Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:07.7326756Z 2025-08-14T21:53:07.7326858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7327072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7327144Z return mod(**inputs) 2025-08-14T21:53:07.7327416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7327494Z outputs = self.mobilebert( 2025-08-14T21:53:07.7327765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7327836Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7328116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7328186Z layer_outputs = layer_module( 2025-08-14T21:53:07.7328458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7328627Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7328901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:07.7329005Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:07.7329008Z 2025-08-14T21:53:07.7329109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7329305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7329377Z return mod(**inputs) 2025-08-14T21:53:07.7329650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7329730Z outputs = self.mobilebert( 2025-08-14T21:53:07.7330004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7330077Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7330368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7330437Z layer_outputs = layer_module( 2025-08-14T21:53:07.7330705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7330862Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7331128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:07.7331254Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:07.7331523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7331630Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7331635Z 2025-08-14T21:53:07.7331742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7331932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7332024Z return mod(**inputs) 2025-08-14T21:53:07.7332291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7332359Z outputs = self.mobilebert( 2025-08-14T21:53:07.7332632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7332701Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7332983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7333061Z layer_outputs = layer_module( 2025-08-14T21:53:07.7333338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7333497Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7333763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7333883Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7334155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:07.7334236Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:07.7334240Z 2025-08-14T21:53:07.7334345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7334539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7334605Z return mod(**inputs) 2025-08-14T21:53:07.7334890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:53:07.7334962Z outputs = self.mobilebert( 2025-08-14T21:53:07.7335265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:07.7335344Z encoder_outputs = self.encoder( 2025-08-14T21:53:07.7335639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:07.7335725Z layer_outputs = layer_module( 2025-08-14T21:53:07.7336023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:07.7336188Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:07.7336501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:07.7336626Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:07.7336915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:07.7337039Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:07.7337317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:07.7337420Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:07.7337424Z 2025-08-14T21:53:07.7337528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7337740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7337830Z return mod(**inputs) 2025-08-14T21:53:07.7338118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-14T21:53:07.7338228Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:53:07.7338527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-14T21:53:07.7338640Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:53:07.7338924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 631, in forward 2025-08-14T21:53:07.7339013Z hidden_states = self.transform(hidden_states) 2025-08-14T21:53:07.7339305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 609, in forward 2025-08-14T21:53:07.7339392Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:07.7339395Z 2025-08-14T21:53:07.7339510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7339714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7339778Z return mod(**inputs) 2025-08-14T21:53:07.7340056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-14T21:53:07.7340147Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:53:07.7340423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-14T21:53:07.7340537Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:53:07.7340814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-08-14T21:53:07.7341030Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-08-14T21:53:07.7341035Z 2025-08-14T21:53:07.7341134Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7341326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7341398Z return mod(**inputs) 2025-08-14T21:53:07.7341668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-14T21:53:07.7341756Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:53:07.7342033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-14T21:53:07.7342140Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:53:07.7342418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 633, in forward 2025-08-14T21:53:07.7342499Z hidden_states += self.decoder.bias 2025-08-14T21:53:07.7342504Z 2025-08-14T21:53:07.7342603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:07.7342803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:07.7342867Z return mod(**inputs) 2025-08-14T21:53:07.7343144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 994, in forward 2025-08-14T21:53:07.7343333Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:53:07.7343336Z 2025-08-14T21:53:20.5871323Z Compilation time (from dynamo_timed): 39.027366612 2025-08-14T21:53:20.5894486Z pass 2025-08-14T21:53:20.5895000Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:20.5895874Z TIMING: _recursive_pre_grad_passes:0.02447 _recursive_joint_graph_passes:1.39083 _recursive_post_grad_passes:0.22633 async_compile.wait:0.7587 code_gen:9.69063 inductor_compile:14.29085 backend_compile:27.07377 gc:0.00079 entire_frame_compile:39.02737 total_wall_time:39.02737 2025-08-14T21:53:20.5897201Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:56776 | FakeTensor.__torch_dispatch__:16414 | ProxyTorchDispatchMode.__torch_dispatch__:21632 2025-08-14T21:53:20.5897809Z Dynamo produced 1 graphs covering 1449 ops with 0 graph breaks (0 unique) 2025-08-14T21:53:26.8236478Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:53:26.8237806Z from pkg_resources import resource_filename 2025-08-14T21:53:27.3914893Z 2025-08-14T21:53:27.9769256Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:53:27.9772512Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:53:27.9837992Z cpu eval MobileBertForQuestionAnswering 2025-08-14T21:53:28.1884234Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:28.3240391Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:28.4561315Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:55.7620636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7623442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7626549Z return mod(**inputs) 2025-08-14T21:53:55.7627178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7630344Z outputs = self.mobilebert( 2025-08-14T21:53:55.7634177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:53:55.7634746Z embedding_output = self.embeddings( 2025-08-14T21:53:55.7635236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-08-14T21:53:55.7635887Z inputs_embeds = torch.cat( 2025-08-14T21:53:55.7636047Z 2025-08-14T21:53:55.7636176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7636603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7636957Z return mod(**inputs) 2025-08-14T21:53:55.7637415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7637885Z outputs = self.mobilebert( 2025-08-14T21:53:55.7638310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:53:55.7638697Z embedding_output = self.embeddings( 2025-08-14T21:53:55.7639095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-08-14T21:53:55.7639539Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-08-14T21:53:55.7639707Z 2025-08-14T21:53:55.7639820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7640212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7640558Z return mod(**inputs) 2025-08-14T21:53:55.7640988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7641434Z outputs = self.mobilebert( 2025-08-14T21:53:55.7642165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:53:55.7642624Z embedding_output = self.embeddings( 2025-08-14T21:53:55.7643222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-08-14T21:53:55.7643722Z embeddings = self.LayerNorm(embeddings) 2025-08-14T21:53:55.7644179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7644649Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7644808Z 2025-08-14T21:53:55.7644927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7645365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7645718Z return mod(**inputs) 2025-08-14T21:53:55.7646143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7646625Z outputs = self.mobilebert( 2025-08-14T21:53:55.7647062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7647522Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7647972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7648421Z layer_outputs = layer_module( 2025-08-14T21:53:55.7648868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.7649422Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.7649981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.7650479Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.7650965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.7651429Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.7651579Z 2025-08-14T21:53:55.7651691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7652077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7652428Z return mod(**inputs) 2025-08-14T21:53:55.7652893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7653331Z outputs = self.mobilebert( 2025-08-14T21:53:55.7653768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7654219Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7654667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7655100Z layer_outputs = layer_module( 2025-08-14T21:53:55.7655545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.7656100Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.7656634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.7657089Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.7657542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.7657989Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.7658409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7658863Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7659046Z 2025-08-14T21:53:55.7659158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7659539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7659875Z return mod(**inputs) 2025-08-14T21:53:55.7660291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7660742Z outputs = self.mobilebert( 2025-08-14T21:53:55.7661196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7661637Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7662065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7662483Z layer_outputs = layer_module( 2025-08-14T21:53:55.7662882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7663333Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7663795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.7664227Z self_outputs = self.self( 2025-08-14T21:53:55.7664648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.7665081Z self.query(query_tensor) 2025-08-14T21:53:55.7665202Z 2025-08-14T21:53:55.7665319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7665716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7666044Z return mod(**inputs) 2025-08-14T21:53:55.7666434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7666850Z outputs = self.mobilebert( 2025-08-14T21:53:55.7667240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7667660Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7668092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7668535Z layer_outputs = layer_module( 2025-08-14T21:53:55.7668953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7669423Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7669884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.7670324Z self_outputs = self.self( 2025-08-14T21:53:55.7670775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.7671222Z self.key(key_tensor) 2025-08-14T21:53:55.7671338Z 2025-08-14T21:53:55.7671458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7671844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7672201Z return mod(**inputs) 2025-08-14T21:53:55.7672629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7673088Z outputs = self.mobilebert( 2025-08-14T21:53:55.7673522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7673972Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7674414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7674875Z layer_outputs = layer_module( 2025-08-14T21:53:55.7675322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7675890Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7676382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.7676827Z self_outputs = self.self( 2025-08-14T21:53:55.7677284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.7677732Z self.value(value_tensor) 2025-08-14T21:53:55.7677854Z 2025-08-14T21:53:55.7677946Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.7678184Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.7678449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7678838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7679181Z return mod(**inputs) 2025-08-14T21:53:55.7679599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7680046Z outputs = self.mobilebert( 2025-08-14T21:53:55.7680471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7680918Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7681356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7681800Z layer_outputs = layer_module( 2025-08-14T21:53:55.7682231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7682692Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7683145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.7683649Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.7684142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.7684613Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7684761Z 2025-08-14T21:53:55.7684878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7685246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7685596Z return mod(**inputs) 2025-08-14T21:53:55.7686026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7686467Z outputs = self.mobilebert( 2025-08-14T21:53:55.7686881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7687320Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7687731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7688143Z layer_outputs = layer_module( 2025-08-14T21:53:55.7688562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.7689121Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.7689651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.7690158Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.7690598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.7691011Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.7691153Z 2025-08-14T21:53:55.7691264Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7691669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7692013Z return mod(**inputs) 2025-08-14T21:53:55.7692439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7692855Z outputs = self.mobilebert( 2025-08-14T21:53:55.7693252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7693687Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7694112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7694536Z layer_outputs = layer_module( 2025-08-14T21:53:55.7694961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7695410Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7695860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.7696351Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.7696855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.7697349Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7697810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7698234Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7698388Z 2025-08-14T21:53:55.7698490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7698839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7699163Z return mod(**inputs) 2025-08-14T21:53:55.7699539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7699953Z outputs = self.mobilebert( 2025-08-14T21:53:55.7700348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7700753Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7701182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7701590Z layer_outputs = layer_module( 2025-08-14T21:53:55.7701991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7702416Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7702856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7703343Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7703825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.7704275Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.7704454Z 2025-08-14T21:53:55.7704575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7704936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7705255Z return mod(**inputs) 2025-08-14T21:53:55.7705646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7706098Z outputs = self.mobilebert( 2025-08-14T21:53:55.7706550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7706992Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7707452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7707894Z layer_outputs = layer_module( 2025-08-14T21:53:55.7708327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7709135Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7709655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7710170Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7710705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.7711215Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.7711402Z 2025-08-14T21:53:55.7711514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7711903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7712246Z return mod(**inputs) 2025-08-14T21:53:55.7712674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7713125Z outputs = self.mobilebert( 2025-08-14T21:53:55.7713576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7714026Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7714475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7714922Z layer_outputs = layer_module( 2025-08-14T21:53:55.7715352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7715883Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7716363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7716860Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7717337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.7717788Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7717935Z 2025-08-14T21:53:55.7718050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7718430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7718764Z return mod(**inputs) 2025-08-14T21:53:55.7719265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7719702Z outputs = self.mobilebert( 2025-08-14T21:53:55.7720134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7720619Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7721067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7721478Z layer_outputs = layer_module( 2025-08-14T21:53:55.7721877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7722347Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7722781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7723266Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7723729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.7724185Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7724642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7725077Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7725234Z 2025-08-14T21:53:55.7725340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7725705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7726040Z return mod(**inputs) 2025-08-14T21:53:55.7726449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7726884Z outputs = self.mobilebert( 2025-08-14T21:53:55.7727313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7727754Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7728180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7728618Z layer_outputs = layer_module( 2025-08-14T21:53:55.7729047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7729500Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7729957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7730409Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7730863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.7731294Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.7731454Z 2025-08-14T21:53:55.7731563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7731937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7732256Z return mod(**inputs) 2025-08-14T21:53:55.7732639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7733052Z outputs = self.mobilebert( 2025-08-14T21:53:55.7733660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7734117Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7734558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7734995Z layer_outputs = layer_module( 2025-08-14T21:53:55.7735421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7735896Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7736360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7736854Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7737351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.7737827Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.7738014Z 2025-08-14T21:53:55.7738126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7738519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7738862Z return mod(**inputs) 2025-08-14T21:53:55.7739285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7739733Z outputs = self.mobilebert( 2025-08-14T21:53:55.7740171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7740613Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7741056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7741510Z layer_outputs = layer_module( 2025-08-14T21:53:55.7741948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7742405Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7742868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7743388Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7743953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.7744409Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7744572Z 2025-08-14T21:53:55.7744685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7745074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7745412Z return mod(**inputs) 2025-08-14T21:53:55.7745820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7746251Z outputs = self.mobilebert( 2025-08-14T21:53:55.7746664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7747089Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7747532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7747963Z layer_outputs = layer_module( 2025-08-14T21:53:55.7748377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7748829Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7749282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7749798Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7750276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.7750757Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7751282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7751747Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7751911Z 2025-08-14T21:53:55.7752026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7752443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7752845Z return mod(**inputs) 2025-08-14T21:53:55.7753271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7753746Z outputs = self.mobilebert( 2025-08-14T21:53:55.7754184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7754634Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7755071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7755520Z layer_outputs = layer_module( 2025-08-14T21:53:55.7756019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7756494Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7756955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7757451Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7757947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.7758405Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.7758560Z 2025-08-14T21:53:55.7758675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7759062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7759414Z return mod(**inputs) 2025-08-14T21:53:55.7759844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7760304Z outputs = self.mobilebert( 2025-08-14T21:53:55.7760809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7761277Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7761721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7762168Z layer_outputs = layer_module( 2025-08-14T21:53:55.7762606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7763071Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7763548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7764038Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7764524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.7765017Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.7766028Z 2025-08-14T21:53:55.7766144Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7766532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7766880Z return mod(**inputs) 2025-08-14T21:53:55.7767308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7767766Z outputs = self.mobilebert( 2025-08-14T21:53:55.7768187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7768619Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7769069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7769505Z layer_outputs = layer_module( 2025-08-14T21:53:55.7769932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7770395Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7770850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7771335Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7771819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.7772259Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7772414Z 2025-08-14T21:53:55.7772525Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7772908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7773249Z return mod(**inputs) 2025-08-14T21:53:55.7773651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7774095Z outputs = self.mobilebert( 2025-08-14T21:53:55.7774521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7774951Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7775377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7775806Z layer_outputs = layer_module( 2025-08-14T21:53:55.7776228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7776673Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7777130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7777621Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7778118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.7778592Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7779069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7779528Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7779689Z 2025-08-14T21:53:55.7779793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7780147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7780483Z return mod(**inputs) 2025-08-14T21:53:55.7780889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7781344Z outputs = self.mobilebert( 2025-08-14T21:53:55.7781772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7782224Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7782669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7783103Z layer_outputs = layer_module( 2025-08-14T21:53:55.7783539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.7784030Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.7784536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.7784980Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.7785133Z 2025-08-14T21:53:55.7785257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7785634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7785974Z return mod(**inputs) 2025-08-14T21:53:55.7786402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7786865Z outputs = self.mobilebert( 2025-08-14T21:53:55.7787297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7787743Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7788189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7788629Z layer_outputs = layer_module( 2025-08-14T21:53:55.7789056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.7789549Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.7790036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.7790518Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.7790702Z 2025-08-14T21:53:55.7790818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7791213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7791576Z return mod(**inputs) 2025-08-14T21:53:55.7792010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7792459Z outputs = self.mobilebert( 2025-08-14T21:53:55.7792899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7793356Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7793795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7794263Z layer_outputs = layer_module( 2025-08-14T21:53:55.7794704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.7795254Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.7795874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.7796362Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.7796552Z 2025-08-14T21:53:55.7796673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7797069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7797413Z return mod(**inputs) 2025-08-14T21:53:55.7797849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7798321Z outputs = self.mobilebert( 2025-08-14T21:53:55.7798748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7799195Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7799642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7800109Z layer_outputs = layer_module( 2025-08-14T21:53:55.7800543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.7801116Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.7801671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.7802181Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.7802688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7803163Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7803324Z 2025-08-14T21:53:55.7803447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7803839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7804192Z return mod(**inputs) 2025-08-14T21:53:55.7804636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7805095Z outputs = self.mobilebert( 2025-08-14T21:53:55.7805546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7806004Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7806467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7806932Z layer_outputs = layer_module( 2025-08-14T21:53:55.7807383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.7807929Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.7808481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.7809240Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.7809731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.7810189Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7810340Z 2025-08-14T21:53:55.7810460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7810835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7811181Z return mod(**inputs) 2025-08-14T21:53:55.7811608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7812059Z outputs = self.mobilebert( 2025-08-14T21:53:55.7812487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7813023Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7813459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7813923Z layer_outputs = layer_module( 2025-08-14T21:53:55.7814343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.7814870Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.7815391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.7815892Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.7816378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.7816883Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7817344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7817770Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7817924Z 2025-08-14T21:53:55.7818028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7818386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7818720Z return mod(**inputs) 2025-08-14T21:53:55.7819121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7819559Z outputs = self.mobilebert( 2025-08-14T21:53:55.7819980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7820409Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7820835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7821307Z layer_outputs = layer_module( 2025-08-14T21:53:55.7821731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.7822257Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.7822794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.7823272Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.7823746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.7824183Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.7824336Z 2025-08-14T21:53:55.7824449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7824822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7825163Z return mod(**inputs) 2025-08-14T21:53:55.7825563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7825998Z outputs = self.mobilebert( 2025-08-14T21:53:55.7826429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7826871Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7827314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7827763Z layer_outputs = layer_module( 2025-08-14T21:53:55.7828200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.7828716Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.7829282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.7829777Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.7830283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.7830759Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.7831246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7831724Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7831880Z 2025-08-14T21:53:55.7832006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7832396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7832749Z return mod(**inputs) 2025-08-14T21:53:55.7833171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7833611Z outputs = self.mobilebert( 2025-08-14T21:53:55.7834041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7834493Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7834935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7835371Z layer_outputs = layer_module( 2025-08-14T21:53:55.7835892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7836361Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7836809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.7837253Z self_outputs = self.self( 2025-08-14T21:53:55.7837676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.7838110Z self.query(query_tensor) 2025-08-14T21:53:55.7838233Z 2025-08-14T21:53:55.7838342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7838724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7839067Z return mod(**inputs) 2025-08-14T21:53:55.7839473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7839909Z outputs = self.mobilebert( 2025-08-14T21:53:55.7840336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7840767Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7841190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7841620Z layer_outputs = layer_module( 2025-08-14T21:53:55.7842040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7842487Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7842920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.7843370Z self_outputs = self.self( 2025-08-14T21:53:55.7843787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.7844206Z self.key(key_tensor) 2025-08-14T21:53:55.7844327Z 2025-08-14T21:53:55.7844456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7844834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7845173Z return mod(**inputs) 2025-08-14T21:53:55.7845580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7846013Z outputs = self.mobilebert( 2025-08-14T21:53:55.7846445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7846889Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7847383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7847817Z layer_outputs = layer_module( 2025-08-14T21:53:55.7848238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7848675Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7849116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.7849545Z self_outputs = self.self( 2025-08-14T21:53:55.7849963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.7850390Z self.value(value_tensor) 2025-08-14T21:53:55.7850520Z 2025-08-14T21:53:55.7850609Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.7850841Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.7851084Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7851464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7851802Z return mod(**inputs) 2025-08-14T21:53:55.7852211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7852639Z outputs = self.mobilebert( 2025-08-14T21:53:55.7853067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7853498Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7853926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7854352Z layer_outputs = layer_module( 2025-08-14T21:53:55.7854780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7855223Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7855652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.7856138Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.7856616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.7857067Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7857213Z 2025-08-14T21:53:55.7857323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7857700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7858065Z return mod(**inputs) 2025-08-14T21:53:55.7858473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7858900Z outputs = self.mobilebert( 2025-08-14T21:53:55.7859319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7859776Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7860201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7860635Z layer_outputs = layer_module( 2025-08-14T21:53:55.7861063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.7861609Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.7862160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.7862642Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.7863121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.7863589Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.7863749Z 2025-08-14T21:53:55.7863856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7864242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7864594Z return mod(**inputs) 2025-08-14T21:53:55.7865021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7865468Z outputs = self.mobilebert( 2025-08-14T21:53:55.7865906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7866447Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7866880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7867329Z layer_outputs = layer_module( 2025-08-14T21:53:55.7867775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.7868243Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.7868707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.7869213Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.7869718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.7870215Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7870729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7871204Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7871367Z 2025-08-14T21:53:55.7871488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7871868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7872222Z return mod(**inputs) 2025-08-14T21:53:55.7872645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7873093Z outputs = self.mobilebert( 2025-08-14T21:53:55.7873525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7873992Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7874440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7874875Z layer_outputs = layer_module( 2025-08-14T21:53:55.7875332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7875894Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7876383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7876878Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7877393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.7877862Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.7878016Z 2025-08-14T21:53:55.7878153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7878536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7878885Z return mod(**inputs) 2025-08-14T21:53:55.7879275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7879683Z outputs = self.mobilebert( 2025-08-14T21:53:55.7880079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7880485Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7880906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7881330Z layer_outputs = layer_module( 2025-08-14T21:53:55.7881754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7882205Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7882655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7883117Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7883587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.7884057Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.7884230Z 2025-08-14T21:53:55.7884340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7884719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7885067Z return mod(**inputs) 2025-08-14T21:53:55.7885476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7885903Z outputs = self.mobilebert( 2025-08-14T21:53:55.7886321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7886753Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7887180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7887604Z layer_outputs = layer_module( 2025-08-14T21:53:55.7888024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7888484Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7888930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7889447Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7889906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.7890343Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7890482Z 2025-08-14T21:53:55.7890586Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7890964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7891311Z return mod(**inputs) 2025-08-14T21:53:55.7891705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7892130Z outputs = self.mobilebert( 2025-08-14T21:53:55.7892530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7892955Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7893352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7893759Z layer_outputs = layer_module( 2025-08-14T21:53:55.7894162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7894595Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7895014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7895473Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7895935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.7896394Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7896844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7897279Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7897429Z 2025-08-14T21:53:55.7897542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7897897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7898222Z return mod(**inputs) 2025-08-14T21:53:55.7898613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7899028Z outputs = self.mobilebert( 2025-08-14T21:53:55.7899422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7899834Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7900245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7900652Z layer_outputs = layer_module( 2025-08-14T21:53:55.7901046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7901476Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7901902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7902347Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7902793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.7903233Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.7903372Z 2025-08-14T21:53:55.7903485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7903831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7904154Z return mod(**inputs) 2025-08-14T21:53:55.7904560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7904967Z outputs = self.mobilebert( 2025-08-14T21:53:55.7905357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7905765Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7906184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7906600Z layer_outputs = layer_module( 2025-08-14T21:53:55.7907025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7907462Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7907921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7908393Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7908998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.7909492Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.7909665Z 2025-08-14T21:53:55.7909783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7910161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7910508Z return mod(**inputs) 2025-08-14T21:53:55.7910922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7911350Z outputs = self.mobilebert( 2025-08-14T21:53:55.7911772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7912208Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7912646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7913083Z layer_outputs = layer_module( 2025-08-14T21:53:55.7913516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7913972Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7914450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7914943Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7915445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.7915967Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7916126Z 2025-08-14T21:53:55.7916240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7916633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7916985Z return mod(**inputs) 2025-08-14T21:53:55.7917382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7917782Z outputs = self.mobilebert( 2025-08-14T21:53:55.7918170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7918621Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7919012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7919432Z layer_outputs = layer_module( 2025-08-14T21:53:55.7919823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7920246Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7920655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7921099Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7921582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.7922061Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7922518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7922941Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7923088Z 2025-08-14T21:53:55.7923200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7923552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7923869Z return mod(**inputs) 2025-08-14T21:53:55.7924259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7924672Z outputs = self.mobilebert( 2025-08-14T21:53:55.7925068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7925482Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7925892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7926309Z layer_outputs = layer_module( 2025-08-14T21:53:55.7926699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7927133Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7927570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7928023Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7928466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.7928895Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.7929037Z 2025-08-14T21:53:55.7929152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7929504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7929833Z return mod(**inputs) 2025-08-14T21:53:55.7930229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7930645Z outputs = self.mobilebert( 2025-08-14T21:53:55.7931043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7931454Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7931864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7932275Z layer_outputs = layer_module( 2025-08-14T21:53:55.7932688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7933114Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7933542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.7934002Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.7934456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.7934909Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.7935080Z 2025-08-14T21:53:55.7935194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7935564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7935895Z return mod(**inputs) 2025-08-14T21:53:55.7936298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7936715Z outputs = self.mobilebert( 2025-08-14T21:53:55.7937105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7937518Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7937923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7938323Z layer_outputs = layer_module( 2025-08-14T21:53:55.7938724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7939163Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7939581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7940027Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7940482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.7940904Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7941045Z 2025-08-14T21:53:55.7942736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7943086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7943422Z return mod(**inputs) 2025-08-14T21:53:55.7943800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7944194Z outputs = self.mobilebert( 2025-08-14T21:53:55.7944585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7944990Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7945382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7945779Z layer_outputs = layer_module( 2025-08-14T21:53:55.7946176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.7946598Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.7947016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.7947458Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.7947939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.7948438Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7948897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7949351Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7949547Z 2025-08-14T21:53:55.7949656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7950033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7950366Z return mod(**inputs) 2025-08-14T21:53:55.7950777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7951205Z outputs = self.mobilebert( 2025-08-14T21:53:55.7951655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7952084Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7952544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7952976Z layer_outputs = layer_module( 2025-08-14T21:53:55.7953390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.7953885Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.7954371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.7954829Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.7954976Z 2025-08-14T21:53:55.7955085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7955457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7955885Z return mod(**inputs) 2025-08-14T21:53:55.7956316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7956764Z outputs = self.mobilebert( 2025-08-14T21:53:55.7957199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7957660Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7958085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7958521Z layer_outputs = layer_module( 2025-08-14T21:53:55.7958945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.7959426Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.7959904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.7960383Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.7960556Z 2025-08-14T21:53:55.7960673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7961049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7961385Z return mod(**inputs) 2025-08-14T21:53:55.7961794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7962225Z outputs = self.mobilebert( 2025-08-14T21:53:55.7962639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7963070Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7963519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7963951Z layer_outputs = layer_module( 2025-08-14T21:53:55.7964365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.7964901Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.7965424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.7965879Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.7966036Z 2025-08-14T21:53:55.7966144Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7966536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7966884Z return mod(**inputs) 2025-08-14T21:53:55.7967306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7967739Z outputs = self.mobilebert( 2025-08-14T21:53:55.7968158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7968594Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7969021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7969474Z layer_outputs = layer_module( 2025-08-14T21:53:55.7969860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.7970343Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.7970811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.7971258Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.7971703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7972115Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7972265Z 2025-08-14T21:53:55.7972367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7972719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7973040Z return mod(**inputs) 2025-08-14T21:53:55.7973420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7973831Z outputs = self.mobilebert( 2025-08-14T21:53:55.7974224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7974640Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7975021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7975421Z layer_outputs = layer_module( 2025-08-14T21:53:55.7975817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.7976288Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.7976765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.7977215Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.7977668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.7978095Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.7978237Z 2025-08-14T21:53:55.7978338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7978684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7979014Z return mod(**inputs) 2025-08-14T21:53:55.7979380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7979778Z outputs = self.mobilebert( 2025-08-14T21:53:55.7980159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7980551Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7980961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7981374Z layer_outputs = layer_module( 2025-08-14T21:53:55.7981800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.7982271Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.7982750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.7983196Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.7983639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.7984077Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.7984526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7984947Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7985091Z 2025-08-14T21:53:55.7985200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7985539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7985856Z return mod(**inputs) 2025-08-14T21:53:55.7986234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7986627Z outputs = self.mobilebert( 2025-08-14T21:53:55.7987016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7987419Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7987815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7988211Z layer_outputs = layer_module( 2025-08-14T21:53:55.7988605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.7989149Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.7989677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.7990114Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.7990543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.7990955Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.7991092Z 2025-08-14T21:53:55.7991193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7991588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7991929Z return mod(**inputs) 2025-08-14T21:53:55.7992320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.7992724Z outputs = self.mobilebert( 2025-08-14T21:53:55.7993139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.7993597Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.7994026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.7994466Z layer_outputs = layer_module( 2025-08-14T21:53:55.7994933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.7995468Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.7996128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.7996636Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.7997144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.7997623Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.7998047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.7998461Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.7998601Z 2025-08-14T21:53:55.7998713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.7999054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.7999356Z return mod(**inputs) 2025-08-14T21:53:55.7999731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8000122Z outputs = self.mobilebert( 2025-08-14T21:53:55.8000495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8000887Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8001272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8001662Z layer_outputs = layer_module( 2025-08-14T21:53:55.8002083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8002501Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8002918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8003356Z self_outputs = self.self( 2025-08-14T21:53:55.8003752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8004188Z self.query(query_tensor) 2025-08-14T21:53:55.8004304Z 2025-08-14T21:53:55.8004413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8004766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8005077Z return mod(**inputs) 2025-08-14T21:53:55.8005456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8005861Z outputs = self.mobilebert( 2025-08-14T21:53:55.8006244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8006678Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8007070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8007483Z layer_outputs = layer_module( 2025-08-14T21:53:55.8007885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8008302Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8008870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8009280Z self_outputs = self.self( 2025-08-14T21:53:55.8009723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8010126Z self.key(key_tensor) 2025-08-14T21:53:55.8010233Z 2025-08-14T21:53:55.8010367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8010730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8011042Z return mod(**inputs) 2025-08-14T21:53:55.8011424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8011816Z outputs = self.mobilebert( 2025-08-14T21:53:55.8012203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8012603Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8013003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8013391Z layer_outputs = layer_module( 2025-08-14T21:53:55.8013782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8014190Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8014604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8015013Z self_outputs = self.self( 2025-08-14T21:53:55.8015410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8015818Z self.value(value_tensor) 2025-08-14T21:53:55.8015931Z 2025-08-14T21:53:55.8016015Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8016231Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8016465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8016820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8017134Z return mod(**inputs) 2025-08-14T21:53:55.8017519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8017918Z outputs = self.mobilebert( 2025-08-14T21:53:55.8018297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8018695Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8019090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8019487Z layer_outputs = layer_module( 2025-08-14T21:53:55.8019868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8020274Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8020709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8021147Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8021598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8022034Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8022168Z 2025-08-14T21:53:55.8022278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8022626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8022951Z return mod(**inputs) 2025-08-14T21:53:55.8023352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8023765Z outputs = self.mobilebert( 2025-08-14T21:53:55.8024173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8024586Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8024991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8025397Z layer_outputs = layer_module( 2025-08-14T21:53:55.8025801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8026296Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8026791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8027236Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8027685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8028106Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8028245Z 2025-08-14T21:53:55.8028354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8028702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8029030Z return mod(**inputs) 2025-08-14T21:53:55.8029420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8029827Z outputs = self.mobilebert( 2025-08-14T21:53:55.8030230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8030650Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8031083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8031517Z layer_outputs = layer_module( 2025-08-14T21:53:55.8031943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8032405Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8032849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8033327Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8033817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8034314Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8034802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8035276Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8035440Z 2025-08-14T21:53:55.8035552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8035994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8036360Z return mod(**inputs) 2025-08-14T21:53:55.8036787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8037225Z outputs = self.mobilebert( 2025-08-14T21:53:55.8037627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8038033Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8038466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8038880Z layer_outputs = layer_module( 2025-08-14T21:53:55.8039294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8039739Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8040170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8040628Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8041079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8041507Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8041648Z 2025-08-14T21:53:55.8041762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8042123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8042445Z return mod(**inputs) 2025-08-14T21:53:55.8042838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8043261Z outputs = self.mobilebert( 2025-08-14T21:53:55.8043679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8044122Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8044568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8044981Z layer_outputs = layer_module( 2025-08-14T21:53:55.8045401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8045863Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8046323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8046804Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8047264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8047723Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8047890Z 2025-08-14T21:53:55.8048002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8048352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8048677Z return mod(**inputs) 2025-08-14T21:53:55.8049073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8049493Z outputs = self.mobilebert( 2025-08-14T21:53:55.8049917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8050327Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8050736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8051192Z layer_outputs = layer_module( 2025-08-14T21:53:55.8051603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8052045Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8052500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8053046Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8053552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8054041Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8054189Z 2025-08-14T21:53:55.8054310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8054674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8055025Z return mod(**inputs) 2025-08-14T21:53:55.8055449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8055879Z outputs = self.mobilebert( 2025-08-14T21:53:55.8056303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8056740Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8057174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8057623Z layer_outputs = layer_module( 2025-08-14T21:53:55.8058052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8058508Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8058995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8059496Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8059990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8060561Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8061058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8061516Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8061673Z 2025-08-14T21:53:55.8061784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8062168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8062521Z return mod(**inputs) 2025-08-14T21:53:55.8062928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8063372Z outputs = self.mobilebert( 2025-08-14T21:53:55.8063791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8064221Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8064642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8065096Z layer_outputs = layer_module( 2025-08-14T21:53:55.8065520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8065977Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8066429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8066882Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8067311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8067715Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8067857Z 2025-08-14T21:53:55.8067974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8068325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8068644Z return mod(**inputs) 2025-08-14T21:53:55.8069034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8069437Z outputs = self.mobilebert( 2025-08-14T21:53:55.8069824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8070231Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8070630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8071039Z layer_outputs = layer_module( 2025-08-14T21:53:55.8071443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8071869Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8072298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8072760Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8073231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8073697Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8073878Z 2025-08-14T21:53:55.8073991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8074369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8074712Z return mod(**inputs) 2025-08-14T21:53:55.8075125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8075563Z outputs = self.mobilebert( 2025-08-14T21:53:55.8076086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8076532Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8076972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8077418Z layer_outputs = layer_module( 2025-08-14T21:53:55.8077825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8078241Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8078660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8079111Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8079562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8080006Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8080153Z 2025-08-14T21:53:55.8080256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8080611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8080946Z return mod(**inputs) 2025-08-14T21:53:55.8081331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8081739Z outputs = self.mobilebert( 2025-08-14T21:53:55.8082138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8082554Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8082960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8083373Z layer_outputs = layer_module( 2025-08-14T21:53:55.8083781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8084205Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8084633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8085094Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8085543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8085997Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8086452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8086880Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8087028Z 2025-08-14T21:53:55.8087130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8087487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8087809Z return mod(**inputs) 2025-08-14T21:53:55.8088195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8088600Z outputs = self.mobilebert( 2025-08-14T21:53:55.8089002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8089413Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8089809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8090221Z layer_outputs = layer_module( 2025-08-14T21:53:55.8090625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8091055Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8091479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8091930Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8092373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8092794Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8092932Z 2025-08-14T21:53:55.8093035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8093392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8093734Z return mod(**inputs) 2025-08-14T21:53:55.8094113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8094524Z outputs = self.mobilebert( 2025-08-14T21:53:55.8094954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8095411Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8095840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8096286Z layer_outputs = layer_module( 2025-08-14T21:53:55.8096734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8097167Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8097605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8098052Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8098498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8098937Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8099106Z 2025-08-14T21:53:55.8099210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8099567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8099887Z return mod(**inputs) 2025-08-14T21:53:55.8100282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8100715Z outputs = self.mobilebert( 2025-08-14T21:53:55.8101137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8101570Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8101991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8102420Z layer_outputs = layer_module( 2025-08-14T21:53:55.8102847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8103299Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8103750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8104240Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8104728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8105170Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8105325Z 2025-08-14T21:53:55.8105433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8105807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8106146Z return mod(**inputs) 2025-08-14T21:53:55.8106548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8106979Z outputs = self.mobilebert( 2025-08-14T21:53:55.8107392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8107812Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8108239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8108851Z layer_outputs = layer_module( 2025-08-14T21:53:55.8109289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8109740Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8110243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8110725Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8111206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8111677Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8112188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8112647Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8112802Z 2025-08-14T21:53:55.8112933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8113312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8113658Z return mod(**inputs) 2025-08-14T21:53:55.8114068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8114497Z outputs = self.mobilebert( 2025-08-14T21:53:55.8114921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8115356Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8115836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8116289Z layer_outputs = layer_module( 2025-08-14T21:53:55.8116711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8117202Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8117679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8118138Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8118286Z 2025-08-14T21:53:55.8118388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8118745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8119061Z return mod(**inputs) 2025-08-14T21:53:55.8119449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8119857Z outputs = self.mobilebert( 2025-08-14T21:53:55.8120244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8120655Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8121053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8121456Z layer_outputs = layer_module( 2025-08-14T21:53:55.8121845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8122306Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8122760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8123205Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8123408Z 2025-08-14T21:53:55.8123511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8123871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8124203Z return mod(**inputs) 2025-08-14T21:53:55.8124594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8125022Z outputs = self.mobilebert( 2025-08-14T21:53:55.8125416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8125827Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8126226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8126654Z layer_outputs = layer_module( 2025-08-14T21:53:55.8127056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8127568Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8128049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8128468Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8128618Z 2025-08-14T21:53:55.8128727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8129089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8129393Z return mod(**inputs) 2025-08-14T21:53:55.8129779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8130183Z outputs = self.mobilebert( 2025-08-14T21:53:55.8130563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8130973Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8131392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8131804Z layer_outputs = layer_module( 2025-08-14T21:53:55.8132193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8132711Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8133189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8133646Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8134089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8134507Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8134648Z 2025-08-14T21:53:55.8134754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8135088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8135400Z return mod(**inputs) 2025-08-14T21:53:55.8135772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8136167Z outputs = self.mobilebert( 2025-08-14T21:53:55.8136542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8136944Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8137338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8137762Z layer_outputs = layer_module( 2025-08-14T21:53:55.8138146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8138628Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8139126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8139572Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8140003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8140421Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8140553Z 2025-08-14T21:53:55.8140660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8140991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8141319Z return mod(**inputs) 2025-08-14T21:53:55.8141691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8142084Z outputs = self.mobilebert( 2025-08-14T21:53:55.8142452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8142836Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8143217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8143597Z layer_outputs = layer_module( 2025-08-14T21:53:55.8143979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8144449Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8144923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8145361Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8145808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8146251Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8146693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8147105Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8147257Z 2025-08-14T21:53:55.8147361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8147720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8148036Z return mod(**inputs) 2025-08-14T21:53:55.8148413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8148817Z outputs = self.mobilebert( 2025-08-14T21:53:55.8149206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8149600Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8149995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8150428Z layer_outputs = layer_module( 2025-08-14T21:53:55.8150829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8151347Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8151849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8152298Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8152771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8153227Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8153386Z 2025-08-14T21:53:55.8153501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8153885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8154244Z return mod(**inputs) 2025-08-14T21:53:55.8154690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8155125Z outputs = self.mobilebert( 2025-08-14T21:53:55.8155568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8156151Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8156613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8157076Z layer_outputs = layer_module( 2025-08-14T21:53:55.8157500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8157991Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8158483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8158929Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8159359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8159775Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8160178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8160595Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8160737Z 2025-08-14T21:53:55.8160838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8161188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8161507Z return mod(**inputs) 2025-08-14T21:53:55.8161889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8162288Z outputs = self.mobilebert( 2025-08-14T21:53:55.8190512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8190969Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8191379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8191798Z layer_outputs = layer_module( 2025-08-14T21:53:55.8192216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8192650Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8193067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8193493Z self_outputs = self.self( 2025-08-14T21:53:55.8193927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8194441Z self.query(query_tensor) 2025-08-14T21:53:55.8194571Z 2025-08-14T21:53:55.8194684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8195058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8195444Z return mod(**inputs) 2025-08-14T21:53:55.8196111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8196558Z outputs = self.mobilebert( 2025-08-14T21:53:55.8197008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8197462Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8197939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8198381Z layer_outputs = layer_module( 2025-08-14T21:53:55.8198843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8199291Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8199755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8200192Z self_outputs = self.self( 2025-08-14T21:53:55.8200628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8201071Z self.key(key_tensor) 2025-08-14T21:53:55.8201188Z 2025-08-14T21:53:55.8201310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8201695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8202045Z return mod(**inputs) 2025-08-14T21:53:55.8202467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8202909Z outputs = self.mobilebert( 2025-08-14T21:53:55.8203330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8203777Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8204214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8204657Z layer_outputs = layer_module( 2025-08-14T21:53:55.8205098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8205561Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8206014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8206447Z self_outputs = self.self( 2025-08-14T21:53:55.8206845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8207261Z self.value(value_tensor) 2025-08-14T21:53:55.8207378Z 2025-08-14T21:53:55.8207464Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8207685Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8207926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8208287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8208607Z return mod(**inputs) 2025-08-14T21:53:55.8209204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8209686Z outputs = self.mobilebert( 2025-08-14T21:53:55.8210081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8210496Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8210909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8211348Z layer_outputs = layer_module( 2025-08-14T21:53:55.8211744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8212168Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8212589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8213071Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8213549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8213977Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8214117Z 2025-08-14T21:53:55.8214232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8214582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8214906Z return mod(**inputs) 2025-08-14T21:53:55.8215291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8215702Z outputs = self.mobilebert( 2025-08-14T21:53:55.8216095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8216507Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8216910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8217319Z layer_outputs = layer_module( 2025-08-14T21:53:55.8217713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8218209Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8218710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8219151Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8219598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8220017Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8220155Z 2025-08-14T21:53:55.8220270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8220624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8220949Z return mod(**inputs) 2025-08-14T21:53:55.8221347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8221746Z outputs = self.mobilebert( 2025-08-14T21:53:55.8222124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8222524Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8222921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8223314Z layer_outputs = layer_module( 2025-08-14T21:53:55.8223714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8224152Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8224571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8225008Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8225467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8225920Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8226374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8226797Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8226950Z 2025-08-14T21:53:55.8227068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8227420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8227751Z return mod(**inputs) 2025-08-14T21:53:55.8228122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8228522Z outputs = self.mobilebert( 2025-08-14T21:53:55.8228908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8229306Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8229698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8230096Z layer_outputs = layer_module( 2025-08-14T21:53:55.8230496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8230924Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8231378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8231853Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8232325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8232765Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8232920Z 2025-08-14T21:53:55.8233029Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8233407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8233741Z return mod(**inputs) 2025-08-14T21:53:55.8234151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8234591Z outputs = self.mobilebert( 2025-08-14T21:53:55.8235015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8235443Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8235932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8236376Z layer_outputs = layer_module( 2025-08-14T21:53:55.8236809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8237268Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8237739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8238213Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8238674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8239171Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8239352Z 2025-08-14T21:53:55.8239463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8239860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8240192Z return mod(**inputs) 2025-08-14T21:53:55.8240605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8241038Z outputs = self.mobilebert( 2025-08-14T21:53:55.8241456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8241916Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8242348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8242795Z layer_outputs = layer_module( 2025-08-14T21:53:55.8243213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8243667Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8244148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8244666Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8245107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8245517Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8245658Z 2025-08-14T21:53:55.8245757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8246106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8246412Z return mod(**inputs) 2025-08-14T21:53:55.8246792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8247195Z outputs = self.mobilebert( 2025-08-14T21:53:55.8247585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8247994Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8248405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8248800Z layer_outputs = layer_module( 2025-08-14T21:53:55.8249181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8249599Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8250014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8250459Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8250899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8251341Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8251781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8252193Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8252344Z 2025-08-14T21:53:55.8252446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8252794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8253127Z return mod(**inputs) 2025-08-14T21:53:55.8253497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8253892Z outputs = self.mobilebert( 2025-08-14T21:53:55.8254296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8254689Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8255071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8255463Z layer_outputs = layer_module( 2025-08-14T21:53:55.8255907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8256314Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8256760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8257194Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8257625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8258028Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8258171Z 2025-08-14T21:53:55.8258270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8258618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8258932Z return mod(**inputs) 2025-08-14T21:53:55.8259303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8259705Z outputs = self.mobilebert( 2025-08-14T21:53:55.8260095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8260484Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8260878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8261275Z layer_outputs = layer_module( 2025-08-14T21:53:55.8261673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8262072Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8262473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8262896Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8263313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8263732Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8263895Z 2025-08-14T21:53:55.8263993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8264331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8264626Z return mod(**inputs) 2025-08-14T21:53:55.8264993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8265382Z outputs = self.mobilebert( 2025-08-14T21:53:55.8265758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8266157Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8266560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8266970Z layer_outputs = layer_module( 2025-08-14T21:53:55.8267348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8267765Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8268170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8268605Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8269049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8269476Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8269619Z 2025-08-14T21:53:55.8269719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8270066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8270388Z return mod(**inputs) 2025-08-14T21:53:55.8270774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8271184Z outputs = self.mobilebert( 2025-08-14T21:53:55.8271580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8272008Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8272442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8272883Z layer_outputs = layer_module( 2025-08-14T21:53:55.8273299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8273750Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8274209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8274692Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8275170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8275715Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8276217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8276672Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8276837Z 2025-08-14T21:53:55.8276938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8277278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8277591Z return mod(**inputs) 2025-08-14T21:53:55.8277958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8278349Z outputs = self.mobilebert( 2025-08-14T21:53:55.8278736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8279139Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8279529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8279943Z layer_outputs = layer_module( 2025-08-14T21:53:55.8280328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8280744Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8281170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8281603Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8282030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8282442Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8282585Z 2025-08-14T21:53:55.8282686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8283031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8283345Z return mod(**inputs) 2025-08-14T21:53:55.8283741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8284155Z outputs = self.mobilebert( 2025-08-14T21:53:55.8284574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8284992Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8285377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8285775Z layer_outputs = layer_module( 2025-08-14T21:53:55.8286168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8286585Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8286999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8287437Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8287867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8288299Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8288468Z 2025-08-14T21:53:55.8288574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8288925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8289238Z return mod(**inputs) 2025-08-14T21:53:55.8289609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8290004Z outputs = self.mobilebert( 2025-08-14T21:53:55.8290389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8290781Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8291178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8291579Z layer_outputs = layer_module( 2025-08-14T21:53:55.8291969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8292381Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8292794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8293243Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8293689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8294102Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8294249Z 2025-08-14T21:53:55.8294352Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8294719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8295033Z return mod(**inputs) 2025-08-14T21:53:55.8295421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8295860Z outputs = self.mobilebert( 2025-08-14T21:53:55.8296257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8296674Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8297103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8297179Z layer_outputs = layer_module( 2025-08-14T21:53:55.8297539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8297642Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8297954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8298085Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8298385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8298512Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8298786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8298885Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8298889Z 2025-08-14T21:53:55.8298993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8299188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8299263Z return mod(**inputs) 2025-08-14T21:53:55.8299543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8299618Z outputs = self.mobilebert( 2025-08-14T21:53:55.8299919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8299996Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8300295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8300368Z layer_outputs = layer_module( 2025-08-14T21:53:55.8300659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8300800Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8301092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8301182Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8301186Z 2025-08-14T21:53:55.8301289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8301485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8301557Z return mod(**inputs) 2025-08-14T21:53:55.8301833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8301902Z outputs = self.mobilebert( 2025-08-14T21:53:55.8302193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8302269Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8302600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8302675Z layer_outputs = layer_module( 2025-08-14T21:53:55.8302961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8303104Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8303377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8303497Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8303501Z 2025-08-14T21:53:55.8303605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8303814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8303889Z return mod(**inputs) 2025-08-14T21:53:55.8304186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8304259Z outputs = self.mobilebert( 2025-08-14T21:53:55.8304540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8304614Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8304896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8304968Z layer_outputs = layer_module( 2025-08-14T21:53:55.8305242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8305407Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8305678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8305782Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8305785Z 2025-08-14T21:53:55.8305887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8306081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8306158Z return mod(**inputs) 2025-08-14T21:53:55.8306437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8306505Z outputs = self.mobilebert( 2025-08-14T21:53:55.8306783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8306854Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8307135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8307207Z layer_outputs = layer_module( 2025-08-14T21:53:55.8307477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8307639Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8307917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8308045Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8308318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8308412Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8308415Z 2025-08-14T21:53:55.8308526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8308914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8308991Z return mod(**inputs) 2025-08-14T21:53:55.8309270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8309384Z outputs = self.mobilebert( 2025-08-14T21:53:55.8309670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8309741Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8310018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8310098Z layer_outputs = layer_module( 2025-08-14T21:53:55.8310398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8310584Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8310857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8310979Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8311260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8311344Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8311348Z 2025-08-14T21:53:55.8311455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8311651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8311718Z return mod(**inputs) 2025-08-14T21:53:55.8312004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8312077Z outputs = self.mobilebert( 2025-08-14T21:53:55.8312348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8312428Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8312728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8312808Z layer_outputs = layer_module( 2025-08-14T21:53:55.8313108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8313270Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8313578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8313708Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8314008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8314128Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8314401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8314505Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8314509Z 2025-08-14T21:53:55.8314614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8314823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8314892Z return mod(**inputs) 2025-08-14T21:53:55.8315183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8315289Z outputs = self.mobilebert( 2025-08-14T21:53:55.8315579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8315721Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8316046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8316119Z layer_outputs = layer_module( 2025-08-14T21:53:55.8316417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8316592Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8316907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8317051Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8317359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8317457Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8317463Z 2025-08-14T21:53:55.8317571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8317775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8317854Z return mod(**inputs) 2025-08-14T21:53:55.8318147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8318223Z outputs = self.mobilebert( 2025-08-14T21:53:55.8318525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8318604Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8318903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8318977Z layer_outputs = layer_module( 2025-08-14T21:53:55.8319267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8319443Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8319732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8319854Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8320144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8320234Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8320537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8320632Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8320636Z 2025-08-14T21:53:55.8320748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8320953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8321022Z return mod(**inputs) 2025-08-14T21:53:55.8321322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8321396Z outputs = self.mobilebert( 2025-08-14T21:53:55.8321695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8321779Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8322098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8322179Z layer_outputs = layer_module( 2025-08-14T21:53:55.8322472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8322575Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8322850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8322919Z self_outputs = self.self( 2025-08-14T21:53:55.8323185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8323261Z self.query(query_tensor) 2025-08-14T21:53:55.8323281Z 2025-08-14T21:53:55.8323381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8323578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8324392Z return mod(**inputs) 2025-08-14T21:53:55.8324673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8324753Z outputs = self.mobilebert( 2025-08-14T21:53:55.8325020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8325095Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8325365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8325433Z layer_outputs = layer_module( 2025-08-14T21:53:55.8325704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8325789Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8326055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8326130Z self_outputs = self.self( 2025-08-14T21:53:55.8326396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8326469Z self.key(key_tensor) 2025-08-14T21:53:55.8326472Z 2025-08-14T21:53:55.8326572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8326760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8326831Z return mod(**inputs) 2025-08-14T21:53:55.8327100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8327176Z outputs = self.mobilebert( 2025-08-14T21:53:55.8327445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8327515Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8327789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8327858Z layer_outputs = layer_module( 2025-08-14T21:53:55.8328124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8328212Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8328479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8328555Z self_outputs = self.self( 2025-08-14T21:53:55.8328819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8328909Z self.value(value_tensor) 2025-08-14T21:53:55.8328913Z 2025-08-14T21:53:55.8329001Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8329079Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8329178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8329390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8329453Z return mod(**inputs) 2025-08-14T21:53:55.8329731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8329799Z outputs = self.mobilebert( 2025-08-14T21:53:55.8330079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8330157Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8330438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8330514Z layer_outputs = layer_module( 2025-08-14T21:53:55.8330776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8330859Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8331128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8331247Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8331512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8331603Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8331606Z 2025-08-14T21:53:55.8331703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8331900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8331964Z return mod(**inputs) 2025-08-14T21:53:55.8332237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8332314Z outputs = self.mobilebert( 2025-08-14T21:53:55.8332578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8332654Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8332921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8332990Z layer_outputs = layer_module( 2025-08-14T21:53:55.8333263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8333420Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8333684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8333799Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8334062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8334148Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8334152Z 2025-08-14T21:53:55.8334249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8334436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8334510Z return mod(**inputs) 2025-08-14T21:53:55.8334776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8334872Z outputs = self.mobilebert( 2025-08-14T21:53:55.8335144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8335230Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8335514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8335584Z layer_outputs = layer_module( 2025-08-14T21:53:55.8335871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8335958Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8336244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8336371Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8336649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8336770Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8337045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8337133Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8337137Z 2025-08-14T21:53:55.8337244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8337432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8337497Z return mod(**inputs) 2025-08-14T21:53:55.8337782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8337854Z outputs = self.mobilebert( 2025-08-14T21:53:55.8338136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8338207Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8338482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8338558Z layer_outputs = layer_module( 2025-08-14T21:53:55.8338830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8338932Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8339208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8339316Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8339591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8339671Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8339674Z 2025-08-14T21:53:55.8339769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8339970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8340034Z return mod(**inputs) 2025-08-14T21:53:55.8340308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8340374Z outputs = self.mobilebert( 2025-08-14T21:53:55.8340638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8340713Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8340991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8341061Z layer_outputs = layer_module( 2025-08-14T21:53:55.8341334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8341444Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8341724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8341834Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8342105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8342238Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8342242Z 2025-08-14T21:53:55.8342343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8342568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8342635Z return mod(**inputs) 2025-08-14T21:53:55.8342916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8342995Z outputs = self.mobilebert( 2025-08-14T21:53:55.8343272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8343344Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8343624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8343695Z layer_outputs = layer_module( 2025-08-14T21:53:55.8343977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8344074Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8344349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8344480Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8344756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8344845Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8344848Z 2025-08-14T21:53:55.8344952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8345147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8345223Z return mod(**inputs) 2025-08-14T21:53:55.8345503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8345577Z outputs = self.mobilebert( 2025-08-14T21:53:55.8345860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8345929Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8346216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8346284Z layer_outputs = layer_module( 2025-08-14T21:53:55.8346563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8346663Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8346946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8347095Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8347371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8347492Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8347784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8347876Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8347879Z 2025-08-14T21:53:55.8347986Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8348178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8348243Z return mod(**inputs) 2025-08-14T21:53:55.8348542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8348615Z outputs = self.mobilebert( 2025-08-14T21:53:55.8348909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8348989Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8349263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8349341Z layer_outputs = layer_module( 2025-08-14T21:53:55.8349617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8349708Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8349988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8350097Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8350379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8350462Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8350466Z 2025-08-14T21:53:55.8350566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8350765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8350829Z return mod(**inputs) 2025-08-14T21:53:55.8351103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8351178Z outputs = self.mobilebert( 2025-08-14T21:53:55.8351448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8351524Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8351797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8351864Z layer_outputs = layer_module( 2025-08-14T21:53:55.8352140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8352232Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8352502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8352618Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8352887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8353003Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8353007Z 2025-08-14T21:53:55.8353127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8353324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8353397Z return mod(**inputs) 2025-08-14T21:53:55.8353682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8353782Z outputs = self.mobilebert( 2025-08-14T21:53:55.8354083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8354156Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8354456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8354545Z layer_outputs = layer_module( 2025-08-14T21:53:55.8354844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8354961Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8355253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8355389Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8355778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8355877Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8355889Z 2025-08-14T21:53:55.8355998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8356208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8356290Z return mod(**inputs) 2025-08-14T21:53:55.8356592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8356669Z outputs = self.mobilebert( 2025-08-14T21:53:55.8356970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8357047Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8357362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8357437Z layer_outputs = layer_module( 2025-08-14T21:53:55.8357724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8357826Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8358125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8358256Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8358565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8358695Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8358995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8359090Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8359094Z 2025-08-14T21:53:55.8359199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8359425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8359493Z return mod(**inputs) 2025-08-14T21:53:55.8359795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8359895Z outputs = self.mobilebert( 2025-08-14T21:53:55.8360195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8360278Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8360577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8360670Z layer_outputs = layer_module( 2025-08-14T21:53:55.8360966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8361062Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8361383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8361501Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8361819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8361916Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8361919Z 2025-08-14T21:53:55.8362025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8362247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8362315Z return mod(**inputs) 2025-08-14T21:53:55.8362603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8362686Z outputs = self.mobilebert( 2025-08-14T21:53:55.8362991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8363068Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8363371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8363445Z layer_outputs = layer_module( 2025-08-14T21:53:55.8363745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8363844Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8364141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8364263Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8364609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8364734Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8364738Z 2025-08-14T21:53:55.8364845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8365054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8365130Z return mod(**inputs) 2025-08-14T21:53:55.8365421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8365495Z outputs = self.mobilebert( 2025-08-14T21:53:55.8365795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8365865Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8366152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8366220Z layer_outputs = layer_module( 2025-08-14T21:53:55.8366493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8366612Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8366890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8367028Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8367323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8367404Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8367408Z 2025-08-14T21:53:55.8367515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8367711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8367811Z return mod(**inputs) 2025-08-14T21:53:55.8368092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8368164Z outputs = self.mobilebert( 2025-08-14T21:53:55.8368461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8368533Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8368809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8368886Z layer_outputs = layer_module( 2025-08-14T21:53:55.8369161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8369259Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8369541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8369664Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8369945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8370064Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8370344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8370437Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8370440Z 2025-08-14T21:53:55.8370542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8370746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8370811Z return mod(**inputs) 2025-08-14T21:53:55.8371087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8371167Z outputs = self.mobilebert( 2025-08-14T21:53:55.8371438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8371518Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8371789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8371860Z layer_outputs = layer_module( 2025-08-14T21:53:55.8372137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8372258Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8372542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8372626Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8372646Z 2025-08-14T21:53:55.8372747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8372949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8373014Z return mod(**inputs) 2025-08-14T21:53:55.8373287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8373379Z outputs = self.mobilebert( 2025-08-14T21:53:55.8373655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8373732Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8374023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8374094Z layer_outputs = layer_module( 2025-08-14T21:53:55.8374378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8374517Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8374797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8374906Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8374910Z 2025-08-14T21:53:55.8375012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8375214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8375279Z return mod(**inputs) 2025-08-14T21:53:55.8375556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8375633Z outputs = self.mobilebert( 2025-08-14T21:53:55.8375904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8375984Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8376256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8376326Z layer_outputs = layer_module( 2025-08-14T21:53:55.8376607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8376767Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8377047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8377144Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8377148Z 2025-08-14T21:53:55.8377247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8377454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8377521Z return mod(**inputs) 2025-08-14T21:53:55.8377797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8377876Z outputs = self.mobilebert( 2025-08-14T21:53:55.8378149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8378228Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8378505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8378575Z layer_outputs = layer_module( 2025-08-14T21:53:55.8378861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8379043Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8379328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8379447Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8379733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8379831Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8379834Z 2025-08-14T21:53:55.8379942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8380133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8380211Z return mod(**inputs) 2025-08-14T21:53:55.8380476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8380568Z outputs = self.mobilebert( 2025-08-14T21:53:55.8380831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8380898Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8381164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8381232Z layer_outputs = layer_module( 2025-08-14T21:53:55.8381500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8381647Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8381908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8382034Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8382290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8382379Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8382383Z 2025-08-14T21:53:55.8382478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8382661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8382730Z return mod(**inputs) 2025-08-14T21:53:55.8382992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8383058Z outputs = self.mobilebert( 2025-08-14T21:53:55.8383326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8383397Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8383670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8383738Z layer_outputs = layer_module( 2025-08-14T21:53:55.8384013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8384167Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8384427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8384549Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8384810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8384939Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8385209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8385299Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8385316Z 2025-08-14T21:53:55.8385420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8385609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8385672Z return mod(**inputs) 2025-08-14T21:53:55.8385954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8386021Z outputs = self.mobilebert( 2025-08-14T21:53:55.8386292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8386370Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8386647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8386722Z layer_outputs = layer_module( 2025-08-14T21:53:55.8386986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8387142Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8387417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8387525Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8387808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8387893Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8387896Z 2025-08-14T21:53:55.8387998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8388197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8388262Z return mod(**inputs) 2025-08-14T21:53:55.8388540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8388616Z outputs = self.mobilebert( 2025-08-14T21:53:55.8388893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8388973Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8389245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8389323Z layer_outputs = layer_module( 2025-08-14T21:53:55.8389597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8389750Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8390023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8390131Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8390395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8390487Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8390752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8390841Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8390869Z 2025-08-14T21:53:55.8390967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8391154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8391223Z return mod(**inputs) 2025-08-14T21:53:55.8391492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8391585Z outputs = self.mobilebert( 2025-08-14T21:53:55.8391858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8391928Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8392204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8392294Z layer_outputs = layer_module( 2025-08-14T21:53:55.8392567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8392681Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8392958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8393031Z self_outputs = self.self( 2025-08-14T21:53:55.8393340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8393416Z self.query(query_tensor) 2025-08-14T21:53:55.8393420Z 2025-08-14T21:53:55.8393534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8393741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8393811Z return mod(**inputs) 2025-08-14T21:53:55.8394112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8394187Z outputs = self.mobilebert( 2025-08-14T21:53:55.8394486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8394562Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8394851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8394933Z layer_outputs = layer_module( 2025-08-14T21:53:55.8395221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8395309Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8395622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8395772Z self_outputs = self.self( 2025-08-14T21:53:55.8396082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8396154Z self.key(key_tensor) 2025-08-14T21:53:55.8396158Z 2025-08-14T21:53:55.8396265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8396483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8396552Z return mod(**inputs) 2025-08-14T21:53:55.8396854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8396930Z outputs = self.mobilebert( 2025-08-14T21:53:55.8397233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8397319Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8397641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8397715Z layer_outputs = layer_module( 2025-08-14T21:53:55.8398014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8398161Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8398457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8398529Z self_outputs = self.self( 2025-08-14T21:53:55.8398815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8398897Z self.value(value_tensor) 2025-08-14T21:53:55.8398916Z 2025-08-14T21:53:55.8399005Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8399095Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8399203Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8399421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8399501Z return mod(**inputs) 2025-08-14T21:53:55.8399792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8399870Z outputs = self.mobilebert( 2025-08-14T21:53:55.8400166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8400240Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8400540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8400619Z layer_outputs = layer_module( 2025-08-14T21:53:55.8400908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8401013Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8401301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8401432Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8401728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8401816Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8401820Z 2025-08-14T21:53:55.8401932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8402136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8402205Z return mod(**inputs) 2025-08-14T21:53:55.8402505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8402582Z outputs = self.mobilebert( 2025-08-14T21:53:55.8402879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8402958Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8403244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8403325Z layer_outputs = layer_module( 2025-08-14T21:53:55.8403614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8403784Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8404082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8404234Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8404511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8404594Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8404614Z 2025-08-14T21:53:55.8404716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8404915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8404979Z return mod(**inputs) 2025-08-14T21:53:55.8405261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8405333Z outputs = self.mobilebert( 2025-08-14T21:53:55.8405620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8405701Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8405991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8406062Z layer_outputs = layer_module( 2025-08-14T21:53:55.8406342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8406425Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8406705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8406835Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8407099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8407230Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8407498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8407592Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8407597Z 2025-08-14T21:53:55.8407696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8407884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8407953Z return mod(**inputs) 2025-08-14T21:53:55.8408223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8408301Z outputs = self.mobilebert( 2025-08-14T21:53:55.8408564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8408787Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8409073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8409142Z layer_outputs = layer_module( 2025-08-14T21:53:55.8409420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8409523Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8409791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8409911Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8410188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8410272Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8410323Z 2025-08-14T21:53:55.8410443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8410629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8410699Z return mod(**inputs) 2025-08-14T21:53:55.8410965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8411056Z outputs = self.mobilebert( 2025-08-14T21:53:55.8411329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8411399Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8411664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8411765Z layer_outputs = layer_module( 2025-08-14T21:53:55.8412037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8412158Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8412424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8412532Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8412801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8412909Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8412913Z 2025-08-14T21:53:55.8413019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8413211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8413274Z return mod(**inputs) 2025-08-14T21:53:55.8413555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8413626Z outputs = self.mobilebert( 2025-08-14T21:53:55.8413895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8413975Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8414244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8414322Z layer_outputs = layer_module( 2025-08-14T21:53:55.8414597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8414691Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8414971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8415099Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8415373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8415455Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8415459Z 2025-08-14T21:53:55.8415559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8415757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8415820Z return mod(**inputs) 2025-08-14T21:53:55.8416095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8416177Z outputs = self.mobilebert( 2025-08-14T21:53:55.8416448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8416555Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8416838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8416909Z layer_outputs = layer_module( 2025-08-14T21:53:55.8417219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8417309Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8417585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8417705Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8417988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8418117Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8418401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8418492Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8418503Z 2025-08-14T21:53:55.8418602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8418792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8418863Z return mod(**inputs) 2025-08-14T21:53:55.8419135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8419206Z outputs = self.mobilebert( 2025-08-14T21:53:55.8419480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8419551Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8419827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8419897Z layer_outputs = layer_module( 2025-08-14T21:53:55.8420170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8420271Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8420554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8420659Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8420987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8421073Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8421078Z 2025-08-14T21:53:55.8421185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8421380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8421444Z return mod(**inputs) 2025-08-14T21:53:55.8421727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8421797Z outputs = self.mobilebert( 2025-08-14T21:53:55.8422078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8422150Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8422424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8422501Z layer_outputs = layer_module( 2025-08-14T21:53:55.8422830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8422942Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8423224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8423347Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8423628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8423737Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8423741Z 2025-08-14T21:53:55.8423843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8424064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8424129Z return mod(**inputs) 2025-08-14T21:53:55.8424411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8424498Z outputs = self.mobilebert( 2025-08-14T21:53:55.8424775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8424856Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8425137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8425206Z layer_outputs = layer_module( 2025-08-14T21:53:55.8425490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8425581Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8425866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8425991Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8426269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8426361Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8426366Z 2025-08-14T21:53:55.8426467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8426670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8426735Z return mod(**inputs) 2025-08-14T21:53:55.8427013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8427090Z outputs = self.mobilebert( 2025-08-14T21:53:55.8427390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8427476Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8427831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8427901Z layer_outputs = layer_module( 2025-08-14T21:53:55.8428187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8428281Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8428556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8428688Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8428967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8429116Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8429393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8429485Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8429489Z 2025-08-14T21:53:55.8429612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8429805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8429878Z return mod(**inputs) 2025-08-14T21:53:55.8430153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8430224Z outputs = self.mobilebert( 2025-08-14T21:53:55.8430518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8430592Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8430885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8430967Z layer_outputs = layer_module( 2025-08-14T21:53:55.8431241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8431340Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8431617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8431732Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8432014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8432095Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8432100Z 2025-08-14T21:53:55.8432206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8432400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8432462Z return mod(**inputs) 2025-08-14T21:53:55.8432757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8432830Z outputs = self.mobilebert( 2025-08-14T21:53:55.8433130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8433211Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8433511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8433589Z layer_outputs = layer_module( 2025-08-14T21:53:55.8433877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8433977Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8434272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8434391Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8434685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8434798Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8434802Z 2025-08-14T21:53:55.8434908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8435124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8435191Z return mod(**inputs) 2025-08-14T21:53:55.8435480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8435579Z outputs = self.mobilebert( 2025-08-14T21:53:55.8435938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8436051Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8436354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8436427Z layer_outputs = layer_module( 2025-08-14T21:53:55.8436726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8436823Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8437156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8437298Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8437598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8437690Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8437696Z 2025-08-14T21:53:55.8437797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8437999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8438063Z return mod(**inputs) 2025-08-14T21:53:55.8438336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8438408Z outputs = self.mobilebert( 2025-08-14T21:53:55.8438687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8438759Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8439041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8439108Z layer_outputs = layer_module( 2025-08-14T21:53:55.8439400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8439500Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8439790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8439925Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8440215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8440351Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8440647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8440741Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8440744Z 2025-08-14T21:53:55.8440862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8441070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8441138Z return mod(**inputs) 2025-08-14T21:53:55.8441439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8441510Z outputs = self.mobilebert( 2025-08-14T21:53:55.8441807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8441883Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8442196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8442277Z layer_outputs = layer_module( 2025-08-14T21:53:55.8442568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8442718Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8443005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8443090Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8443094Z 2025-08-14T21:53:55.8443206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8443430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8443502Z return mod(**inputs) 2025-08-14T21:53:55.8443824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8443899Z outputs = self.mobilebert( 2025-08-14T21:53:55.8444192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8444266Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8444553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8444632Z layer_outputs = layer_module( 2025-08-14T21:53:55.8444918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8445049Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8445338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8445454Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8445457Z 2025-08-14T21:53:55.8445569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8445773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8445844Z return mod(**inputs) 2025-08-14T21:53:55.8446140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8446214Z outputs = self.mobilebert( 2025-08-14T21:53:55.8446507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8446582Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8446868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8446953Z layer_outputs = layer_module( 2025-08-14T21:53:55.8447239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8447415Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8447706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8447806Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8447810Z 2025-08-14T21:53:55.8447930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8448121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8448187Z return mod(**inputs) 2025-08-14T21:53:55.8448470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8448558Z outputs = self.mobilebert( 2025-08-14T21:53:55.8448840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8448925Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8449199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8449276Z layer_outputs = layer_module( 2025-08-14T21:53:55.8449548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8449714Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8450007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8450149Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8450433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8450524Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8450529Z 2025-08-14T21:53:55.8450635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8450830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8450896Z return mod(**inputs) 2025-08-14T21:53:55.8451181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8451250Z outputs = self.mobilebert( 2025-08-14T21:53:55.8451526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8451606Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8451882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8451959Z layer_outputs = layer_module( 2025-08-14T21:53:55.8452234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8452385Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8452669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8452788Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8453075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8453162Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8453166Z 2025-08-14T21:53:55.8453269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8453472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8453539Z return mod(**inputs) 2025-08-14T21:53:55.8453816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8453896Z outputs = self.mobilebert( 2025-08-14T21:53:55.8454170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8454248Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8454526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8454616Z layer_outputs = layer_module( 2025-08-14T21:53:55.8454899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8455052Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8455351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8455474Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8455750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8455877Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8456169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8456263Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8456272Z 2025-08-14T21:53:55.8456391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8456586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8456656Z return mod(**inputs) 2025-08-14T21:53:55.8456932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8457001Z outputs = self.mobilebert( 2025-08-14T21:53:55.8457282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8457353Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8457630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8457700Z layer_outputs = layer_module( 2025-08-14T21:53:55.8457974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8458138Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8458409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8458525Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8458796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8458877Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8458881Z 2025-08-14T21:53:55.8458988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8459185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8459252Z return mod(**inputs) 2025-08-14T21:53:55.8459536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8459605Z outputs = self.mobilebert( 2025-08-14T21:53:55.8459883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8459955Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8460228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8460307Z layer_outputs = layer_module( 2025-08-14T21:53:55.8460579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8460743Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8461036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8461144Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8461422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8461523Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8461795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8461894Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8461897Z 2025-08-14T21:53:55.8461998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8462218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8462285Z return mod(**inputs) 2025-08-14T21:53:55.8462582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8462664Z outputs = self.mobilebert( 2025-08-14T21:53:55.8462938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8463019Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8463295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8463365Z layer_outputs = layer_module( 2025-08-14T21:53:55.8463656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8463739Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8463998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8464076Z self_outputs = self.self( 2025-08-14T21:53:55.8464336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8464411Z self.query(query_tensor) 2025-08-14T21:53:55.8464416Z 2025-08-14T21:53:55.8464514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8464699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8464769Z return mod(**inputs) 2025-08-14T21:53:55.8465032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8465104Z outputs = self.mobilebert( 2025-08-14T21:53:55.8465366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8465436Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8465711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8465778Z layer_outputs = layer_module( 2025-08-14T21:53:55.8466048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8466141Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8466417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8466492Z self_outputs = self.self( 2025-08-14T21:53:55.8466773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8466839Z self.key(key_tensor) 2025-08-14T21:53:55.8466876Z 2025-08-14T21:53:55.8466988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8467192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8467263Z return mod(**inputs) 2025-08-14T21:53:55.8467531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8467623Z outputs = self.mobilebert( 2025-08-14T21:53:55.8467895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8467965Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8468228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8468318Z layer_outputs = layer_module( 2025-08-14T21:53:55.8468584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8468693Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8468962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8469031Z self_outputs = self.self( 2025-08-14T21:53:55.8469304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8469372Z self.value(value_tensor) 2025-08-14T21:53:55.8469376Z 2025-08-14T21:53:55.8469461Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8469536Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8469637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8469831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8469893Z return mod(**inputs) 2025-08-14T21:53:55.8470166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8470239Z outputs = self.mobilebert( 2025-08-14T21:53:55.8470501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8470578Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8470844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8470912Z layer_outputs = layer_module( 2025-08-14T21:53:55.8471197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8471281Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8471550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8471685Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8471955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8472046Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8472049Z 2025-08-14T21:53:55.8472149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8472342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8472413Z return mod(**inputs) 2025-08-14T21:53:55.8472689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8472766Z outputs = self.mobilebert( 2025-08-14T21:53:55.8473050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8473146Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8473439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8473512Z layer_outputs = layer_module( 2025-08-14T21:53:55.8473820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8473994Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8474295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8474416Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8474734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8474823Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8474842Z 2025-08-14T21:53:55.8474959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8475164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8475241Z return mod(**inputs) 2025-08-14T21:53:55.8475532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8475606Z outputs = self.mobilebert( 2025-08-14T21:53:55.8476000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8476081Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8476399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8476485Z layer_outputs = layer_module( 2025-08-14T21:53:55.8476795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8476894Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8477185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8477323Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8477607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8477732Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8478027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8478120Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8478125Z 2025-08-14T21:53:55.8478227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8478427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8478494Z return mod(**inputs) 2025-08-14T21:53:55.8478767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8478844Z outputs = self.mobilebert( 2025-08-14T21:53:55.8479110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8479189Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8479460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8479531Z layer_outputs = layer_module( 2025-08-14T21:53:55.8479838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8479929Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8480203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8480329Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8480593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8480683Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8480686Z 2025-08-14T21:53:55.8480784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8480993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8481058Z return mod(**inputs) 2025-08-14T21:53:55.8481346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8481424Z outputs = self.mobilebert( 2025-08-14T21:53:55.8481696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8481765Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8482044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8482111Z layer_outputs = layer_module( 2025-08-14T21:53:55.8482386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8482480Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8482749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8482868Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8483143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8483261Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8483265Z 2025-08-14T21:53:55.8483367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8483561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8483633Z return mod(**inputs) 2025-08-14T21:53:55.8483913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8483987Z outputs = self.mobilebert( 2025-08-14T21:53:55.8484273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8484348Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8484631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8484701Z layer_outputs = layer_module( 2025-08-14T21:53:55.8484988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8485087Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8485354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8485482Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8485752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8485850Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8485853Z 2025-08-14T21:53:55.8485960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8486148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8486237Z return mod(**inputs) 2025-08-14T21:53:55.8486523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8486592Z outputs = self.mobilebert( 2025-08-14T21:53:55.8486875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8486944Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8487235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8487315Z layer_outputs = layer_module( 2025-08-14T21:53:55.8487603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8487702Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8487972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8488096Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8488372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8488490Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8488766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8488862Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8488867Z 2025-08-14T21:53:55.8488967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8489165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8489230Z return mod(**inputs) 2025-08-14T21:53:55.8489508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8489586Z outputs = self.mobilebert( 2025-08-14T21:53:55.8489857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8489935Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8490208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8490278Z layer_outputs = layer_module( 2025-08-14T21:53:55.8490558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8490649Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8490918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8491035Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8491306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8491396Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8491399Z 2025-08-14T21:53:55.8491499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8491693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8491765Z return mod(**inputs) 2025-08-14T21:53:55.8492062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8492141Z outputs = self.mobilebert( 2025-08-14T21:53:55.8492417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8492503Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8492774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8492842Z layer_outputs = layer_module( 2025-08-14T21:53:55.8493103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8493220Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8493485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8493616Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8493881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8493989Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8493993Z 2025-08-14T21:53:55.8494101Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8494295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8494369Z return mod(**inputs) 2025-08-14T21:53:55.8494640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8494711Z outputs = self.mobilebert( 2025-08-14T21:53:55.8494989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8495063Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8495333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8495410Z layer_outputs = layer_module( 2025-08-14T21:53:55.8495690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8495789Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8496066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8496186Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8496471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8496557Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8496560Z 2025-08-14T21:53:55.8496670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8496863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8496929Z return mod(**inputs) 2025-08-14T21:53:55.8497210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8497279Z outputs = self.mobilebert( 2025-08-14T21:53:55.8497562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8497631Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8497903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8498040Z layer_outputs = layer_module( 2025-08-14T21:53:55.8498313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8498403Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8498686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8498824Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8499112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8499233Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8499528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8499627Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8499632Z 2025-08-14T21:53:55.8499747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8499949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8500016Z return mod(**inputs) 2025-08-14T21:53:55.8500292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8500370Z outputs = self.mobilebert( 2025-08-14T21:53:55.8500646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8500718Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8501003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8501073Z layer_outputs = layer_module( 2025-08-14T21:53:55.8501353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8501443Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8501714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8501833Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8502106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8502195Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8502198Z 2025-08-14T21:53:55.8502298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8502495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8502568Z return mod(**inputs) 2025-08-14T21:53:55.8502846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8502917Z outputs = self.mobilebert( 2025-08-14T21:53:55.8503197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8503271Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8503552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8503623Z layer_outputs = layer_module( 2025-08-14T21:53:55.8503894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8503995Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8504268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8504418Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8504712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8504842Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8504846Z 2025-08-14T21:53:55.8504958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8505171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8505235Z return mod(**inputs) 2025-08-14T21:53:55.8505516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8505604Z outputs = self.mobilebert( 2025-08-14T21:53:55.8505889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8505979Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8506257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8506332Z layer_outputs = layer_module( 2025-08-14T21:53:55.8506609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8506706Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8506984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8507105Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8507391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8507476Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8507479Z 2025-08-14T21:53:55.8507588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8507783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8507849Z return mod(**inputs) 2025-08-14T21:53:55.8508135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8508205Z outputs = self.mobilebert( 2025-08-14T21:53:55.8508481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8508563Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8509018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8509103Z layer_outputs = layer_module( 2025-08-14T21:53:55.8509379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8509472Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8509752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8509875Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8510156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8510279Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8510555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8510655Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8510709Z 2025-08-14T21:53:55.8510813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8511005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8511076Z return mod(**inputs) 2025-08-14T21:53:55.8511381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8511460Z outputs = self.mobilebert( 2025-08-14T21:53:55.8511739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8511812Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8512117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8512192Z layer_outputs = layer_module( 2025-08-14T21:53:55.8512508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8512627Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8512896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8512984Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8512987Z 2025-08-14T21:53:55.8513086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8513290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8513365Z return mod(**inputs) 2025-08-14T21:53:55.8513665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8513746Z outputs = self.mobilebert( 2025-08-14T21:53:55.8514044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8514119Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8514420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8514495Z layer_outputs = layer_module( 2025-08-14T21:53:55.8514812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8514938Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8515248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8515376Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8515380Z 2025-08-14T21:53:55.8515487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8515751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8515837Z return mod(**inputs) 2025-08-14T21:53:55.8516138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8516225Z outputs = self.mobilebert( 2025-08-14T21:53:55.8516533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8516608Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8516916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8516991Z layer_outputs = layer_module( 2025-08-14T21:53:55.8517308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8517509Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8517815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8517925Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8517947Z 2025-08-14T21:53:55.8518054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8518258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8518332Z return mod(**inputs) 2025-08-14T21:53:55.8518625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8518726Z outputs = self.mobilebert( 2025-08-14T21:53:55.8519017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8519093Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8519416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8519492Z layer_outputs = layer_module( 2025-08-14T21:53:55.8519791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8519957Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8520246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8520383Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8520686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8520784Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8520797Z 2025-08-14T21:53:55.8520903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8521107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8521184Z return mod(**inputs) 2025-08-14T21:53:55.8521478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8521550Z outputs = self.mobilebert( 2025-08-14T21:53:55.8521860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8521934Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8522241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8522316Z layer_outputs = layer_module( 2025-08-14T21:53:55.8522605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8522776Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8523069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8523207Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8523496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8523583Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8523589Z 2025-08-14T21:53:55.8523703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8523907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8523999Z return mod(**inputs) 2025-08-14T21:53:55.8524300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8524374Z outputs = self.mobilebert( 2025-08-14T21:53:55.8524691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8524767Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8525057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8525136Z layer_outputs = layer_module( 2025-08-14T21:53:55.8525449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8525620Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8525925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8526055Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8526347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8526476Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8526762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8526868Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8526872Z 2025-08-14T21:53:55.8526980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8527193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8527265Z return mod(**inputs) 2025-08-14T21:53:55.8527558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8527641Z outputs = self.mobilebert( 2025-08-14T21:53:55.8527930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8528013Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8528299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8528374Z layer_outputs = layer_module( 2025-08-14T21:53:55.8528673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8528841Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8529139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8529256Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8529542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8529637Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8529641Z 2025-08-14T21:53:55.8529748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8529952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8530027Z return mod(**inputs) 2025-08-14T21:53:55.8530322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8530404Z outputs = self.mobilebert( 2025-08-14T21:53:55.8530726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8530802Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8531116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8531207Z layer_outputs = layer_module( 2025-08-14T21:53:55.8531508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8531675Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8531996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8532122Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8532439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8532530Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8532828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8532924Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8532927Z 2025-08-14T21:53:55.8533038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8533248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8533318Z return mod(**inputs) 2025-08-14T21:53:55.8533630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8533705Z outputs = self.mobilebert( 2025-08-14T21:53:55.8534002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8534071Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8534344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8534422Z layer_outputs = layer_module( 2025-08-14T21:53:55.8534693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8534779Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8535058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8535127Z self_outputs = self.self( 2025-08-14T21:53:55.8535404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8535477Z self.query(query_tensor) 2025-08-14T21:53:55.8535481Z 2025-08-14T21:53:55.8535581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8535785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8535856Z return mod(**inputs) 2025-08-14T21:53:55.8536152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8536225Z outputs = self.mobilebert( 2025-08-14T21:53:55.8536524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8536605Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8536904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8536996Z layer_outputs = layer_module( 2025-08-14T21:53:55.8537295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8537381Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8537665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8537752Z self_outputs = self.self( 2025-08-14T21:53:55.8538023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8538096Z self.key(key_tensor) 2025-08-14T21:53:55.8538099Z 2025-08-14T21:53:55.8538200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8538413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8538480Z return mod(**inputs) 2025-08-14T21:53:55.8538772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8538848Z outputs = self.mobilebert( 2025-08-14T21:53:55.8539120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8539192Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8539474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8539543Z layer_outputs = layer_module( 2025-08-14T21:53:55.8539826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8539914Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8540187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8540267Z self_outputs = self.self( 2025-08-14T21:53:55.8540545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8540616Z self.value(value_tensor) 2025-08-14T21:53:55.8540628Z 2025-08-14T21:53:55.8540710Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8540785Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8540894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8541084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8541149Z return mod(**inputs) 2025-08-14T21:53:55.8541434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8541504Z outputs = self.mobilebert( 2025-08-14T21:53:55.8541787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8541860Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8542131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8542209Z layer_outputs = layer_module( 2025-08-14T21:53:55.8542486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8542570Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8542853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8542976Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8543260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8543362Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8543366Z 2025-08-14T21:53:55.8543468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8543671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8543753Z return mod(**inputs) 2025-08-14T21:53:55.8544035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8544104Z outputs = self.mobilebert( 2025-08-14T21:53:55.8544379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8544457Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8544752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8544825Z layer_outputs = layer_module( 2025-08-14T21:53:55.8545142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8545309Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8545622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8545732Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8546004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8546094Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8546097Z 2025-08-14T21:53:55.8546198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8546396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8546463Z return mod(**inputs) 2025-08-14T21:53:55.8546737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8546814Z outputs = self.mobilebert( 2025-08-14T21:53:55.8547090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8547161Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8547444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8547512Z layer_outputs = layer_module( 2025-08-14T21:53:55.8547793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8547876Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8548153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8548284Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8548558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8548690Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8548986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8549082Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8549085Z 2025-08-14T21:53:55.8549201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8549416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8549504Z return mod(**inputs) 2025-08-14T21:53:55.8549803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8549877Z outputs = self.mobilebert( 2025-08-14T21:53:55.8550176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8550270Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8550577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8550658Z layer_outputs = layer_module( 2025-08-14T21:53:55.8550978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8551087Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8551407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8551524Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8551829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8551919Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8551922Z 2025-08-14T21:53:55.8552034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8552250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8552318Z return mod(**inputs) 2025-08-14T21:53:55.8552614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8552687Z outputs = self.mobilebert( 2025-08-14T21:53:55.8552983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8553070Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8553369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8553450Z layer_outputs = layer_module( 2025-08-14T21:53:55.8553736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8553832Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8554124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8554242Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8554529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8554659Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8554662Z 2025-08-14T21:53:55.8554767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8554977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8555046Z return mod(**inputs) 2025-08-14T21:53:55.8555337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8555418Z outputs = self.mobilebert( 2025-08-14T21:53:55.8555784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8555879Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8556180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8556281Z layer_outputs = layer_module( 2025-08-14T21:53:55.8556581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8556678Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8556996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8557128Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8557431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8557529Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8557532Z 2025-08-14T21:53:55.8557665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8557871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8557948Z return mod(**inputs) 2025-08-14T21:53:55.8558258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8558340Z outputs = self.mobilebert( 2025-08-14T21:53:55.8558634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8558709Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8559005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8559079Z layer_outputs = layer_module( 2025-08-14T21:53:55.8559377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8559474Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8559762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8559899Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8560185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8560314Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8560623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8560720Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8560724Z 2025-08-14T21:53:55.8560839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8561046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8561116Z return mod(**inputs) 2025-08-14T21:53:55.8561416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8561489Z outputs = self.mobilebert( 2025-08-14T21:53:55.8561782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8561858Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8562161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8562244Z layer_outputs = layer_module( 2025-08-14T21:53:55.8562537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8562628Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8562925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8563036Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8563317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8563423Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8563426Z 2025-08-14T21:53:55.8563525Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8563734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8563797Z return mod(**inputs) 2025-08-14T21:53:55.8564087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8564156Z outputs = self.mobilebert( 2025-08-14T21:53:55.8564420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8564515Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8564783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8564853Z layer_outputs = layer_module( 2025-08-14T21:53:55.8565126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8565215Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8565488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8565599Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8565871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8565992Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8565995Z 2025-08-14T21:53:55.8566097Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8566300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8566368Z return mod(**inputs) 2025-08-14T21:53:55.8566645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8566733Z outputs = self.mobilebert( 2025-08-14T21:53:55.8566996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8567064Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8567341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8567408Z layer_outputs = layer_module( 2025-08-14T21:53:55.8567679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8567768Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8568036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8568164Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8568438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8568528Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8568531Z 2025-08-14T21:53:55.8568644Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8568835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8568921Z return mod(**inputs) 2025-08-14T21:53:55.8569192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8569261Z outputs = self.mobilebert( 2025-08-14T21:53:55.8569550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8569618Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8569892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8569959Z layer_outputs = layer_module( 2025-08-14T21:53:55.8571042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8571156Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8571440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8571565Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8571828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8571945Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8572221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8572311Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8572315Z 2025-08-14T21:53:55.8572420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8572614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8572680Z return mod(**inputs) 2025-08-14T21:53:55.8572963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8573035Z outputs = self.mobilebert( 2025-08-14T21:53:55.8573305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8573385Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8573657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8573734Z layer_outputs = layer_module( 2025-08-14T21:53:55.8574016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8574105Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8574378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8574489Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8574760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8574844Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8574847Z 2025-08-14T21:53:55.8574947Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8575147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8575212Z return mod(**inputs) 2025-08-14T21:53:55.8575488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8575566Z outputs = self.mobilebert( 2025-08-14T21:53:55.8575837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8575940Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8576212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8576296Z layer_outputs = layer_module( 2025-08-14T21:53:55.8576576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8576677Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8576950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8577057Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8577336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8577469Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8577472Z 2025-08-14T21:53:55.8577574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8577765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8577840Z return mod(**inputs) 2025-08-14T21:53:55.8578116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8578193Z outputs = self.mobilebert( 2025-08-14T21:53:55.8578478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8578548Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8578822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8578891Z layer_outputs = layer_module( 2025-08-14T21:53:55.8579162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8579254Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8579520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8579649Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8579927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8580011Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8580022Z 2025-08-14T21:53:55.8580124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8580317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8580391Z return mod(**inputs) 2025-08-14T21:53:55.8580668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8580739Z outputs = self.mobilebert( 2025-08-14T21:53:55.8581023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8581095Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8581380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8581453Z layer_outputs = layer_module( 2025-08-14T21:53:55.8581725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8581825Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8582117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8582237Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8582516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8582658Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8582942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8583033Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8583039Z 2025-08-14T21:53:55.8583140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8583356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8583424Z return mod(**inputs) 2025-08-14T21:53:55.8583724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8583796Z outputs = self.mobilebert( 2025-08-14T21:53:55.8584070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8584153Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8584431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8584507Z layer_outputs = layer_module( 2025-08-14T21:53:55.8584805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8584933Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8585241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8585328Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8585332Z 2025-08-14T21:53:55.8585436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8585681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8585762Z return mod(**inputs) 2025-08-14T21:53:55.8586045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8586114Z outputs = self.mobilebert( 2025-08-14T21:53:55.8586388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8586470Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8586747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8586825Z layer_outputs = layer_module( 2025-08-14T21:53:55.8587101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8587220Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8587502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8587612Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8587616Z 2025-08-14T21:53:55.8587722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8587939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8588009Z return mod(**inputs) 2025-08-14T21:53:55.8588316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8588403Z outputs = self.mobilebert( 2025-08-14T21:53:55.8588683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8588777Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8589073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8589154Z layer_outputs = layer_module( 2025-08-14T21:53:55.8589453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8589620Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8589940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8590043Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8590063Z 2025-08-14T21:53:55.8590171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8590393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8590464Z return mod(**inputs) 2025-08-14T21:53:55.8590763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8590836Z outputs = self.mobilebert( 2025-08-14T21:53:55.8591136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8591218Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8591521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8591603Z layer_outputs = layer_module( 2025-08-14T21:53:55.8591903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8592068Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8592375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8592503Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8592804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8592908Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8592912Z 2025-08-14T21:53:55.8593018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8593240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8593313Z return mod(**inputs) 2025-08-14T21:53:55.8593616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8593699Z outputs = self.mobilebert( 2025-08-14T21:53:55.8594008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8594092Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8594401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8594476Z layer_outputs = layer_module( 2025-08-14T21:53:55.8594783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8594950Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8595282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8595414Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8595813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8595919Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8595923Z 2025-08-14T21:53:55.8596033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8596244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8596329Z return mod(**inputs) 2025-08-14T21:53:55.8596659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8596748Z outputs = self.mobilebert( 2025-08-14T21:53:55.8597116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8597197Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8597494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8597569Z layer_outputs = layer_module( 2025-08-14T21:53:55.8597874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8598033Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8598338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8598476Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8598777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8598907Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8599208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8599309Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8599313Z 2025-08-14T21:53:55.8599426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8599635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8599707Z return mod(**inputs) 2025-08-14T21:53:55.8600016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8600095Z outputs = self.mobilebert( 2025-08-14T21:53:55.8600398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8600475Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8600771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8600856Z layer_outputs = layer_module( 2025-08-14T21:53:55.8601163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8601333Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8601640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8601758Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8602089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8602180Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8602184Z 2025-08-14T21:53:55.8602293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8602530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8602599Z return mod(**inputs) 2025-08-14T21:53:55.8602906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8602982Z outputs = self.mobilebert( 2025-08-14T21:53:55.8603304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8603391Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8603703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8603782Z layer_outputs = layer_module( 2025-08-14T21:53:55.8604091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8604264Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8604574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8604690Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8604989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8605090Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8605390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8605498Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8605501Z 2025-08-14T21:53:55.8605610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8605822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8605902Z return mod(**inputs) 2025-08-14T21:53:55.8606199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8606281Z outputs = self.mobilebert( 2025-08-14T21:53:55.8606579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8606657Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8606961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8607039Z layer_outputs = layer_module( 2025-08-14T21:53:55.8607335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8607432Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8607730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8607812Z self_outputs = self.self( 2025-08-14T21:53:55.8608108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8608184Z self.query(query_tensor) 2025-08-14T21:53:55.8608188Z 2025-08-14T21:53:55.8608306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8608517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8608625Z return mod(**inputs) 2025-08-14T21:53:55.8609085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8609164Z outputs = self.mobilebert( 2025-08-14T21:53:55.8609518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8609596Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8609896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8609979Z layer_outputs = layer_module( 2025-08-14T21:53:55.8610303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8610405Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8610727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8610805Z self_outputs = self.self( 2025-08-14T21:53:55.8611117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8611194Z self.key(key_tensor) 2025-08-14T21:53:55.8611198Z 2025-08-14T21:53:55.8611318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8611531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8611601Z return mod(**inputs) 2025-08-14T21:53:55.8611911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8611989Z outputs = self.mobilebert( 2025-08-14T21:53:55.8612293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8612381Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8612686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8612770Z layer_outputs = layer_module( 2025-08-14T21:53:55.8613068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8613160Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8613463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8613533Z self_outputs = self.self( 2025-08-14T21:53:55.8613807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8613888Z self.value(value_tensor) 2025-08-14T21:53:55.8613891Z 2025-08-14T21:53:55.8613972Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8614058Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8614159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8614352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8614427Z return mod(**inputs) 2025-08-14T21:53:55.8614704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8614779Z outputs = self.mobilebert( 2025-08-14T21:53:55.8615050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8615122Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8615400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8615500Z layer_outputs = layer_module( 2025-08-14T21:53:55.8615771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8615877Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8616149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8616276Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8616549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8616635Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8616656Z 2025-08-14T21:53:55.8616764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8616960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8617048Z return mod(**inputs) 2025-08-14T21:53:55.8617326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8617397Z outputs = self.mobilebert( 2025-08-14T21:53:55.8617674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8617745Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8618014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8618090Z layer_outputs = layer_module( 2025-08-14T21:53:55.8618362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8618528Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8618801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8618909Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8619190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8619271Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8619274Z 2025-08-14T21:53:55.8619381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8619571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8619639Z return mod(**inputs) 2025-08-14T21:53:55.8619937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8620012Z outputs = self.mobilebert( 2025-08-14T21:53:55.8620308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8620386Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8620664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8620739Z layer_outputs = layer_module( 2025-08-14T21:53:55.8621015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8621098Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8621377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8621497Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8621798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8621922Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8622196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8622310Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8622313Z 2025-08-14T21:53:55.8622411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8622601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8622670Z return mod(**inputs) 2025-08-14T21:53:55.8622959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8623036Z outputs = self.mobilebert( 2025-08-14T21:53:55.8623325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8623398Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8623679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8623751Z layer_outputs = layer_module( 2025-08-14T21:53:55.8624030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8624122Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8624398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8624519Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8624794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8624880Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8624891Z 2025-08-14T21:53:55.8624992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8625186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8625260Z return mod(**inputs) 2025-08-14T21:53:55.8625535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8625604Z outputs = self.mobilebert( 2025-08-14T21:53:55.8625885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8625958Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8626242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8626315Z layer_outputs = layer_module( 2025-08-14T21:53:55.8626588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8626690Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8626970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8627083Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8627368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8627485Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8627490Z 2025-08-14T21:53:55.8627605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8627828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8627898Z return mod(**inputs) 2025-08-14T21:53:55.8628205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8628305Z outputs = self.mobilebert( 2025-08-14T21:53:55.8628587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8628657Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8628932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8629009Z layer_outputs = layer_module( 2025-08-14T21:53:55.8629297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8629394Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8629693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8629820Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8630101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8630186Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8630189Z 2025-08-14T21:53:55.8630290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8630498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8630562Z return mod(**inputs) 2025-08-14T21:53:55.8630847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8630919Z outputs = self.mobilebert( 2025-08-14T21:53:55.8631194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8631275Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8631549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8631626Z layer_outputs = layer_module( 2025-08-14T21:53:55.8631898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8631989Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8632271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8632393Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8632671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8632798Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8633082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8633186Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8633190Z 2025-08-14T21:53:55.8633297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8633503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8633580Z return mod(**inputs) 2025-08-14T21:53:55.8633876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8633957Z outputs = self.mobilebert( 2025-08-14T21:53:55.8634275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8634352Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8634652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8634744Z layer_outputs = layer_module( 2025-08-14T21:53:55.8635031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8635137Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8635424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8635563Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8635912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8636029Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8636033Z 2025-08-14T21:53:55.8636150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8636352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8636433Z return mod(**inputs) 2025-08-14T21:53:55.8636724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8636799Z outputs = self.mobilebert( 2025-08-14T21:53:55.8637092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8637168Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8637456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8637540Z layer_outputs = layer_module( 2025-08-14T21:53:55.8637828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8637935Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8638257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8638372Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8638669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8638786Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8638791Z 2025-08-14T21:53:55.8638905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8639113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8639182Z return mod(**inputs) 2025-08-14T21:53:55.8639480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8639555Z outputs = self.mobilebert( 2025-08-14T21:53:55.8639868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8639949Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8640235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8640313Z layer_outputs = layer_module( 2025-08-14T21:53:55.8640635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8640750Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8641050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8641177Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8641495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8641582Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8641585Z 2025-08-14T21:53:55.8641688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8641902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8641970Z return mod(**inputs) 2025-08-14T21:53:55.8642280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8642364Z outputs = self.mobilebert( 2025-08-14T21:53:55.8642668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8642752Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8643045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8643120Z layer_outputs = layer_module( 2025-08-14T21:53:55.8643424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8643519Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8643846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8643973Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8644268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8644402Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8644705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8644804Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8644808Z 2025-08-14T21:53:55.8644908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8645102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8645174Z return mod(**inputs) 2025-08-14T21:53:55.8645464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8645532Z outputs = self.mobilebert( 2025-08-14T21:53:55.8645812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8645883Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8646160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8646229Z layer_outputs = layer_module( 2025-08-14T21:53:55.8646496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8646591Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8646860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8646972Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8647239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8647337Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8647340Z 2025-08-14T21:53:55.8647443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8647631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8647709Z return mod(**inputs) 2025-08-14T21:53:55.8647992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8648063Z outputs = self.mobilebert( 2025-08-14T21:53:55.8648342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8648428Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8648708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8648804Z layer_outputs = layer_module( 2025-08-14T21:53:55.8649084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8649180Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8649446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8649552Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8649832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8649939Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8649944Z 2025-08-14T21:53:55.8650045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8650245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8650312Z return mod(**inputs) 2025-08-14T21:53:55.8650593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8650664Z outputs = self.mobilebert( 2025-08-14T21:53:55.8650940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8651025Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8651313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8651393Z layer_outputs = layer_module( 2025-08-14T21:53:55.8651680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8651774Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8652055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8652174Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8652449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8652539Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8652543Z 2025-08-14T21:53:55.8652641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8652841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8652905Z return mod(**inputs) 2025-08-14T21:53:55.8653180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8653276Z outputs = self.mobilebert( 2025-08-14T21:53:55.8653554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8653630Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8653903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8653990Z layer_outputs = layer_module( 2025-08-14T21:53:55.8654266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8654356Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8654655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8654785Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8655074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8655201Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8655475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8655567Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8655570Z 2025-08-14T21:53:55.8655676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8655874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8655946Z return mod(**inputs) 2025-08-14T21:53:55.8656240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8656314Z outputs = self.mobilebert( 2025-08-14T21:53:55.8656612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8656683Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8656954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8657031Z layer_outputs = layer_module( 2025-08-14T21:53:55.8657304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8657430Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8657714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8657795Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8657798Z 2025-08-14T21:53:55.8657903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8658093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8658162Z return mod(**inputs) 2025-08-14T21:53:55.8658431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8658501Z outputs = self.mobilebert( 2025-08-14T21:53:55.8658773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8658842Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8659108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8659186Z layer_outputs = layer_module( 2025-08-14T21:53:55.8659450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8659594Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8659857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8659962Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8659983Z 2025-08-14T21:53:55.8660089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8660277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8660347Z return mod(**inputs) 2025-08-14T21:53:55.8660613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8660699Z outputs = self.mobilebert( 2025-08-14T21:53:55.8660976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8661049Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8661346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8661420Z layer_outputs = layer_module( 2025-08-14T21:53:55.8661694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8661860Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8662138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8662233Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8662246Z 2025-08-14T21:53:55.8662348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8662541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8662617Z return mod(**inputs) 2025-08-14T21:53:55.8662892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8662964Z outputs = self.mobilebert( 2025-08-14T21:53:55.8663245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8663315Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8663597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8663666Z layer_outputs = layer_module( 2025-08-14T21:53:55.8663942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8664107Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8664379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8664504Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8664800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8664896Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8664900Z 2025-08-14T21:53:55.8665014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8665219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8665288Z return mod(**inputs) 2025-08-14T21:53:55.8665589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8665682Z outputs = self.mobilebert( 2025-08-14T21:53:55.8665979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8666053Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8666344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8666456Z layer_outputs = layer_module( 2025-08-14T21:53:55.8666732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8666886Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8667187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8667308Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8667607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8667690Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8667694Z 2025-08-14T21:53:55.8667798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8668000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8668065Z return mod(**inputs) 2025-08-14T21:53:55.8668350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8668421Z outputs = self.mobilebert( 2025-08-14T21:53:55.8668699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8668781Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8669085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8669164Z layer_outputs = layer_module( 2025-08-14T21:53:55.8669459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8669621Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8669920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8670050Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8670350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8670483Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8670783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8670887Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8670891Z 2025-08-14T21:53:55.8670997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8671216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8671292Z return mod(**inputs) 2025-08-14T21:53:55.8671585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8671665Z outputs = self.mobilebert( 2025-08-14T21:53:55.8671961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8672037Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8672357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8672430Z layer_outputs = layer_module( 2025-08-14T21:53:55.8672728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8672919Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8673221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8673342Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8673671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8673759Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8673765Z 2025-08-14T21:53:55.8673881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8674121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8674200Z return mod(**inputs) 2025-08-14T21:53:55.8674506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8674585Z outputs = self.mobilebert( 2025-08-14T21:53:55.8674899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8674979Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8675289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8675373Z layer_outputs = layer_module( 2025-08-14T21:53:55.8675763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8675956Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8676269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8676390Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8676711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8676804Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8677126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8677218Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8677222Z 2025-08-14T21:53:55.8677324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8677528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8677595Z return mod(**inputs) 2025-08-14T21:53:55.8677880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8677954Z outputs = self.mobilebert( 2025-08-14T21:53:55.8678230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8678314Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8678590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8678661Z layer_outputs = layer_module( 2025-08-14T21:53:55.8678944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8679050Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8679331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8679402Z self_outputs = self.self( 2025-08-14T21:53:55.8679701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8679783Z self.query(query_tensor) 2025-08-14T21:53:55.8679786Z 2025-08-14T21:53:55.8679887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8680087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8680153Z return mod(**inputs) 2025-08-14T21:53:55.8680445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8680525Z outputs = self.mobilebert( 2025-08-14T21:53:55.8680815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8680888Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8681170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8681240Z layer_outputs = layer_module( 2025-08-14T21:53:55.8681526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8681610Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8681890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8681975Z self_outputs = self.self( 2025-08-14T21:53:55.8682265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8682338Z self.key(key_tensor) 2025-08-14T21:53:55.8682349Z 2025-08-14T21:53:55.8682455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8682659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8682735Z return mod(**inputs) 2025-08-14T21:53:55.8683028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8683102Z outputs = self.mobilebert( 2025-08-14T21:53:55.8683410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8683487Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8683783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8683858Z layer_outputs = layer_module( 2025-08-14T21:53:55.8684148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8684244Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8684537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8684609Z self_outputs = self.self( 2025-08-14T21:53:55.8684918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8684991Z self.value(value_tensor) 2025-08-14T21:53:55.8684995Z 2025-08-14T21:53:55.8685089Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8685172Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8685279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8685511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8685579Z return mod(**inputs) 2025-08-14T21:53:55.8685870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8685968Z outputs = self.mobilebert( 2025-08-14T21:53:55.8686260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8686342Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8686638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8686711Z layer_outputs = layer_module( 2025-08-14T21:53:55.8687060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8687153Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8687474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8687606Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8687912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8688010Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8688013Z 2025-08-14T21:53:55.8688119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8688328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8688396Z return mod(**inputs) 2025-08-14T21:53:55.8688688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8688772Z outputs = self.mobilebert( 2025-08-14T21:53:55.8689060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8689134Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8689430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8689504Z layer_outputs = layer_module( 2025-08-14T21:53:55.8689811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8689978Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8690281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8690409Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8690698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8690791Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8690796Z 2025-08-14T21:53:55.8690902Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8691107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8691183Z return mod(**inputs) 2025-08-14T21:53:55.8691476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8691552Z outputs = self.mobilebert( 2025-08-14T21:53:55.8691852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8691943Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8692227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8692297Z layer_outputs = layer_module( 2025-08-14T21:53:55.8692570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8692676Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8692947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8693073Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8693363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8693498Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8693811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8693909Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8693912Z 2025-08-14T21:53:55.8694028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8694231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8694294Z return mod(**inputs) 2025-08-14T21:53:55.8694577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8694648Z outputs = self.mobilebert( 2025-08-14T21:53:55.8694922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8695000Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8695278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8695359Z layer_outputs = layer_module( 2025-08-14T21:53:55.8695647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8695745Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8696039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8696156Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8696445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8696540Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8696544Z 2025-08-14T21:53:55.8696649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8696868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8696937Z return mod(**inputs) 2025-08-14T21:53:55.8697228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8697312Z outputs = self.mobilebert( 2025-08-14T21:53:55.8697598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8697678Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8697967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8698041Z layer_outputs = layer_module( 2025-08-14T21:53:55.8698335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8698452Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8698739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8698861Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8699169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8699293Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8699296Z 2025-08-14T21:53:55.8699402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8699607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8699728Z return mod(**inputs) 2025-08-14T21:53:55.8700025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8700107Z outputs = self.mobilebert( 2025-08-14T21:53:55.8700411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8700488Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8700792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8700866Z layer_outputs = layer_module( 2025-08-14T21:53:55.8701159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8701265Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8701571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8701710Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8702004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8702094Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8702097Z 2025-08-14T21:53:55.8702211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8702417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8702493Z return mod(**inputs) 2025-08-14T21:53:55.8702790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8702866Z outputs = self.mobilebert( 2025-08-14T21:53:55.8703166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8703242Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8703539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8703624Z layer_outputs = layer_module( 2025-08-14T21:53:55.8703920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8704028Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8704323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8704450Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8704763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8704891Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8705211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8705306Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8705309Z 2025-08-14T21:53:55.8705415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8705844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8705919Z return mod(**inputs) 2025-08-14T21:53:55.8706221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8706296Z outputs = self.mobilebert( 2025-08-14T21:53:55.8706620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8706706Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8707012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8707087Z layer_outputs = layer_module( 2025-08-14T21:53:55.8707387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8707486Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8707789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8707905Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8708211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8708308Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8708311Z 2025-08-14T21:53:55.8708419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8708769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8708853Z return mod(**inputs) 2025-08-14T21:53:55.8709148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8709232Z outputs = self.mobilebert( 2025-08-14T21:53:55.8709523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8709602Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8709922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8710003Z layer_outputs = layer_module( 2025-08-14T21:53:55.8710325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8710432Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8710733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8710865Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8711169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8711302Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8711306Z 2025-08-14T21:53:55.8711421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8711636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8711724Z return mod(**inputs) 2025-08-14T21:53:55.8712031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8712158Z outputs = self.mobilebert( 2025-08-14T21:53:55.8712463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8712540Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8712865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8712943Z layer_outputs = layer_module( 2025-08-14T21:53:55.8713237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8713346Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8713666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8713811Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8714132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8714225Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8714229Z 2025-08-14T21:53:55.8714348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8714558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8714628Z return mod(**inputs) 2025-08-14T21:53:55.8714937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8715015Z outputs = self.mobilebert( 2025-08-14T21:53:55.8715322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8715398Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8715746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8715842Z layer_outputs = layer_module( 2025-08-14T21:53:55.8716137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8716244Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8716538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8716670Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8716976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8717105Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8717412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8717511Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8717515Z 2025-08-14T21:53:55.8717624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8717842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8717913Z return mod(**inputs) 2025-08-14T21:53:55.8718211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8718295Z outputs = self.mobilebert( 2025-08-14T21:53:55.8718592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8718678Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8719003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8719080Z layer_outputs = layer_module( 2025-08-14T21:53:55.8719385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8719504Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8719808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8719925Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8720224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8720337Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8720341Z 2025-08-14T21:53:55.8720450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8720676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8720753Z return mod(**inputs) 2025-08-14T21:53:55.8721052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8721136Z outputs = self.mobilebert( 2025-08-14T21:53:55.8721430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8721507Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8721812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8721886Z layer_outputs = layer_module( 2025-08-14T21:53:55.8722193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8722279Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8722538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8722646Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8722907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8723014Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8723024Z 2025-08-14T21:53:55.8723120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8723308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8723377Z return mod(**inputs) 2025-08-14T21:53:55.8723646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8723714Z outputs = self.mobilebert( 2025-08-14T21:53:55.8723985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8724056Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8724327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8724395Z layer_outputs = layer_module( 2025-08-14T21:53:55.8724656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8724751Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8725016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8725134Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8725434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8725514Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8725517Z 2025-08-14T21:53:55.8725634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8725815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8725877Z return mod(**inputs) 2025-08-14T21:53:55.8726144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8726210Z outputs = self.mobilebert( 2025-08-14T21:53:55.8726488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8726557Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8726837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8726913Z layer_outputs = layer_module( 2025-08-14T21:53:55.8727177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8727266Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8727535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8727651Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8727921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8728037Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8728309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8728407Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8728410Z 2025-08-14T21:53:55.8728510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8728714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8728779Z return mod(**inputs) 2025-08-14T21:53:55.8729051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8729130Z outputs = self.mobilebert( 2025-08-14T21:53:55.8729401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8729471Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8729749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8729822Z layer_outputs = layer_module( 2025-08-14T21:53:55.8730100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8730218Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8730494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8730583Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8730586Z 2025-08-14T21:53:55.8730685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8730889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8730952Z return mod(**inputs) 2025-08-14T21:53:55.8731213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8731308Z outputs = self.mobilebert( 2025-08-14T21:53:55.8731569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8731653Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8731917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8731982Z layer_outputs = layer_module( 2025-08-14T21:53:55.8732248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8732359Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8732634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8732749Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8732765Z 2025-08-14T21:53:55.8732862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8733057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8733120Z return mod(**inputs) 2025-08-14T21:53:55.8733387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8733462Z outputs = self.mobilebert( 2025-08-14T21:53:55.8733733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8733812Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8734087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8734160Z layer_outputs = layer_module( 2025-08-14T21:53:55.8734444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8734603Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8734888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8734988Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8734991Z 2025-08-14T21:53:55.8735089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8735285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8735348Z return mod(**inputs) 2025-08-14T21:53:55.8735628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8735702Z outputs = self.mobilebert( 2025-08-14T21:53:55.8735966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8736041Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8736310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8736378Z layer_outputs = layer_module( 2025-08-14T21:53:55.8736665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8736812Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8737073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8737212Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8737471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8737568Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8737571Z 2025-08-14T21:53:55.8737685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8737868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8737941Z return mod(**inputs) 2025-08-14T21:53:55.8738214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8738289Z outputs = self.mobilebert( 2025-08-14T21:53:55.8738582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8738655Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8738951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8739022Z layer_outputs = layer_module( 2025-08-14T21:53:55.8739295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8739458Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8739794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8739918Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8740186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8740267Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8740271Z 2025-08-14T21:53:55.8740377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8740565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8740635Z return mod(**inputs) 2025-08-14T21:53:55.8740904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8740971Z outputs = self.mobilebert( 2025-08-14T21:53:55.8741241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8741311Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8741577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8741654Z layer_outputs = layer_module( 2025-08-14T21:53:55.8741925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8742082Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8742348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8742466Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8742738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8742853Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8743128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8743217Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8743238Z 2025-08-14T21:53:55.8743338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8743536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8743601Z return mod(**inputs) 2025-08-14T21:53:55.8743875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8743962Z outputs = self.mobilebert( 2025-08-14T21:53:55.8744228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8744305Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8744584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8744655Z layer_outputs = layer_module( 2025-08-14T21:53:55.8744926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8745096Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8745370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8745477Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8745744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8745831Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8745835Z 2025-08-14T21:53:55.8745932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8746129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8746192Z return mod(**inputs) 2025-08-14T21:53:55.8746464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8746546Z outputs = self.mobilebert( 2025-08-14T21:53:55.8746834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8746912Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8747211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8747281Z layer_outputs = layer_module( 2025-08-14T21:53:55.8747563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8747720Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8747993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8748111Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8748382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8748476Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8748747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8748839Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8748842Z 2025-08-14T21:53:55.8748952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8749146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8749218Z return mod(**inputs) 2025-08-14T21:53:55.8749493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8749580Z outputs = self.mobilebert( 2025-08-14T21:53:55.8749864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8749952Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8750229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8750307Z layer_outputs = layer_module( 2025-08-14T21:53:55.8750585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8750677Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8750978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8751052Z self_outputs = self.self( 2025-08-14T21:53:55.8751345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8751417Z self.query(query_tensor) 2025-08-14T21:53:55.8751420Z 2025-08-14T21:53:55.8751531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8751724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8751788Z return mod(**inputs) 2025-08-14T21:53:55.8752074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8752142Z outputs = self.mobilebert( 2025-08-14T21:53:55.8752421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8752499Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8752777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8752853Z layer_outputs = layer_module( 2025-08-14T21:53:55.8753130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8753216Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8753524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8753596Z self_outputs = self.self( 2025-08-14T21:53:55.8753895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8753974Z self.key(key_tensor) 2025-08-14T21:53:55.8753978Z 2025-08-14T21:53:55.8754083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8754295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8754364Z return mod(**inputs) 2025-08-14T21:53:55.8754660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8754743Z outputs = self.mobilebert( 2025-08-14T21:53:55.8755044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8755125Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8755426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8755500Z layer_outputs = layer_module( 2025-08-14T21:53:55.8755888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8756004Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8756305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8756388Z self_outputs = self.self( 2025-08-14T21:53:55.8756704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8756788Z self.value(value_tensor) 2025-08-14T21:53:55.8756792Z 2025-08-14T21:53:55.8756877Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8756962Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8757079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8757302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8757368Z return mod(**inputs) 2025-08-14T21:53:55.8757647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8757735Z outputs = self.mobilebert( 2025-08-14T21:53:55.8758013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8758083Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8758351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8758426Z layer_outputs = layer_module( 2025-08-14T21:53:55.8758694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8758783Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8759055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8759174Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8759452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8759533Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8759538Z 2025-08-14T21:53:55.8759646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8759836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8759900Z return mod(**inputs) 2025-08-14T21:53:55.8760177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8760246Z outputs = self.mobilebert( 2025-08-14T21:53:55.8760518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8760596Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8760867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8760943Z layer_outputs = layer_module( 2025-08-14T21:53:55.8761222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8761373Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8761641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8761745Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8762014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8762111Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8762115Z 2025-08-14T21:53:55.8762211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8762401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8762462Z return mod(**inputs) 2025-08-14T21:53:55.8762746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8762825Z outputs = self.mobilebert( 2025-08-14T21:53:55.8763129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8763204Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8763488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8763558Z layer_outputs = layer_module( 2025-08-14T21:53:55.8763844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8763927Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8764198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8764317Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8764580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8764706Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8764972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8765060Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8765073Z 2025-08-14T21:53:55.8765172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8765361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8765432Z return mod(**inputs) 2025-08-14T21:53:55.8765705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8765776Z outputs = self.mobilebert( 2025-08-14T21:53:55.8766055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8766125Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8766407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8766480Z layer_outputs = layer_module( 2025-08-14T21:53:55.8766752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8766863Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8767129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8767238Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8767510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8767589Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8767593Z 2025-08-14T21:53:55.8767698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8767889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8767953Z return mod(**inputs) 2025-08-14T21:53:55.8768226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8768314Z outputs = self.mobilebert( 2025-08-14T21:53:55.8768585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8768672Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8768936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8769011Z layer_outputs = layer_module( 2025-08-14T21:53:55.8769275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8769368Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8769661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8769769Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8770059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8770167Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8770172Z 2025-08-14T21:53:55.8770270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8770466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8770528Z return mod(**inputs) 2025-08-14T21:53:55.8770803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8770870Z outputs = self.mobilebert( 2025-08-14T21:53:55.8771136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8771213Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8771479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8771546Z layer_outputs = layer_module( 2025-08-14T21:53:55.8771826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8771914Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8772178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8772296Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8772562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8772650Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8772655Z 2025-08-14T21:53:55.8772754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8772954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8773020Z return mod(**inputs) 2025-08-14T21:53:55.8773297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8773377Z outputs = self.mobilebert( 2025-08-14T21:53:55.8773658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8773727Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8774002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8774070Z layer_outputs = layer_module( 2025-08-14T21:53:55.8774361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8774452Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8774718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8774870Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8775136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8775261Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8775545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8775633Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8775638Z 2025-08-14T21:53:55.8775741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8775941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8776014Z return mod(**inputs) 2025-08-14T21:53:55.8776286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8776358Z outputs = self.mobilebert( 2025-08-14T21:53:55.8776627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8776697Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8776965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8777044Z layer_outputs = layer_module( 2025-08-14T21:53:55.8777308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8777405Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8777674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8777795Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8778061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8778140Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8778144Z 2025-08-14T21:53:55.8778246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8778434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8778498Z return mod(**inputs) 2025-08-14T21:53:55.8778771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8778841Z outputs = self.mobilebert( 2025-08-14T21:53:55.8779106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8779182Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8779447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8779523Z layer_outputs = layer_module( 2025-08-14T21:53:55.8779789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8779878Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8780150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8780272Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8780544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8780651Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8780673Z 2025-08-14T21:53:55.8780771Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8780967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8781030Z return mod(**inputs) 2025-08-14T21:53:55.8781298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8781374Z outputs = self.mobilebert( 2025-08-14T21:53:55.8781653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8781730Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8782011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8782080Z layer_outputs = layer_module( 2025-08-14T21:53:55.8782355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8782446Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8782721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8782841Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8783112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8783199Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8783204Z 2025-08-14T21:53:55.8783304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8783492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8783563Z return mod(**inputs) 2025-08-14T21:53:55.8783836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8783910Z outputs = self.mobilebert( 2025-08-14T21:53:55.8784177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8784246Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8784525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8784592Z layer_outputs = layer_module( 2025-08-14T21:53:55.8784868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8784956Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8785226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8785351Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8785619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8785734Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8786011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8786099Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8786120Z 2025-08-14T21:53:55.8786227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8786416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8786479Z return mod(**inputs) 2025-08-14T21:53:55.8786753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8786839Z outputs = self.mobilebert( 2025-08-14T21:53:55.8787121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8787192Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8787467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8787558Z layer_outputs = layer_module( 2025-08-14T21:53:55.8787829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8787933Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8788204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8788313Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8788586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8788667Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8788670Z 2025-08-14T21:53:55.8788768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8788971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8789038Z return mod(**inputs) 2025-08-14T21:53:55.8789321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8789395Z outputs = self.mobilebert( 2025-08-14T21:53:55.8789671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8789753Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8790029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8790100Z layer_outputs = layer_module( 2025-08-14T21:53:55.8790382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8790474Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8790754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8790871Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8791159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8791285Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8791290Z 2025-08-14T21:53:55.8791398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8791612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8791681Z return mod(**inputs) 2025-08-14T21:53:55.8791974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8792058Z outputs = self.mobilebert( 2025-08-14T21:53:55.8792364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8792465Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8792754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8792827Z layer_outputs = layer_module( 2025-08-14T21:53:55.8793122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8793236Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8793526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8793663Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8793982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8794077Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8794082Z 2025-08-14T21:53:55.8794205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8794413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8794489Z return mod(**inputs) 2025-08-14T21:53:55.8794781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8794864Z outputs = self.mobilebert( 2025-08-14T21:53:55.8795151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8795225Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8795523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8795595Z layer_outputs = layer_module( 2025-08-14T21:53:55.8796128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8796238Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8796540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8796680Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8796981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8797109Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8797420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8797517Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8797521Z 2025-08-14T21:53:55.8797638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8797857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8797927Z return mod(**inputs) 2025-08-14T21:53:55.8798236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8798315Z outputs = self.mobilebert( 2025-08-14T21:53:55.8798615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8798701Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8799053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8799140Z layer_outputs = layer_module( 2025-08-14T21:53:55.8799453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8799623Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8799940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8800073Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8800077Z 2025-08-14T21:53:55.8800192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8800411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8800479Z return mod(**inputs) 2025-08-14T21:53:55.8800778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8800870Z outputs = self.mobilebert( 2025-08-14T21:53:55.8801170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8801268Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8801566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8801648Z layer_outputs = layer_module( 2025-08-14T21:53:55.8801950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8802075Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8802385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8802497Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8802500Z 2025-08-14T21:53:55.8802616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8802821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8802893Z return mod(**inputs) 2025-08-14T21:53:55.8803191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8803266Z outputs = self.mobilebert( 2025-08-14T21:53:55.8803565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8803647Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8803946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8804025Z layer_outputs = layer_module( 2025-08-14T21:53:55.8804324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8804491Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8804795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8804895Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8804900Z 2025-08-14T21:53:55.8805013Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8805228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8805296Z return mod(**inputs) 2025-08-14T21:53:55.8805594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8805669Z outputs = self.mobilebert( 2025-08-14T21:53:55.8805973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8806065Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8806353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8806435Z layer_outputs = layer_module( 2025-08-14T21:53:55.8806722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8806905Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8807212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8807340Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8807646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8807745Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8807750Z 2025-08-14T21:53:55.8807873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8808087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8808160Z return mod(**inputs) 2025-08-14T21:53:55.8808466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8808542Z outputs = self.mobilebert( 2025-08-14T21:53:55.8809025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8809114Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8809419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8809494Z layer_outputs = layer_module( 2025-08-14T21:53:55.8809795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8809962Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8810260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8810389Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8810691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8810790Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8810794Z 2025-08-14T21:53:55.8810901Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8811117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8811188Z return mod(**inputs) 2025-08-14T21:53:55.8811481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8811563Z outputs = self.mobilebert( 2025-08-14T21:53:55.8811851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8811927Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8812226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8812300Z layer_outputs = layer_module( 2025-08-14T21:53:55.8812602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8812754Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8813062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8813190Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8813453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8813597Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8813866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8813955Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8813958Z 2025-08-14T21:53:55.8814065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8814283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8814361Z return mod(**inputs) 2025-08-14T21:53:55.8814656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8814729Z outputs = self.mobilebert( 2025-08-14T21:53:55.8815010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8815094Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8815358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8815434Z layer_outputs = layer_module( 2025-08-14T21:53:55.8815697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8815858Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8816122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8816232Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8816505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8816587Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8816590Z 2025-08-14T21:53:55.8816695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8816881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8816946Z return mod(**inputs) 2025-08-14T21:53:55.8817221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8817289Z outputs = self.mobilebert( 2025-08-14T21:53:55.8817551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8817629Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8817891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8817967Z layer_outputs = layer_module( 2025-08-14T21:53:55.8818230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8818381Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8818649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8818755Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8819025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8819130Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8819391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8819845Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8819849Z 2025-08-14T21:53:55.8819948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8820143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8820206Z return mod(**inputs) 2025-08-14T21:53:55.8820474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8820564Z outputs = self.mobilebert( 2025-08-14T21:53:55.8820833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8820906Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8821205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8821275Z layer_outputs = layer_module( 2025-08-14T21:53:55.8821547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8821628Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8821890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8821967Z self_outputs = self.self( 2025-08-14T21:53:55.8822232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8822302Z self.query(query_tensor) 2025-08-14T21:53:55.8822315Z 2025-08-14T21:53:55.8822416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8822603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8822674Z return mod(**inputs) 2025-08-14T21:53:55.8822940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8823011Z outputs = self.mobilebert( 2025-08-14T21:53:55.8823279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8823350Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8823623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8823692Z layer_outputs = layer_module( 2025-08-14T21:53:55.8823958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8824047Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8824311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8824381Z self_outputs = self.self( 2025-08-14T21:53:55.8824653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8824719Z self.key(key_tensor) 2025-08-14T21:53:55.8824722Z 2025-08-14T21:53:55.8824825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8825013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8825077Z return mod(**inputs) 2025-08-14T21:53:55.8825351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8825439Z outputs = self.mobilebert( 2025-08-14T21:53:55.8825737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8825813Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8826120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8826199Z layer_outputs = layer_module( 2025-08-14T21:53:55.8826488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8826576Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8826888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8826959Z self_outputs = self.self( 2025-08-14T21:53:55.8827251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8827323Z self.value(value_tensor) 2025-08-14T21:53:55.8827327Z 2025-08-14T21:53:55.8827415Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8827499Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8827599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8827787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8827858Z return mod(**inputs) 2025-08-14T21:53:55.8828130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8828215Z outputs = self.mobilebert( 2025-08-14T21:53:55.8828480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8828550Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8828829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8828896Z layer_outputs = layer_module( 2025-08-14T21:53:55.8829170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8829252Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8829516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8829640Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8829906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8829990Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8830000Z 2025-08-14T21:53:55.8830099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8830286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8830357Z return mod(**inputs) 2025-08-14T21:53:55.8830627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8830695Z outputs = self.mobilebert( 2025-08-14T21:53:55.8830970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8831040Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8831318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8831388Z layer_outputs = layer_module( 2025-08-14T21:53:55.8831681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8831845Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8832117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8832246Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8832525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8832607Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8832611Z 2025-08-14T21:53:55.8832739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8832948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8833018Z return mod(**inputs) 2025-08-14T21:53:55.8833331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8833407Z outputs = self.mobilebert( 2025-08-14T21:53:55.8833703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8833780Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8834080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8834162Z layer_outputs = layer_module( 2025-08-14T21:53:55.8834465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8834552Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8834854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8834983Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8835286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8835418Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8835782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8835893Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8835897Z 2025-08-14T21:53:55.8836004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8836226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8836297Z return mod(**inputs) 2025-08-14T21:53:55.8836601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8836689Z outputs = self.mobilebert( 2025-08-14T21:53:55.8836989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8837077Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8837386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8837463Z layer_outputs = layer_module( 2025-08-14T21:53:55.8837775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8837878Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8838190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8838326Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8838586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8838691Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8838694Z 2025-08-14T21:53:55.8838789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8838970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8839039Z return mod(**inputs) 2025-08-14T21:53:55.8839300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8839387Z outputs = self.mobilebert( 2025-08-14T21:53:55.8839646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8839716Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8840001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8840071Z layer_outputs = layer_module( 2025-08-14T21:53:55.8840342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8840440Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8840713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8840828Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8841103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8841214Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8841217Z 2025-08-14T21:53:55.8841324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8841515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8841588Z return mod(**inputs) 2025-08-14T21:53:55.8841863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8841929Z outputs = self.mobilebert( 2025-08-14T21:53:55.8842195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8842262Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8842524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8842599Z layer_outputs = layer_module( 2025-08-14T21:53:55.8842860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8842954Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8843214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8843333Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8843600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8843679Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8843682Z 2025-08-14T21:53:55.8843784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8843973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8844051Z return mod(**inputs) 2025-08-14T21:53:55.8844321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8844389Z outputs = self.mobilebert( 2025-08-14T21:53:55.8844647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8844738Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8844997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8845071Z layer_outputs = layer_module( 2025-08-14T21:53:55.8845328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8845440Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8845710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8845842Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8846109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8846223Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8846498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8846594Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8846597Z 2025-08-14T21:53:55.8846695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8846895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8846958Z return mod(**inputs) 2025-08-14T21:53:55.8847234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8847312Z outputs = self.mobilebert( 2025-08-14T21:53:55.8847583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8847657Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8847941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8848011Z layer_outputs = layer_module( 2025-08-14T21:53:55.8848294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8848399Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8848661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8848781Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8849046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8849134Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8849137Z 2025-08-14T21:53:55.8849234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8849422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8849491Z return mod(**inputs) 2025-08-14T21:53:55.8849757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8849825Z outputs = self.mobilebert( 2025-08-14T21:53:55.8850098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8850183Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8850458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8850527Z layer_outputs = layer_module( 2025-08-14T21:53:55.8850807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8850901Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8851168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8851278Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8851560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8851668Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8851672Z 2025-08-14T21:53:55.8851789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8851980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8852042Z return mod(**inputs) 2025-08-14T21:53:55.8852319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8852387Z outputs = self.mobilebert( 2025-08-14T21:53:55.8852659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8852727Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8852993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8853071Z layer_outputs = layer_module( 2025-08-14T21:53:55.8853340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8853436Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8853702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8853823Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8854104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8854187Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8854191Z 2025-08-14T21:53:55.8854287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8854485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8854549Z return mod(**inputs) 2025-08-14T21:53:55.8854827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8854896Z outputs = self.mobilebert( 2025-08-14T21:53:55.8855162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8855241Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8855507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8855581Z layer_outputs = layer_module( 2025-08-14T21:53:55.8855848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8855937Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8856210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8856349Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8856615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8856754Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8857016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8857113Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8857116Z 2025-08-14T21:53:55.8857214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8857415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8857487Z return mod(**inputs) 2025-08-14T21:53:55.8857776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8857853Z outputs = self.mobilebert( 2025-08-14T21:53:55.8858119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8858191Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8858464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8858533Z layer_outputs = layer_module( 2025-08-14T21:53:55.8858798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8858896Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8859161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8859281Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8859548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8859629Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8859634Z 2025-08-14T21:53:55.8859740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8859931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8860001Z return mod(**inputs) 2025-08-14T21:53:55.8860270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8860337Z outputs = self.mobilebert( 2025-08-14T21:53:55.8860610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8860681Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8860946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8861021Z layer_outputs = layer_module( 2025-08-14T21:53:55.8861287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8861381Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8861646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8861753Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8862028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8862151Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8862155Z 2025-08-14T21:53:55.8862264Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8862454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8862518Z return mod(**inputs) 2025-08-14T21:53:55.8862821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8862893Z outputs = self.mobilebert( 2025-08-14T21:53:55.8863174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8863246Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8863534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8863613Z layer_outputs = layer_module( 2025-08-14T21:53:55.8863901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8864001Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8864263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8864380Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8864645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8864723Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8864727Z 2025-08-14T21:53:55.8864823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8865039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8865102Z return mod(**inputs) 2025-08-14T21:53:55.8865369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8865437Z outputs = self.mobilebert( 2025-08-14T21:53:55.8865692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8865768Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8866025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8866091Z layer_outputs = layer_module( 2025-08-14T21:53:55.8866356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8866444Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8866708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8866825Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8867081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8867204Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8867460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8867554Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8867557Z 2025-08-14T21:53:55.8867655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8867844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8867915Z return mod(**inputs) 2025-08-14T21:53:55.8868209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8868281Z outputs = self.mobilebert( 2025-08-14T21:53:55.8868559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8868657Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8868955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8869028Z layer_outputs = layer_module( 2025-08-14T21:53:55.8869328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8869475Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8869768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8869879Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8869883Z 2025-08-14T21:53:55.8869991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8870195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8870274Z return mod(**inputs) 2025-08-14T21:53:55.8870566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8870641Z outputs = self.mobilebert( 2025-08-14T21:53:55.8870946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8871020Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8871320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8871395Z layer_outputs = layer_module( 2025-08-14T21:53:55.8871684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8871817Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8872111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8872234Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8872238Z 2025-08-14T21:53:55.8872346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8872553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8872631Z return mod(**inputs) 2025-08-14T21:53:55.8872927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8873004Z outputs = self.mobilebert( 2025-08-14T21:53:55.8873307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8873385Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8873690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8873767Z layer_outputs = layer_module( 2025-08-14T21:53:55.8874073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8874254Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8874571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8874715Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8874720Z 2025-08-14T21:53:55.8874831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8875041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8875120Z return mod(**inputs) 2025-08-14T21:53:55.8875446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8875522Z outputs = self.mobilebert( 2025-08-14T21:53:55.8875923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8876007Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8876345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8876424Z layer_outputs = layer_module( 2025-08-14T21:53:55.8876740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8876922Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8877228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8877376Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8877643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8877730Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8877734Z 2025-08-14T21:53:55.8877841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8878038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8878112Z return mod(**inputs) 2025-08-14T21:53:55.8878375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8878444Z outputs = self.mobilebert( 2025-08-14T21:53:55.8878715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8878788Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8879046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8879125Z layer_outputs = layer_module( 2025-08-14T21:53:55.8879390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8879547Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8879814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8879931Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8880202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8880287Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8880290Z 2025-08-14T21:53:55.8880396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8880586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8880652Z return mod(**inputs) 2025-08-14T21:53:55.8880937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8881015Z outputs = self.mobilebert( 2025-08-14T21:53:55.8881308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8881384Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8881652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8881747Z layer_outputs = layer_module( 2025-08-14T21:53:55.8882011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8882162Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8882441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8882573Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8882864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8882982Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8883249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8883347Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8883351Z 2025-08-14T21:53:55.8883453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8883653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8883717Z return mod(**inputs) 2025-08-14T21:53:55.8883991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8884069Z outputs = self.mobilebert( 2025-08-14T21:53:55.8884344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8884415Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8884695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8884766Z layer_outputs = layer_module( 2025-08-14T21:53:55.8885046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8885205Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8885478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8885597Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8885872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8885964Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8885968Z 2025-08-14T21:53:55.8886069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8886260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8886335Z return mod(**inputs) 2025-08-14T21:53:55.8886610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8886679Z outputs = self.mobilebert( 2025-08-14T21:53:55.8886959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8887032Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8887316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8887404Z layer_outputs = layer_module( 2025-08-14T21:53:55.8887683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8887846Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8888133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8888258Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8888525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8888625Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8888903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8889008Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8889011Z 2025-08-14T21:53:55.8889119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8889307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8889372Z return mod(**inputs) 2025-08-14T21:53:55.8889646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8889714Z outputs = self.mobilebert( 2025-08-14T21:53:55.8889977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8890054Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8890324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8890404Z layer_outputs = layer_module( 2025-08-14T21:53:55.8890683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8890769Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8891051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8891123Z self_outputs = self.self( 2025-08-14T21:53:55.8891393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8891473Z self.query(query_tensor) 2025-08-14T21:53:55.8891476Z 2025-08-14T21:53:55.8891577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8891778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8891846Z return mod(**inputs) 2025-08-14T21:53:55.8892121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8892198Z outputs = self.mobilebert( 2025-08-14T21:53:55.8892470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8892560Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8892839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8892907Z layer_outputs = layer_module( 2025-08-14T21:53:55.8893184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8893268Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8893542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8893646Z self_outputs = self.self( 2025-08-14T21:53:55.8893918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8894012Z self.key(key_tensor) 2025-08-14T21:53:55.8894016Z 2025-08-14T21:53:55.8894122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8894327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8894404Z return mod(**inputs) 2025-08-14T21:53:55.8894709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8894789Z outputs = self.mobilebert( 2025-08-14T21:53:55.8895091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8895169Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8895480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8895554Z layer_outputs = layer_module( 2025-08-14T21:53:55.8895846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8895943Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8896246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8896326Z self_outputs = self.self( 2025-08-14T21:53:55.8896628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8896703Z self.value(value_tensor) 2025-08-14T21:53:55.8896708Z 2025-08-14T21:53:55.8896803Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8896886Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8896993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8897202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8897273Z return mod(**inputs) 2025-08-14T21:53:55.8897579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8897652Z outputs = self.mobilebert( 2025-08-14T21:53:55.8897952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8898035Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8898336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8898419Z layer_outputs = layer_module( 2025-08-14T21:53:55.8898706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8898793Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8899087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8899214Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8899500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8899596Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8899599Z 2025-08-14T21:53:55.8899703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8899917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8900030Z return mod(**inputs) 2025-08-14T21:53:55.8900320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8900403Z outputs = self.mobilebert( 2025-08-14T21:53:55.8900710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8900791Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8901079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8901152Z layer_outputs = layer_module( 2025-08-14T21:53:55.8901464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8901630Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8901936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8902059Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8902348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8902443Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8902447Z 2025-08-14T21:53:55.8902553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8902754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8902832Z return mod(**inputs) 2025-08-14T21:53:55.8903131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8903212Z outputs = self.mobilebert( 2025-08-14T21:53:55.8903505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8903580Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8903878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8903953Z layer_outputs = layer_module( 2025-08-14T21:53:55.8904244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8904339Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8904630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8904767Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8905056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8905190Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8905491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8905587Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8905591Z 2025-08-14T21:53:55.8905707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8905912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8905981Z return mod(**inputs) 2025-08-14T21:53:55.8906285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8906362Z outputs = self.mobilebert( 2025-08-14T21:53:55.8906668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8906767Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8907067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8907167Z layer_outputs = layer_module( 2025-08-14T21:53:55.8907463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8907563Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8907870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8908007Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8908315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8908406Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8908431Z 2025-08-14T21:53:55.8908541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8908943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8909025Z return mod(**inputs) 2025-08-14T21:53:55.8909338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8909416Z outputs = self.mobilebert( 2025-08-14T21:53:55.8909718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8909807Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8910122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8910200Z layer_outputs = layer_module( 2025-08-14T21:53:55.8910514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8910615Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8910928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8911049Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8911361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8911500Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8911504Z 2025-08-14T21:53:55.8911614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8911828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8911899Z return mod(**inputs) 2025-08-14T21:53:55.8912197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8912279Z outputs = self.mobilebert( 2025-08-14T21:53:55.8912573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8912648Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8912967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8913042Z layer_outputs = layer_module( 2025-08-14T21:53:55.8913362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8913462Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8913808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8913952Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8914256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8914377Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8914381Z 2025-08-14T21:53:55.8914491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8914701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8914779Z return mod(**inputs) 2025-08-14T21:53:55.8915103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8915182Z outputs = self.mobilebert( 2025-08-14T21:53:55.8915523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8915603Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8915987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8916072Z layer_outputs = layer_module( 2025-08-14T21:53:55.8916383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8916491Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8916787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8916929Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8917238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8917371Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8917679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8917779Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8917783Z 2025-08-14T21:53:55.8917901Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8918114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8918184Z return mod(**inputs) 2025-08-14T21:53:55.8918493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8918571Z outputs = self.mobilebert( 2025-08-14T21:53:55.8918873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8918963Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8919260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8919348Z layer_outputs = layer_module( 2025-08-14T21:53:55.8919646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8919744Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8920051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8920172Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8920480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8920592Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8920598Z 2025-08-14T21:53:55.8920706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8920925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8921012Z return mod(**inputs) 2025-08-14T21:53:55.8921313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8921397Z outputs = self.mobilebert( 2025-08-14T21:53:55.8921695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8921779Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8922092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8922171Z layer_outputs = layer_module( 2025-08-14T21:53:55.8922492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8922593Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8922899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8923018Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8923316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8923443Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8923447Z 2025-08-14T21:53:55.8923559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8923769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8923849Z return mod(**inputs) 2025-08-14T21:53:55.8924152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8924237Z outputs = self.mobilebert( 2025-08-14T21:53:55.8924536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8924613Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8924916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8924984Z layer_outputs = layer_module( 2025-08-14T21:53:55.8925256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8925343Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8925627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8925763Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8926052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8926150Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8926160Z 2025-08-14T21:53:55.8926261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8926454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8926525Z return mod(**inputs) 2025-08-14T21:53:55.8926801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8926873Z outputs = self.mobilebert( 2025-08-14T21:53:55.8927170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8927241Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8927528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8927612Z layer_outputs = layer_module( 2025-08-14T21:53:55.8927876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8927974Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8928235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8928369Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8928641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8928777Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8929061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8929156Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8929160Z 2025-08-14T21:53:55.8929260Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8929470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8929533Z return mod(**inputs) 2025-08-14T21:53:55.8929807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8929877Z outputs = self.mobilebert( 2025-08-14T21:53:55.8930141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8930220Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8930485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8930555Z layer_outputs = layer_module( 2025-08-14T21:53:55.8930834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8930927Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8931210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8931318Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8931593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8931686Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8931691Z 2025-08-14T21:53:55.8931793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8931993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8932060Z return mod(**inputs) 2025-08-14T21:53:55.8932337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8932415Z outputs = self.mobilebert( 2025-08-14T21:53:55.8932718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8932793Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8933097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8933182Z layer_outputs = layer_module( 2025-08-14T21:53:55.8933467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8933557Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8933837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8933968Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8934243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8934357Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8934361Z 2025-08-14T21:53:55.8934485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8934671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8934743Z return mod(**inputs) 2025-08-14T21:53:55.8935029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8935105Z outputs = self.mobilebert( 2025-08-14T21:53:55.8935401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8935478Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8935776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8935850Z layer_outputs = layer_module( 2025-08-14T21:53:55.8936141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8936245Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8936540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8936677Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8936963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8937047Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8937050Z 2025-08-14T21:53:55.8937155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8937344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8937413Z return mod(**inputs) 2025-08-14T21:53:55.8937684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8937752Z outputs = self.mobilebert( 2025-08-14T21:53:55.8938031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8938103Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8938378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8938459Z layer_outputs = layer_module( 2025-08-14T21:53:55.8938737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8938835Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8939111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8939235Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8939517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8939654Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8939931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8940039Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8940043Z 2025-08-14T21:53:55.8940143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8940346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8940410Z return mod(**inputs) 2025-08-14T21:53:55.8940687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8940791Z outputs = self.mobilebert( 2025-08-14T21:53:55.8941065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8941162Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8941444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8941520Z layer_outputs = layer_module( 2025-08-14T21:53:55.8941814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8941939Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8942249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8942335Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8942340Z 2025-08-14T21:53:55.8942446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8942661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8942732Z return mod(**inputs) 2025-08-14T21:53:55.8943026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8943109Z outputs = self.mobilebert( 2025-08-14T21:53:55.8943395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8943472Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8943744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8943813Z layer_outputs = layer_module( 2025-08-14T21:53:55.8944094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.8944214Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.8944498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8944606Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8944611Z 2025-08-14T21:53:55.8944712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8944915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8944979Z return mod(**inputs) 2025-08-14T21:53:55.8945254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8945332Z outputs = self.mobilebert( 2025-08-14T21:53:55.8945610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8945703Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8945979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8946050Z layer_outputs = layer_module( 2025-08-14T21:53:55.8946333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8946505Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8946791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.8946885Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.8946889Z 2025-08-14T21:53:55.8947005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8947209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8947275Z return mod(**inputs) 2025-08-14T21:53:55.8947565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8947646Z outputs = self.mobilebert( 2025-08-14T21:53:55.8947919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8948000Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8948273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8948342Z layer_outputs = layer_module( 2025-08-14T21:53:55.8948624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8948781Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8949069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.8949192Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.8949466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8949566Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8949570Z 2025-08-14T21:53:55.8949670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8949871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8949935Z return mod(**inputs) 2025-08-14T21:53:55.8950213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8950288Z outputs = self.mobilebert( 2025-08-14T21:53:55.8950563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8950636Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8950917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8950987Z layer_outputs = layer_module( 2025-08-14T21:53:55.8951275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8951440Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8951730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8951865Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8952181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.8952276Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8952279Z 2025-08-14T21:53:55.8952385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8952606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8952682Z return mod(**inputs) 2025-08-14T21:53:55.8952972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8953047Z outputs = self.mobilebert( 2025-08-14T21:53:55.8953371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8953446Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8953748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8953837Z layer_outputs = layer_module( 2025-08-14T21:53:55.8954124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.8954294Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.8954594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.8954728Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.8955029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.8955156Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8955452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8955550Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8955553Z 2025-08-14T21:53:55.8955724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8955940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8956009Z return mod(**inputs) 2025-08-14T21:53:55.8956315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8956393Z outputs = self.mobilebert( 2025-08-14T21:53:55.8956702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8956791Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8957089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8957176Z layer_outputs = layer_module( 2025-08-14T21:53:55.8957471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8957645Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8957944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8958053Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8958333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8958419Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8958423Z 2025-08-14T21:53:55.8958524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8958745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8958813Z return mod(**inputs) 2025-08-14T21:53:55.8959088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8959186Z outputs = self.mobilebert( 2025-08-14T21:53:55.8959476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8959552Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8959827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8959901Z layer_outputs = layer_module( 2025-08-14T21:53:55.8960194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8960362Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8960635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.8960749Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.8961009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.8961097Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.8961356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8961439Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8961451Z 2025-08-14T21:53:55.8961547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8961732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8961801Z return mod(**inputs) 2025-08-14T21:53:55.8962061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8962127Z outputs = self.mobilebert( 2025-08-14T21:53:55.8962392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8962459Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8962721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8962787Z layer_outputs = layer_module( 2025-08-14T21:53:55.8963045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8963134Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8963398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8963464Z self_outputs = self.self( 2025-08-14T21:53:55.8963726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.8963795Z self.query(query_tensor) 2025-08-14T21:53:55.8963798Z 2025-08-14T21:53:55.8963900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8964082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8964144Z return mod(**inputs) 2025-08-14T21:53:55.8964412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8964477Z outputs = self.mobilebert( 2025-08-14T21:53:55.8964759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8964828Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8965090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8965192Z layer_outputs = layer_module( 2025-08-14T21:53:55.8965452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8965536Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8965806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8965891Z self_outputs = self.self( 2025-08-14T21:53:55.8966173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.8966241Z self.key(key_tensor) 2025-08-14T21:53:55.8966259Z 2025-08-14T21:53:55.8966362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8966559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8966624Z return mod(**inputs) 2025-08-14T21:53:55.8966902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8966970Z outputs = self.mobilebert( 2025-08-14T21:53:55.8967241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8967318Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8967591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8967662Z layer_outputs = layer_module( 2025-08-14T21:53:55.8967953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8968035Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8968305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.8968373Z self_outputs = self.self( 2025-08-14T21:53:55.8968634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.8968713Z self.value(value_tensor) 2025-08-14T21:53:55.8968716Z 2025-08-14T21:53:55.8968794Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8968871Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.8968978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8969163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8969235Z return mod(**inputs) 2025-08-14T21:53:55.8969501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8969569Z outputs = self.mobilebert( 2025-08-14T21:53:55.8969840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8969907Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8970175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8970243Z layer_outputs = layer_module( 2025-08-14T21:53:55.8970513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8970616Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8970878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8970996Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8971260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.8971357Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8971360Z 2025-08-14T21:53:55.8971466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8971651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8971714Z return mod(**inputs) 2025-08-14T21:53:55.8972003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8972075Z outputs = self.mobilebert( 2025-08-14T21:53:55.8972362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8972432Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8972707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8972786Z layer_outputs = layer_module( 2025-08-14T21:53:55.8973060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.8973217Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.8973508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.8973618Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.8973916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.8973997Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.8974000Z 2025-08-14T21:53:55.8974098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8974296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8974359Z return mod(**inputs) 2025-08-14T21:53:55.8974646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8974716Z outputs = self.mobilebert( 2025-08-14T21:53:55.8974996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8975073Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8975353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8975423Z layer_outputs = layer_module( 2025-08-14T21:53:55.8975707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.8975792Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.8976076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.8976197Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.8976478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.8976612Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8976889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8977003Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8977007Z 2025-08-14T21:53:55.8977106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8977298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8977390Z return mod(**inputs) 2025-08-14T21:53:55.8977667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8977744Z outputs = self.mobilebert( 2025-08-14T21:53:55.8978024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8978112Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8978396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8978483Z layer_outputs = layer_module( 2025-08-14T21:53:55.8978764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8978868Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8979147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8979266Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8979548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8979631Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8979634Z 2025-08-14T21:53:55.8979744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8979941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8980013Z return mod(**inputs) 2025-08-14T21:53:55.8980292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8980361Z outputs = self.mobilebert( 2025-08-14T21:53:55.8980647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8980718Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8980998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8981075Z layer_outputs = layer_module( 2025-08-14T21:53:55.8981353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8981454Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8981733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8981844Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8982151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8982268Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8982272Z 2025-08-14T21:53:55.8982384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8982603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8982668Z return mod(**inputs) 2025-08-14T21:53:55.8982961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8983049Z outputs = self.mobilebert( 2025-08-14T21:53:55.8983328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8983408Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8983683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8983774Z layer_outputs = layer_module( 2025-08-14T21:53:55.8984051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8984141Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8984435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8984560Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8984859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8984944Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8984947Z 2025-08-14T21:53:55.8985048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8985251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8985316Z return mod(**inputs) 2025-08-14T21:53:55.8985594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8985674Z outputs = self.mobilebert( 2025-08-14T21:53:55.8985947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8986024Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8986300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8986373Z layer_outputs = layer_module( 2025-08-14T21:53:55.8986650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8986742Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8987023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8987145Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8987419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8987546Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8987821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8987915Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8987925Z 2025-08-14T21:53:55.8988026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8988221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8988294Z return mod(**inputs) 2025-08-14T21:53:55.8988572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8988647Z outputs = self.mobilebert( 2025-08-14T21:53:55.8988952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8989030Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8989336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8989435Z layer_outputs = layer_module( 2025-08-14T21:53:55.8989746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8989852Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8990168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8990285Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8990591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.8990678Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.8990681Z 2025-08-14T21:53:55.8990812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8991018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8991104Z return mod(**inputs) 2025-08-14T21:53:55.8991418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8991493Z outputs = self.mobilebert( 2025-08-14T21:53:55.8991799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8991875Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8992170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8992252Z layer_outputs = layer_module( 2025-08-14T21:53:55.8992550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8992649Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8992953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.8993069Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.8993372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.8993490Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.8993493Z 2025-08-14T21:53:55.8993602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8993827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8993895Z return mod(**inputs) 2025-08-14T21:53:55.8994203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8994281Z outputs = self.mobilebert( 2025-08-14T21:53:55.8994582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8994667Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8994963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8995040Z layer_outputs = layer_module( 2025-08-14T21:53:55.8995352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8995451Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8995846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8995987Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8996313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.8996414Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.8996418Z 2025-08-14T21:53:55.8996529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.8996762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.8996834Z return mod(**inputs) 2025-08-14T21:53:55.8997132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.8997219Z outputs = self.mobilebert( 2025-08-14T21:53:55.8997535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.8997623Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.8997935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.8998059Z layer_outputs = layer_module( 2025-08-14T21:53:55.8998359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.8998459Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.8998760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.8998897Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.8999189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.8999326Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.8999616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.8999713Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.8999717Z 2025-08-14T21:53:55.8999831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9000037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9000117Z return mod(**inputs) 2025-08-14T21:53:55.9000411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9000486Z outputs = self.mobilebert( 2025-08-14T21:53:55.9000789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9000866Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9001155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9001235Z layer_outputs = layer_module( 2025-08-14T21:53:55.9001525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9001627Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9001919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9002036Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9002333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9002420Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9002424Z 2025-08-14T21:53:55.9002540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9002744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9002832Z return mod(**inputs) 2025-08-14T21:53:55.9003137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9003210Z outputs = self.mobilebert( 2025-08-14T21:53:55.9003532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9003617Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9003917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9004010Z layer_outputs = layer_module( 2025-08-14T21:53:55.9004331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9004429Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9004741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9004858Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9005155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9005271Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9005275Z 2025-08-14T21:53:55.9005380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9005598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9005666Z return mod(**inputs) 2025-08-14T21:53:55.9005973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9006056Z outputs = self.mobilebert( 2025-08-14T21:53:55.9006355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9006440Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9006740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9006815Z layer_outputs = layer_module( 2025-08-14T21:53:55.9007118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9007214Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9007521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9007652Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9007951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9008045Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9008049Z 2025-08-14T21:53:55.9008157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9008374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9008443Z return mod(**inputs) 2025-08-14T21:53:55.9008888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9008976Z outputs = self.mobilebert( 2025-08-14T21:53:55.9009268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9009347Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9009649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9009764Z layer_outputs = layer_module( 2025-08-14T21:53:55.9010061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9010180Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9010469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9010604Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9010890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9011047Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9011342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9011471Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9011476Z 2025-08-14T21:53:55.9011594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9011801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9011871Z return mod(**inputs) 2025-08-14T21:53:55.9012173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9012249Z outputs = self.mobilebert( 2025-08-14T21:53:55.9012546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9012623Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9012913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9012997Z layer_outputs = layer_module( 2025-08-14T21:53:55.9013292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9013419Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9013693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9013777Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9013781Z 2025-08-14T21:53:55.9013892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9014086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9014151Z return mod(**inputs) 2025-08-14T21:53:55.9014436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9014506Z outputs = self.mobilebert( 2025-08-14T21:53:55.9014788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9014860Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9015134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9015210Z layer_outputs = layer_module( 2025-08-14T21:53:55.9015485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9015611Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9015888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9016014Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9016018Z 2025-08-14T21:53:55.9016127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9016320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9016387Z return mod(**inputs) 2025-08-14T21:53:55.9016705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9016779Z outputs = self.mobilebert( 2025-08-14T21:53:55.9017081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9017154Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9017472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9017554Z layer_outputs = layer_module( 2025-08-14T21:53:55.9017858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9018032Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9018330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9018422Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9018425Z 2025-08-14T21:53:55.9018530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9018715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9018777Z return mod(**inputs) 2025-08-14T21:53:55.9019054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9019121Z outputs = self.mobilebert( 2025-08-14T21:53:55.9019391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9019460Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9019730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9019808Z layer_outputs = layer_module( 2025-08-14T21:53:55.9020079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9020252Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9020554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9020682Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9020982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9021079Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9021083Z 2025-08-14T21:53:55.9021195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9021400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9021463Z return mod(**inputs) 2025-08-14T21:53:55.9021738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9021806Z outputs = self.mobilebert( 2025-08-14T21:53:55.9022085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9022165Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9022456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9022535Z layer_outputs = layer_module( 2025-08-14T21:53:55.9022812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9022982Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9023266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9023390Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9023676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9023775Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9023778Z 2025-08-14T21:53:55.9023881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9024096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9024162Z return mod(**inputs) 2025-08-14T21:53:55.9024442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9024521Z outputs = self.mobilebert( 2025-08-14T21:53:55.9024795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9024875Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9025152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9025223Z layer_outputs = layer_module( 2025-08-14T21:53:55.9025505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9025668Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9025964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9026092Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9026382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9026514Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9026807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9026904Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9026914Z 2025-08-14T21:53:55.9027021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9027230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9027304Z return mod(**inputs) 2025-08-14T21:53:55.9027603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9027674Z outputs = self.mobilebert( 2025-08-14T21:53:55.9027956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9028029Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9028307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9028378Z layer_outputs = layer_module( 2025-08-14T21:53:55.9028653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9028837Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9029109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9029243Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9029513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9029595Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9029599Z 2025-08-14T21:53:55.9029705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9029897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9029979Z return mod(**inputs) 2025-08-14T21:53:55.9030261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9030350Z outputs = self.mobilebert( 2025-08-14T21:53:55.9030629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9030704Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9030974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9031052Z layer_outputs = layer_module( 2025-08-14T21:53:55.9031324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9031486Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9031765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9031879Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9032173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9032262Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9032549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9032653Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9032656Z 2025-08-14T21:53:55.9032761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9032970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9033037Z return mod(**inputs) 2025-08-14T21:53:55.9033329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9033413Z outputs = self.mobilebert( 2025-08-14T21:53:55.9033701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9033782Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9034068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9034141Z layer_outputs = layer_module( 2025-08-14T21:53:55.9034437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9034527Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9034817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9034898Z self_outputs = self.self( 2025-08-14T21:53:55.9035206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9035288Z self.query(query_tensor) 2025-08-14T21:53:55.9035292Z 2025-08-14T21:53:55.9035397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9035616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9035749Z return mod(**inputs) 2025-08-14T21:53:55.9036053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9036135Z outputs = self.mobilebert( 2025-08-14T21:53:55.9036442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9036550Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9036868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9036959Z layer_outputs = layer_module( 2025-08-14T21:53:55.9037251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9037364Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9037641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9037718Z self_outputs = self.self( 2025-08-14T21:53:55.9037996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9038065Z self.key(key_tensor) 2025-08-14T21:53:55.9038069Z 2025-08-14T21:53:55.9038182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9038377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9038454Z return mod(**inputs) 2025-08-14T21:53:55.9038731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9038800Z outputs = self.mobilebert( 2025-08-14T21:53:55.9039080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9039152Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9039423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9039501Z layer_outputs = layer_module( 2025-08-14T21:53:55.9039773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9039865Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9040139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9040208Z self_outputs = self.self( 2025-08-14T21:53:55.9040485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9040555Z self.value(value_tensor) 2025-08-14T21:53:55.9040558Z 2025-08-14T21:53:55.9040643Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9040720Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9040821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9041019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9041083Z return mod(**inputs) 2025-08-14T21:53:55.9041361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9041456Z outputs = self.mobilebert( 2025-08-14T21:53:55.9041728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9041808Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9042097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9042165Z layer_outputs = layer_module( 2025-08-14T21:53:55.9042446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9042528Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9042820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9042950Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9043241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9043332Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9043335Z 2025-08-14T21:53:55.9043435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9043627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9043699Z return mod(**inputs) 2025-08-14T21:53:55.9043975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9044052Z outputs = self.mobilebert( 2025-08-14T21:53:55.9044323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9044395Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9044677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9044748Z layer_outputs = layer_module( 2025-08-14T21:53:55.9045018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9045185Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9045462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9045578Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9045852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9045933Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9045937Z 2025-08-14T21:53:55.9046044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9046238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9046312Z return mod(**inputs) 2025-08-14T21:53:55.9046586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9046662Z outputs = self.mobilebert( 2025-08-14T21:53:55.9046959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9047035Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9047331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9047414Z layer_outputs = layer_module( 2025-08-14T21:53:55.9047707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9047821Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9048113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9048246Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9048590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9048709Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9048982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9049084Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9049087Z 2025-08-14T21:53:55.9049186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9049399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9049463Z return mod(**inputs) 2025-08-14T21:53:55.9049730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9049807Z outputs = self.mobilebert( 2025-08-14T21:53:55.9050082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9050161Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9050483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9050551Z layer_outputs = layer_module( 2025-08-14T21:53:55.9050824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9050917Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9051192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9051300Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9051569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9051659Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9051663Z 2025-08-14T21:53:55.9051759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9051954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9052018Z return mod(**inputs) 2025-08-14T21:53:55.9052287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9052365Z outputs = self.mobilebert( 2025-08-14T21:53:55.9052637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9052706Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9052979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9053046Z layer_outputs = layer_module( 2025-08-14T21:53:55.9053327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9053420Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9053700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9053818Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9054114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9054233Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9054236Z 2025-08-14T21:53:55.9054338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9054551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9054624Z return mod(**inputs) 2025-08-14T21:53:55.9054900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9054971Z outputs = self.mobilebert( 2025-08-14T21:53:55.9055277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9055347Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9055639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9055711Z layer_outputs = layer_module( 2025-08-14T21:53:55.9055984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9056085Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9056356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9056490Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9056762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9056843Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9056846Z 2025-08-14T21:53:55.9056957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9057157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9057225Z return mod(**inputs) 2025-08-14T21:53:55.9057524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9057600Z outputs = self.mobilebert( 2025-08-14T21:53:55.9057905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9057980Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9058267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9058356Z layer_outputs = layer_module( 2025-08-14T21:53:55.9058627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9058727Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9058999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9059121Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9059398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9059515Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9059791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9059893Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9059896Z 2025-08-14T21:53:55.9059997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9060217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9060283Z return mod(**inputs) 2025-08-14T21:53:55.9060573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9060683Z outputs = self.mobilebert( 2025-08-14T21:53:55.9060985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9061068Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9061370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9061445Z layer_outputs = layer_module( 2025-08-14T21:53:55.9061765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9061863Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9062163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9062290Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9062593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9062687Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9062690Z 2025-08-14T21:53:55.9062797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9063006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9063080Z return mod(**inputs) 2025-08-14T21:53:55.9063358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9063435Z outputs = self.mobilebert( 2025-08-14T21:53:55.9063708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9063779Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9064063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9064138Z layer_outputs = layer_module( 2025-08-14T21:53:55.9064439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9064544Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9064844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9064966Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9065261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9065377Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9065380Z 2025-08-14T21:53:55.9065496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9065702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9065776Z return mod(**inputs) 2025-08-14T21:53:55.9066068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9066142Z outputs = self.mobilebert( 2025-08-14T21:53:55.9066443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9066743Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9067798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9067927Z layer_outputs = layer_module( 2025-08-14T21:53:55.9068252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9068394Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9068754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9068944Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9069282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9069414Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9069418Z 2025-08-14T21:53:55.9088901Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9089391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9089487Z return mod(**inputs) 2025-08-14T21:53:55.9089835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9089936Z outputs = self.mobilebert( 2025-08-14T21:53:55.9090253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9090342Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9090660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9090739Z layer_outputs = layer_module( 2025-08-14T21:53:55.9091018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9091131Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9091414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9091556Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9091832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9091955Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9092242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9092341Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9092347Z 2025-08-14T21:53:55.9092467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9092678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9092748Z return mod(**inputs) 2025-08-14T21:53:55.9093040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9093120Z outputs = self.mobilebert( 2025-08-14T21:53:55.9093407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9093486Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9093773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9093856Z layer_outputs = layer_module( 2025-08-14T21:53:55.9094153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9094288Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9094597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9094721Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9095036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9095129Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9095134Z 2025-08-14T21:53:55.9095245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9095468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9095537Z return mod(**inputs) 2025-08-14T21:53:55.9095860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9095942Z outputs = self.mobilebert( 2025-08-14T21:53:55.9096251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9096336Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9096618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9096688Z layer_outputs = layer_module( 2025-08-14T21:53:55.9096981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9097077Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9097361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9097489Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9097791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9097920Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9097924Z 2025-08-14T21:53:55.9098037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9098259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9098337Z return mod(**inputs) 2025-08-14T21:53:55.9098636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9098717Z outputs = self.mobilebert( 2025-08-14T21:53:55.9099016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9099094Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9099402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9099475Z layer_outputs = layer_module( 2025-08-14T21:53:55.9099756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9099859Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9100137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9100270Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9100548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9100636Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9100640Z 2025-08-14T21:53:55.9100769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9100967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9101043Z return mod(**inputs) 2025-08-14T21:53:55.9101335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9101429Z outputs = self.mobilebert( 2025-08-14T21:53:55.9101727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9101804Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9102094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9102190Z layer_outputs = layer_module( 2025-08-14T21:53:55.9102476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9102595Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9102887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9103017Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9103320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9103445Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9103742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9103842Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9103846Z 2025-08-14T21:53:55.9103955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9104171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9104241Z return mod(**inputs) 2025-08-14T21:53:55.9104532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9104616Z outputs = self.mobilebert( 2025-08-14T21:53:55.9104903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9104988Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9105274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9105348Z layer_outputs = layer_module( 2025-08-14T21:53:55.9105647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9105780Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9106081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9106171Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9106176Z 2025-08-14T21:53:55.9106282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9106500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9106569Z return mod(**inputs) 2025-08-14T21:53:55.9106861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9106943Z outputs = self.mobilebert( 2025-08-14T21:53:55.9107232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9107330Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9107630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9107704Z layer_outputs = layer_module( 2025-08-14T21:53:55.9108011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9108152Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9108446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9108566Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9108570Z 2025-08-14T21:53:55.9108936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9110778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9111060Z return mod(**inputs) 2025-08-14T21:53:55.9111762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9111855Z outputs = self.mobilebert( 2025-08-14T21:53:55.9112175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9112284Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9112597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9112680Z layer_outputs = layer_module( 2025-08-14T21:53:55.9113013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9113195Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9113517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9113627Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9113636Z 2025-08-14T21:53:55.9113762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9114008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9114084Z return mod(**inputs) 2025-08-14T21:53:55.9114406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9114488Z outputs = self.mobilebert( 2025-08-14T21:53:55.9114802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9114891Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9115201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9115278Z layer_outputs = layer_module( 2025-08-14T21:53:55.9115599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9116013Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9116340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9116480Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9116794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9116910Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9116966Z 2025-08-14T21:53:55.9117087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9117333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9117411Z return mod(**inputs) 2025-08-14T21:53:55.9117718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9117839Z outputs = self.mobilebert( 2025-08-14T21:53:55.9118155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9118235Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9118557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9118689Z layer_outputs = layer_module( 2025-08-14T21:53:55.9119004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9119202Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9119508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9119649Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9120013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9120114Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9120119Z 2025-08-14T21:53:55.9120230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9120453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9120538Z return mod(**inputs) 2025-08-14T21:53:55.9120813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9120883Z outputs = self.mobilebert( 2025-08-14T21:53:55.9121156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9121231Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9121514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9121589Z layer_outputs = layer_module( 2025-08-14T21:53:55.9121859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9122020Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9122288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9122413Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9122679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9122794Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9123072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9123181Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9123185Z 2025-08-14T21:53:55.9123291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9123483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9123550Z return mod(**inputs) 2025-08-14T21:53:55.9123827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9123924Z outputs = self.mobilebert( 2025-08-14T21:53:55.9124196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9124292Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9124565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9124651Z layer_outputs = layer_module( 2025-08-14T21:53:55.9124929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9125107Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9125375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9125508Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9125789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9125868Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9125874Z 2025-08-14T21:53:55.9125971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9126168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9126237Z return mod(**inputs) 2025-08-14T21:53:55.9126523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9126592Z outputs = self.mobilebert( 2025-08-14T21:53:55.9126871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9126949Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9127214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9127289Z layer_outputs = layer_module( 2025-08-14T21:53:55.9127553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9127701Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9127974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9128078Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9128345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9128442Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9128717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9128813Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9128818Z 2025-08-14T21:53:55.9128916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9129108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9129178Z return mod(**inputs) 2025-08-14T21:53:55.9129449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9129522Z outputs = self.mobilebert( 2025-08-14T21:53:55.9129794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9129885Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9130161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9130229Z layer_outputs = layer_module( 2025-08-14T21:53:55.9130496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9130608Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9130881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9130960Z self_outputs = self.self( 2025-08-14T21:53:55.9131242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9131313Z self.query(query_tensor) 2025-08-14T21:53:55.9131317Z 2025-08-14T21:53:55.9131425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9131648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9131718Z return mod(**inputs) 2025-08-14T21:53:55.9131983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9132053Z outputs = self.mobilebert( 2025-08-14T21:53:55.9132333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9132402Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9132666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9132750Z layer_outputs = layer_module( 2025-08-14T21:53:55.9133028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9133121Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9133400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9133468Z self_outputs = self.self( 2025-08-14T21:53:55.9133761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9133828Z self.key(key_tensor) 2025-08-14T21:53:55.9133832Z 2025-08-14T21:53:55.9133940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9134136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9134200Z return mod(**inputs) 2025-08-14T21:53:55.9134489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9134563Z outputs = self.mobilebert( 2025-08-14T21:53:55.9134843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9134926Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9135242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9135321Z layer_outputs = layer_module( 2025-08-14T21:53:55.9135590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9135671Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9135950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9136018Z self_outputs = self.self( 2025-08-14T21:53:55.9136310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9136388Z self.value(value_tensor) 2025-08-14T21:53:55.9136393Z 2025-08-14T21:53:55.9136473Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9136564Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9136684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9136880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9136952Z return mod(**inputs) 2025-08-14T21:53:55.9137241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9137318Z outputs = self.mobilebert( 2025-08-14T21:53:55.9137609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9137683Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9137977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9138046Z layer_outputs = layer_module( 2025-08-14T21:53:55.9138309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9138401Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9138664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9138793Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9139060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9139143Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9139147Z 2025-08-14T21:53:55.9139253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9139441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9139513Z return mod(**inputs) 2025-08-14T21:53:55.9139783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9139851Z outputs = self.mobilebert( 2025-08-14T21:53:55.9140122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9140196Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9140466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9140544Z layer_outputs = layer_module( 2025-08-14T21:53:55.9140809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9140972Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9141240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9141351Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9141626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9141708Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9141712Z 2025-08-14T21:53:55.9141817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9142008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9142072Z return mod(**inputs) 2025-08-14T21:53:55.9142382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9142451Z outputs = self.mobilebert( 2025-08-14T21:53:55.9142724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9142820Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9143085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9143161Z layer_outputs = layer_module( 2025-08-14T21:53:55.9143429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9143531Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9143818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9143958Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9144244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9144369Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9144655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9144757Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9144762Z 2025-08-14T21:53:55.9144863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9145069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9145137Z return mod(**inputs) 2025-08-14T21:53:55.9145417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9145500Z outputs = self.mobilebert( 2025-08-14T21:53:55.9145779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9145853Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9146141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9146215Z layer_outputs = layer_module( 2025-08-14T21:53:55.9146504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9146598Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9146869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9146987Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9147265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9147353Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9147360Z 2025-08-14T21:53:55.9147460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9147655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9147725Z return mod(**inputs) 2025-08-14T21:53:55.9148016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9148089Z outputs = self.mobilebert( 2025-08-14T21:53:55.9148390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9148493Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9148781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9148857Z layer_outputs = layer_module( 2025-08-14T21:53:55.9149137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9149257Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9149534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9149653Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9149943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9150057Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9150063Z 2025-08-14T21:53:55.9150177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9150404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9150475Z return mod(**inputs) 2025-08-14T21:53:55.9150776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9150851Z outputs = self.mobilebert( 2025-08-14T21:53:55.9151150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9151224Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9151519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9151601Z layer_outputs = layer_module( 2025-08-14T21:53:55.9151890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9151997Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9152293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9152432Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9152730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9152818Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9152822Z 2025-08-14T21:53:55.9152925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9153141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9153209Z return mod(**inputs) 2025-08-14T21:53:55.9153512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9153588Z outputs = self.mobilebert( 2025-08-14T21:53:55.9153877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9153962Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9154249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9154331Z layer_outputs = layer_module( 2025-08-14T21:53:55.9154619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9154720Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9155013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9155163Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9155448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9155583Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9156062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9156179Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9156185Z 2025-08-14T21:53:55.9156294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9156503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9156620Z return mod(**inputs) 2025-08-14T21:53:55.9156913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9157019Z outputs = self.mobilebert( 2025-08-14T21:53:55.9157310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9157390Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9157694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9157769Z layer_outputs = layer_module( 2025-08-14T21:53:55.9158054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9158160Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9158453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9158577Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9158873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9158962Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9158968Z 2025-08-14T21:53:55.9159082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9159294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9159371Z return mod(**inputs) 2025-08-14T21:53:55.9159666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9159740Z outputs = self.mobilebert( 2025-08-14T21:53:55.9160038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9160117Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9160408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9160489Z layer_outputs = layer_module( 2025-08-14T21:53:55.9160779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9160886Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9161179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9161296Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9161597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9161718Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9161743Z 2025-08-14T21:53:55.9161860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9162064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9162134Z return mod(**inputs) 2025-08-14T21:53:55.9162429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9162539Z outputs = self.mobilebert( 2025-08-14T21:53:55.9162832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9162915Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9163233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9163311Z layer_outputs = layer_module( 2025-08-14T21:53:55.9163580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9163688Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9163963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9164087Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9164359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9164440Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9164444Z 2025-08-14T21:53:55.9164541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9164740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9164805Z return mod(**inputs) 2025-08-14T21:53:55.9165089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9165157Z outputs = self.mobilebert( 2025-08-14T21:53:55.9165422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9165501Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9165764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9165837Z layer_outputs = layer_module( 2025-08-14T21:53:55.9166116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9166205Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9166479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9166605Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9166871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9166994Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9167262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9167359Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9167362Z 2025-08-14T21:53:55.9167465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9167657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9167734Z return mod(**inputs) 2025-08-14T21:53:55.9168012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9168112Z outputs = self.mobilebert( 2025-08-14T21:53:55.9168406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9168476Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9168766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9168834Z layer_outputs = layer_module( 2025-08-14T21:53:55.9169101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9169200Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9169479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9169596Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9169882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9169967Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9169973Z 2025-08-14T21:53:55.9170081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9170285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9170353Z return mod(**inputs) 2025-08-14T21:53:55.9170633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9170700Z outputs = self.mobilebert( 2025-08-14T21:53:55.9170971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9171044Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9171312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9171389Z layer_outputs = layer_module( 2025-08-14T21:53:55.9171652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9171752Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9172015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9172122Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9172394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9172503Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9172508Z 2025-08-14T21:53:55.9172613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9172807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9172873Z return mod(**inputs) 2025-08-14T21:53:55.9173150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9173222Z outputs = self.mobilebert( 2025-08-14T21:53:55.9173496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9173578Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9173851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9173932Z layer_outputs = layer_module( 2025-08-14T21:53:55.9174207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9174322Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9174603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9174738Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9175004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9175094Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9175097Z 2025-08-14T21:53:55.9175195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9175402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9175469Z return mod(**inputs) 2025-08-14T21:53:55.9175762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9175837Z outputs = self.mobilebert( 2025-08-14T21:53:55.9176109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9176196Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9176470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9176545Z layer_outputs = layer_module( 2025-08-14T21:53:55.9176832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9176920Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9177181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9177306Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9177564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9177683Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9177944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9178030Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9178033Z 2025-08-14T21:53:55.9178140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9178328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9178399Z return mod(**inputs) 2025-08-14T21:53:55.9178676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9178746Z outputs = self.mobilebert( 2025-08-14T21:53:55.9179009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9179076Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9179343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9179408Z layer_outputs = layer_module( 2025-08-14T21:53:55.9179664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9179782Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9180041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9180142Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9180145Z 2025-08-14T21:53:55.9180249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9180435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9180505Z return mod(**inputs) 2025-08-14T21:53:55.9180789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9180855Z outputs = self.mobilebert( 2025-08-14T21:53:55.9181124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9181194Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9181521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9181595Z layer_outputs = layer_module( 2025-08-14T21:53:55.9181884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9182006Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9182275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9182384Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9182395Z 2025-08-14T21:53:55.9182494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9182684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9182754Z return mod(**inputs) 2025-08-14T21:53:55.9183026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9183094Z outputs = self.mobilebert( 2025-08-14T21:53:55.9183374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9183447Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9183722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9183796Z layer_outputs = layer_module( 2025-08-14T21:53:55.9184064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9184226Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9184505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9184596Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9184609Z 2025-08-14T21:53:55.9184706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9184895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9184966Z return mod(**inputs) 2025-08-14T21:53:55.9185236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9185305Z outputs = self.mobilebert( 2025-08-14T21:53:55.9185578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9185648Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9185919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9185990Z layer_outputs = layer_module( 2025-08-14T21:53:55.9186257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9186451Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9186717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9186853Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9187131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9187223Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9187227Z 2025-08-14T21:53:55.9187332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9187539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9187606Z return mod(**inputs) 2025-08-14T21:53:55.9187907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9187979Z outputs = self.mobilebert( 2025-08-14T21:53:55.9188266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9188340Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9188620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9188696Z layer_outputs = layer_module( 2025-08-14T21:53:55.9188976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9189131Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9189429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9189554Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9189844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9189929Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9189932Z 2025-08-14T21:53:55.9190033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9190238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9190302Z return mod(**inputs) 2025-08-14T21:53:55.9190591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9190664Z outputs = self.mobilebert( 2025-08-14T21:53:55.9190944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9191029Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9191314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9191394Z layer_outputs = layer_module( 2025-08-14T21:53:55.9191671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9191823Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9192111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9192232Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9192511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9192666Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9192940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9193056Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9193060Z 2025-08-14T21:53:55.9193164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9193379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9193456Z return mod(**inputs) 2025-08-14T21:53:55.9193749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9193845Z outputs = self.mobilebert( 2025-08-14T21:53:55.9194148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9194245Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9194547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9194620Z layer_outputs = layer_module( 2025-08-14T21:53:55.9194921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9195095Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9195392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9195518Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9195923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9196021Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9196028Z 2025-08-14T21:53:55.9196145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9196367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9196450Z return mod(**inputs) 2025-08-14T21:53:55.9196753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9196831Z outputs = self.mobilebert( 2025-08-14T21:53:55.9197142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9197218Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9197530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9197609Z layer_outputs = layer_module( 2025-08-14T21:53:55.9197904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9198079Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9198382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9198497Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9198803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9198891Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9199193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9199318Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9199323Z 2025-08-14T21:53:55.9199433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9199651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9199717Z return mod(**inputs) 2025-08-14T21:53:55.9200020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9200091Z outputs = self.mobilebert( 2025-08-14T21:53:55.9200369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9200448Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9200745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9200820Z layer_outputs = layer_module( 2025-08-14T21:53:55.9201140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9201232Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9201533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9201611Z self_outputs = self.self( 2025-08-14T21:53:55.9201902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9201985Z self.query(query_tensor) 2025-08-14T21:53:55.9201990Z 2025-08-14T21:53:55.9202095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9202309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9202378Z return mod(**inputs) 2025-08-14T21:53:55.9202674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9202755Z outputs = self.mobilebert( 2025-08-14T21:53:55.9203046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9203123Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9203428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9203503Z layer_outputs = layer_module( 2025-08-14T21:53:55.9203807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9203895Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9204202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9204287Z self_outputs = self.self( 2025-08-14T21:53:55.9204580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9204658Z self.key(key_tensor) 2025-08-14T21:53:55.9204661Z 2025-08-14T21:53:55.9204769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9204974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9205051Z return mod(**inputs) 2025-08-14T21:53:55.9205345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9205420Z outputs = self.mobilebert( 2025-08-14T21:53:55.9205732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9205834Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9206134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9206207Z layer_outputs = layer_module( 2025-08-14T21:53:55.9206497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9206613Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9206901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9206974Z self_outputs = self.self( 2025-08-14T21:53:55.9207281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9207372Z self.value(value_tensor) 2025-08-14T21:53:55.9207376Z 2025-08-14T21:53:55.9207470Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9207554Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9207682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9207892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9207961Z return mod(**inputs) 2025-08-14T21:53:55.9208265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9208338Z outputs = self.mobilebert( 2025-08-14T21:53:55.9208913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9209044Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9209463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9209540Z layer_outputs = layer_module( 2025-08-14T21:53:55.9209844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9209937Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9210239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9210373Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9210661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9210761Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9210765Z 2025-08-14T21:53:55.9210870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9211087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9211159Z return mod(**inputs) 2025-08-14T21:53:55.9211452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9211532Z outputs = self.mobilebert( 2025-08-14T21:53:55.9211858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9211935Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9212232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9212305Z layer_outputs = layer_module( 2025-08-14T21:53:55.9212596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9212763Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9213124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9213247Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9213542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9213654Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9213658Z 2025-08-14T21:53:55.9213755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9213942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9214012Z return mod(**inputs) 2025-08-14T21:53:55.9214310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9214380Z outputs = self.mobilebert( 2025-08-14T21:53:55.9214654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9214748Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9215028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9215096Z layer_outputs = layer_module( 2025-08-14T21:53:55.9215352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9215437Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9215697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9215819Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9216079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9216204Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9216476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9216568Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9216573Z 2025-08-14T21:53:55.9216681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9216873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9216942Z return mod(**inputs) 2025-08-14T21:53:55.9217228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9217297Z outputs = self.mobilebert( 2025-08-14T21:53:55.9217574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9217655Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9217928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9218005Z layer_outputs = layer_module( 2025-08-14T21:53:55.9218277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9218372Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9218651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9218760Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9219032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9219152Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9219156Z 2025-08-14T21:53:55.9219254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9219448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9219511Z return mod(**inputs) 2025-08-14T21:53:55.9219799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9219875Z outputs = self.mobilebert( 2025-08-14T21:53:55.9220144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9220220Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9220516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9220588Z layer_outputs = layer_module( 2025-08-14T21:53:55.9220884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9220981Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9221264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9221382Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9221650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9221763Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9221767Z 2025-08-14T21:53:55.9221866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9222061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9222135Z return mod(**inputs) 2025-08-14T21:53:55.9222405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9222482Z outputs = self.mobilebert( 2025-08-14T21:53:55.9222750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9222826Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9223107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9223178Z layer_outputs = layer_module( 2025-08-14T21:53:55.9223452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9223556Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9223830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9223964Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9224241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9224327Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9224331Z 2025-08-14T21:53:55.9224462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9224668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9224777Z return mod(**inputs) 2025-08-14T21:53:55.9225049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9225119Z outputs = self.mobilebert( 2025-08-14T21:53:55.9225393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9225487Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9225763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9225847Z layer_outputs = layer_module( 2025-08-14T21:53:55.9226112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9226212Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9226475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9226597Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9226890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9227029Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9227312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9227404Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9227408Z 2025-08-14T21:53:55.9227510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9227721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9227786Z return mod(**inputs) 2025-08-14T21:53:55.9228062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9228130Z outputs = self.mobilebert( 2025-08-14T21:53:55.9228403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9228484Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9228759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9228827Z layer_outputs = layer_module( 2025-08-14T21:53:55.9229109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9229201Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9229479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9229591Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9229870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9229962Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9229966Z 2025-08-14T21:53:55.9230067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9230276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9230344Z return mod(**inputs) 2025-08-14T21:53:55.9230640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9230717Z outputs = self.mobilebert( 2025-08-14T21:53:55.9231005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9231078Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9231382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9231453Z layer_outputs = layer_module( 2025-08-14T21:53:55.9231758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9231850Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9232182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9232326Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9232601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9232722Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9232726Z 2025-08-14T21:53:55.9232832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9233059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9233138Z return mod(**inputs) 2025-08-14T21:53:55.9233453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9233530Z outputs = self.mobilebert( 2025-08-14T21:53:55.9233830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9233907Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9234207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9234281Z layer_outputs = layer_module( 2025-08-14T21:53:55.9234574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9234683Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9234975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9235113Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9235404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9235495Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9235499Z 2025-08-14T21:53:55.9235613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9235928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9236014Z return mod(**inputs) 2025-08-14T21:53:55.9236318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9236398Z outputs = self.mobilebert( 2025-08-14T21:53:55.9236701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9236781Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9237075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9237160Z layer_outputs = layer_module( 2025-08-14T21:53:55.9237447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9237550Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9237827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9237951Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9238235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9238391Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9238675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9238766Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9238784Z 2025-08-14T21:53:55.9238888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9239098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9239164Z return mod(**inputs) 2025-08-14T21:53:55.9239455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9239548Z outputs = self.mobilebert( 2025-08-14T21:53:55.9239835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9239920Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9240208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9240279Z layer_outputs = layer_module( 2025-08-14T21:53:55.9240559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9240650Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9240954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9241098Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9241493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9241639Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9241645Z 2025-08-14T21:53:55.9241803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9242067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9242168Z return mod(**inputs) 2025-08-14T21:53:55.9242641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9242763Z outputs = self.mobilebert( 2025-08-14T21:53:55.9243155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9243230Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9243513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9243587Z layer_outputs = layer_module( 2025-08-14T21:53:55.9243870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9243965Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9244250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9244377Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9244666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9244786Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9244796Z 2025-08-14T21:53:55.9244904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9245109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9245215Z return mod(**inputs) 2025-08-14T21:53:55.9245515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9245590Z outputs = self.mobilebert( 2025-08-14T21:53:55.9245893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9245996Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9246282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9246353Z layer_outputs = layer_module( 2025-08-14T21:53:55.9246630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9246748Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9247023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9247169Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9247467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9247557Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9247560Z 2025-08-14T21:53:55.9247675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9247880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9247950Z return mod(**inputs) 2025-08-14T21:53:55.9248248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9248323Z outputs = self.mobilebert( 2025-08-14T21:53:55.9248621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9248700Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9248990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9249072Z layer_outputs = layer_module( 2025-08-14T21:53:55.9249358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9249453Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9249817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9249957Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9250311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9250478Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9250887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9251028Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9251036Z 2025-08-14T21:53:55.9251187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9251516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9251595Z return mod(**inputs) 2025-08-14T21:53:55.9251897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9251978Z outputs = self.mobilebert( 2025-08-14T21:53:55.9252265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9252370Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9252657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9252731Z layer_outputs = layer_module( 2025-08-14T21:53:55.9253044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9253172Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9253457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9253554Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9253558Z 2025-08-14T21:53:55.9253682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9253897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9253967Z return mod(**inputs) 2025-08-14T21:53:55.9254277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9254360Z outputs = self.mobilebert( 2025-08-14T21:53:55.9254653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9254738Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9255032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9255109Z layer_outputs = layer_module( 2025-08-14T21:53:55.9255411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9255536Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9255830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9255956Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9255960Z 2025-08-14T21:53:55.9256067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9256283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9256353Z return mod(**inputs) 2025-08-14T21:53:55.9256648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9256728Z outputs = self.mobilebert( 2025-08-14T21:53:55.9257023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9257107Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9257403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9257487Z layer_outputs = layer_module( 2025-08-14T21:53:55.9257788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9257955Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9258248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9258359Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9258362Z 2025-08-14T21:53:55.9258468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9258684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9258752Z return mod(**inputs) 2025-08-14T21:53:55.9259074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9259149Z outputs = self.mobilebert( 2025-08-14T21:53:55.9259430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9259532Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9259807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9259878Z layer_outputs = layer_module( 2025-08-14T21:53:55.9260159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9260337Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9260601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9260741Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9261007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9261106Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9261112Z 2025-08-14T21:53:55.9261210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9261395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9261464Z return mod(**inputs) 2025-08-14T21:53:55.9261731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9261807Z outputs = self.mobilebert( 2025-08-14T21:53:55.9262074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9262144Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9262415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9262485Z layer_outputs = layer_module( 2025-08-14T21:53:55.9262747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9262901Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9263165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9263291Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9263556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9263643Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9263647Z 2025-08-14T21:53:55.9263752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9263940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9264013Z return mod(**inputs) 2025-08-14T21:53:55.9264281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9264348Z outputs = self.mobilebert( 2025-08-14T21:53:55.9264619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9264690Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9264960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9265044Z layer_outputs = layer_module( 2025-08-14T21:53:55.9265310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9265464Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9265744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9265860Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9266129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9266256Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9266535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9266643Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9266647Z 2025-08-14T21:53:55.9266745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9266939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9267005Z return mod(**inputs) 2025-08-14T21:53:55.9267283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9267351Z outputs = self.mobilebert( 2025-08-14T21:53:55.9267617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9267694Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9267962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9268031Z layer_outputs = layer_module( 2025-08-14T21:53:55.9268309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9268465Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9268748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9268857Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9269136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9269227Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9269230Z 2025-08-14T21:53:55.9269332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9269543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9269616Z return mod(**inputs) 2025-08-14T21:53:55.9269910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9269991Z outputs = self.mobilebert( 2025-08-14T21:53:55.9270295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9270376Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9270676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9270750Z layer_outputs = layer_module( 2025-08-14T21:53:55.9271055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9271221Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9271535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9271655Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9271960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9272056Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9272357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9272454Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9272458Z 2025-08-14T21:53:55.9272588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9272793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9272870Z return mod(**inputs) 2025-08-14T21:53:55.9273178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9273254Z outputs = self.mobilebert( 2025-08-14T21:53:55.9273551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9273630Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9273921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9274002Z layer_outputs = layer_module( 2025-08-14T21:53:55.9274295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9274394Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9274684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9274759Z self_outputs = self.self( 2025-08-14T21:53:55.9275059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9275135Z self.query(query_tensor) 2025-08-14T21:53:55.9275139Z 2025-08-14T21:53:55.9275252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9275454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9275524Z return mod(**inputs) 2025-08-14T21:53:55.9275928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9276012Z outputs = self.mobilebert( 2025-08-14T21:53:55.9276298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9276390Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9276687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9276774Z layer_outputs = layer_module( 2025-08-14T21:53:55.9277074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9277160Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9277445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9277517Z self_outputs = self.self( 2025-08-14T21:53:55.9277817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9277910Z self.key(key_tensor) 2025-08-14T21:53:55.9277914Z 2025-08-14T21:53:55.9278026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9278244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9278313Z return mod(**inputs) 2025-08-14T21:53:55.9278622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9278705Z outputs = self.mobilebert( 2025-08-14T21:53:55.9278993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9279076Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9279381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9279456Z layer_outputs = layer_module( 2025-08-14T21:53:55.9279771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9279860Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9280147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9280229Z self_outputs = self.self( 2025-08-14T21:53:55.9280519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9280601Z self.value(value_tensor) 2025-08-14T21:53:55.9280605Z 2025-08-14T21:53:55.9280691Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9280772Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9280886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9281092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9281169Z return mod(**inputs) 2025-08-14T21:53:55.9281463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9281535Z outputs = self.mobilebert( 2025-08-14T21:53:55.9281827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9281904Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9282189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9282268Z layer_outputs = layer_module( 2025-08-14T21:53:55.9282555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9282649Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9282938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9283069Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9283367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9283458Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9283462Z 2025-08-14T21:53:55.9283575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9283779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9283847Z return mod(**inputs) 2025-08-14T21:53:55.9284146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9284221Z outputs = self.mobilebert( 2025-08-14T21:53:55.9284539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9284623Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9284912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9285037Z layer_outputs = layer_module( 2025-08-14T21:53:55.9285333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9285499Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9285804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9285935Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9286237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9286345Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9286349Z 2025-08-14T21:53:55.9286457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9286671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9286742Z return mod(**inputs) 2025-08-14T21:53:55.9287034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9287115Z outputs = self.mobilebert( 2025-08-14T21:53:55.9287405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9287499Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9287773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9287846Z layer_outputs = layer_module( 2025-08-14T21:53:55.9288127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9288210Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9288492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9288614Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9288913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9289050Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9289353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9289451Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9289465Z 2025-08-14T21:53:55.9289572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9289776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9289852Z return mod(**inputs) 2025-08-14T21:53:55.9290145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9290213Z outputs = self.mobilebert( 2025-08-14T21:53:55.9290493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9290564Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9290844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9290933Z layer_outputs = layer_module( 2025-08-14T21:53:55.9291208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9291310Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9291586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9291717Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9291997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9292078Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9292082Z 2025-08-14T21:53:55.9292202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9292396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9292462Z return mod(**inputs) 2025-08-14T21:53:55.9292759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9292830Z outputs = self.mobilebert( 2025-08-14T21:53:55.9293121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9293199Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9293490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9293573Z layer_outputs = layer_module( 2025-08-14T21:53:55.9293876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9293974Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9294275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9294391Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9294689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9294809Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9294812Z 2025-08-14T21:53:55.9294918Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9295142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9295212Z return mod(**inputs) 2025-08-14T21:53:55.9295515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9295592Z outputs = self.mobilebert( 2025-08-14T21:53:55.9295894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9295977Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9296280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9296354Z layer_outputs = layer_module( 2025-08-14T21:53:55.9296662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9296761Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9297065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9297196Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9297497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9297621Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9297627Z 2025-08-14T21:53:55.9297733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9297955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9298040Z return mod(**inputs) 2025-08-14T21:53:55.9298337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9298417Z outputs = self.mobilebert( 2025-08-14T21:53:55.9298725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9298820Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9299127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9299202Z layer_outputs = layer_module( 2025-08-14T21:53:55.9299520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9299619Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9299918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9300055Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9300353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9300485Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9300785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9300882Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9300887Z 2025-08-14T21:53:55.9301000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9301215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9301293Z return mod(**inputs) 2025-08-14T21:53:55.9301584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9301658Z outputs = self.mobilebert( 2025-08-14T21:53:55.9301959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9302033Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9302380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9302462Z layer_outputs = layer_module( 2025-08-14T21:53:55.9302763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9302867Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9303170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9303287Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9303584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9303673Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9303677Z 2025-08-14T21:53:55.9303791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9303997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9304083Z return mod(**inputs) 2025-08-14T21:53:55.9304380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9304453Z outputs = self.mobilebert( 2025-08-14T21:53:55.9304749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9304850Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9305152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9305233Z layer_outputs = layer_module( 2025-08-14T21:53:55.9305558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9305658Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9305992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9306113Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9306415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9306540Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9306544Z 2025-08-14T21:53:55.9306652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9306881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9306949Z return mod(**inputs) 2025-08-14T21:53:55.9307249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9307332Z outputs = self.mobilebert( 2025-08-14T21:53:55.9307622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9307705Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9307994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9308070Z layer_outputs = layer_module( 2025-08-14T21:53:55.9308366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9308464Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9309019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9309222Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9309520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9309619Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9309626Z 2025-08-14T21:53:55.9309731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9309958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9310031Z return mod(**inputs) 2025-08-14T21:53:55.9310336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9310419Z outputs = self.mobilebert( 2025-08-14T21:53:55.9310733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9310813Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9311132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9311325Z layer_outputs = layer_module( 2025-08-14T21:53:55.9311642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9311742Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9312073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9312215Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9312532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9312670Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9313009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9313113Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9313140Z 2025-08-14T21:53:55.9313261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9313475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9313549Z return mod(**inputs) 2025-08-14T21:53:55.9313856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9313933Z outputs = self.mobilebert( 2025-08-14T21:53:55.9314247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9314325Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9314636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9314720Z layer_outputs = layer_module( 2025-08-14T21:53:55.9315017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9315123Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9315419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9315539Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9315896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9316067Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9316073Z 2025-08-14T21:53:55.9316189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9316406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9316478Z return mod(**inputs) 2025-08-14T21:53:55.9316786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9316862Z outputs = self.mobilebert( 2025-08-14T21:53:55.9317158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9317258Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9317545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9317624Z layer_outputs = layer_module( 2025-08-14T21:53:55.9317914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9318010Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9318333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9318449Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9318739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9318884Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9318888Z 2025-08-14T21:53:55.9318993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9319208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9319277Z return mod(**inputs) 2025-08-14T21:53:55.9319592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9319675Z outputs = self.mobilebert( 2025-08-14T21:53:55.9319983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9320065Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9320352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9320428Z layer_outputs = layer_module( 2025-08-14T21:53:55.9320723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9320821Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9321111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9321249Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9321538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9321636Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9321640Z 2025-08-14T21:53:55.9321745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9321950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9322028Z return mod(**inputs) 2025-08-14T21:53:55.9322320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9322400Z outputs = self.mobilebert( 2025-08-14T21:53:55.9322687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9322764Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9323066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9323142Z layer_outputs = layer_module( 2025-08-14T21:53:55.9323432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9323538Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9323829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9323964Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9324253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9324379Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9324677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9324792Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9324796Z 2025-08-14T21:53:55.9324909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9325117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9325204Z return mod(**inputs) 2025-08-14T21:53:55.9325511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9325592Z outputs = self.mobilebert( 2025-08-14T21:53:55.9325890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9325971Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9326281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9326362Z layer_outputs = layer_module( 2025-08-14T21:53:55.9326675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9326796Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9327077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9327159Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9327163Z 2025-08-14T21:53:55.9327270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9327469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9327538Z return mod(**inputs) 2025-08-14T21:53:55.9327839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9327915Z outputs = self.mobilebert( 2025-08-14T21:53:55.9328209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9328288Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9328560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9328637Z layer_outputs = layer_module( 2025-08-14T21:53:55.9328909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9329025Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9329334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9329448Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9329453Z 2025-08-14T21:53:55.9329566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9329771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9329839Z return mod(**inputs) 2025-08-14T21:53:55.9330139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9330214Z outputs = self.mobilebert( 2025-08-14T21:53:55.9330513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9330588Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9330890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9330973Z layer_outputs = layer_module( 2025-08-14T21:53:55.9331265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9331452Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9331748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9331863Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9331866Z 2025-08-14T21:53:55.9331980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9332184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9332251Z return mod(**inputs) 2025-08-14T21:53:55.9332574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9332649Z outputs = self.mobilebert( 2025-08-14T21:53:55.9332948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9333041Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9333328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9333408Z layer_outputs = layer_module( 2025-08-14T21:53:55.9333701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9333866Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9334179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9334306Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9334597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9334698Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9334701Z 2025-08-14T21:53:55.9334807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9335022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9335091Z return mod(**inputs) 2025-08-14T21:53:55.9335389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9335461Z outputs = self.mobilebert( 2025-08-14T21:53:55.9335760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9335846Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9336132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9336209Z layer_outputs = layer_module( 2025-08-14T21:53:55.9336502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9336662Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9336954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9339588Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9339887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9339989Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9339993Z 2025-08-14T21:53:55.9340103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9340342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9340413Z return mod(**inputs) 2025-08-14T21:53:55.9340707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9340790Z outputs = self.mobilebert( 2025-08-14T21:53:55.9341079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9341158Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9341488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9341567Z layer_outputs = layer_module( 2025-08-14T21:53:55.9341858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9342013Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9343195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9343332Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9343616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9343739Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9344015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9344117Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9344122Z 2025-08-14T21:53:55.9344221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9344427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9344494Z return mod(**inputs) 2025-08-14T21:53:55.9344769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9344848Z outputs = self.mobilebert( 2025-08-14T21:53:55.9345118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9345192Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9345475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9345547Z layer_outputs = layer_module( 2025-08-14T21:53:55.9345826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9345985Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9346257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9346374Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9346645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9346735Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9346792Z 2025-08-14T21:53:55.9346893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9347094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9347169Z return mod(**inputs) 2025-08-14T21:53:55.9347465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9347558Z outputs = self.mobilebert( 2025-08-14T21:53:55.9347856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9347930Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9348226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9348299Z layer_outputs = layer_module( 2025-08-14T21:53:55.9348599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9348773Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9349092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9349215Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9349519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9349611Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9349911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9350006Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9350010Z 2025-08-14T21:53:55.9350122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9350330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9350398Z return mod(**inputs) 2025-08-14T21:53:55.9350695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9350772Z outputs = self.mobilebert( 2025-08-14T21:53:55.9351069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9351157Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9351452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9351534Z layer_outputs = layer_module( 2025-08-14T21:53:55.9351846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9351940Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9352244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9352322Z self_outputs = self.self( 2025-08-14T21:53:55.9352618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9352704Z self.query(query_tensor) 2025-08-14T21:53:55.9352708Z 2025-08-14T21:53:55.9352815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9353030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9353101Z return mod(**inputs) 2025-08-14T21:53:55.9353401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9353525Z outputs = self.mobilebert( 2025-08-14T21:53:55.9353821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9353904Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9354214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9354306Z layer_outputs = layer_module( 2025-08-14T21:53:55.9354614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9354706Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9355004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9355087Z self_outputs = self.self( 2025-08-14T21:53:55.9355395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9355474Z self.key(key_tensor) 2025-08-14T21:53:55.9355478Z 2025-08-14T21:53:55.9355585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9355926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9356013Z return mod(**inputs) 2025-08-14T21:53:55.9356334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9356420Z outputs = self.mobilebert( 2025-08-14T21:53:55.9356723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9356802Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9357106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9357183Z layer_outputs = layer_module( 2025-08-14T21:53:55.9357494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9357595Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9357885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9357969Z self_outputs = self.self( 2025-08-14T21:53:55.9358254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9358331Z self.value(value_tensor) 2025-08-14T21:53:55.9358335Z 2025-08-14T21:53:55.9358430Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9358513Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9358620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9358835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9358905Z return mod(**inputs) 2025-08-14T21:53:55.9359206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9359283Z outputs = self.mobilebert( 2025-08-14T21:53:55.9359571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9359655Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9359945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9360025Z layer_outputs = layer_module( 2025-08-14T21:53:55.9360309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9360418Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9360712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9360841Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9361158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9361254Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9361258Z 2025-08-14T21:53:55.9361364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9361573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9361641Z return mod(**inputs) 2025-08-14T21:53:55.9361933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9362017Z outputs = self.mobilebert( 2025-08-14T21:53:55.9362303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9362400Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9362697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9362786Z layer_outputs = layer_module( 2025-08-14T21:53:55.9363086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9363262Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9363536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9363654Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9363937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9364026Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9364030Z 2025-08-14T21:53:55.9364127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9364316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9364388Z return mod(**inputs) 2025-08-14T21:53:55.9364657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9364731Z outputs = self.mobilebert( 2025-08-14T21:53:55.9364996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9365067Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9365347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9365417Z layer_outputs = layer_module( 2025-08-14T21:53:55.9365690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9365782Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9366058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9366185Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9366457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9366580Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9366883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9366973Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9366976Z 2025-08-14T21:53:55.9367088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9367283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9367366Z return mod(**inputs) 2025-08-14T21:53:55.9367649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9367729Z outputs = self.mobilebert( 2025-08-14T21:53:55.9368001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9368069Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9368340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9368416Z layer_outputs = layer_module( 2025-08-14T21:53:55.9368695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9368786Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9369081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9369194Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9369472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9369556Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9369560Z 2025-08-14T21:53:55.9369660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9369861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9369924Z return mod(**inputs) 2025-08-14T21:53:55.9370207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9370277Z outputs = self.mobilebert( 2025-08-14T21:53:55.9370549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9370626Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9370907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9370974Z layer_outputs = layer_module( 2025-08-14T21:53:55.9371246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9371339Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9371611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9371717Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9371984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9372100Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9372103Z 2025-08-14T21:53:55.9372201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9372396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9372458Z return mod(**inputs) 2025-08-14T21:53:55.9372735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9372831Z outputs = self.mobilebert( 2025-08-14T21:53:55.9373110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9373184Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9373487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9373582Z layer_outputs = layer_module( 2025-08-14T21:53:55.9373876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9373975Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9374275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9374424Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9374695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9374802Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9374806Z 2025-08-14T21:53:55.9374907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9375114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9375191Z return mod(**inputs) 2025-08-14T21:53:55.9375488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9375561Z outputs = self.mobilebert( 2025-08-14T21:53:55.9375870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9375948Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9376257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9376331Z layer_outputs = layer_module( 2025-08-14T21:53:55.9376631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9376738Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9377032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9377177Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9377452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9377570Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9377875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9377971Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9377975Z 2025-08-14T21:53:55.9378089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9378294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9378367Z return mod(**inputs) 2025-08-14T21:53:55.9378664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9378737Z outputs = self.mobilebert( 2025-08-14T21:53:55.9379027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9379115Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9379405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9379484Z layer_outputs = layer_module( 2025-08-14T21:53:55.9379764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9379853Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9380144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9380257Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9380534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9380628Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9380631Z 2025-08-14T21:53:55.9380734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9380937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9381005Z return mod(**inputs) 2025-08-14T21:53:55.9381296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9381375Z outputs = self.mobilebert( 2025-08-14T21:53:55.9381660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9381740Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9382012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9382080Z layer_outputs = layer_module( 2025-08-14T21:53:55.9382359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9382452Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9382734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9382859Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9383153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9383280Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9383284Z 2025-08-14T21:53:55.9383390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9383593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9383670Z return mod(**inputs) 2025-08-14T21:53:55.9383962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9384046Z outputs = self.mobilebert( 2025-08-14T21:53:55.9384333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9384409Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9384703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9384781Z layer_outputs = layer_module( 2025-08-14T21:53:55.9385067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9385179Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9385453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9385611Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9385893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9385982Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9385986Z 2025-08-14T21:53:55.9386099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9386321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9386398Z return mod(**inputs) 2025-08-14T21:53:55.9386690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9386764Z outputs = self.mobilebert( 2025-08-14T21:53:55.9387060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9387134Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9387430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9387504Z layer_outputs = layer_module( 2025-08-14T21:53:55.9387806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9387913Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9388214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9388344Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9388634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9388758Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9389052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9389146Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9389150Z 2025-08-14T21:53:55.9389257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9389465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9389534Z return mod(**inputs) 2025-08-14T21:53:55.9389835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9389908Z outputs = self.mobilebert( 2025-08-14T21:53:55.9390195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9390278Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9390566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9390641Z layer_outputs = layer_module( 2025-08-14T21:53:55.9390935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9391032Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9391328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9391442Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9391731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9391826Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9391830Z 2025-08-14T21:53:55.9391937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9392201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9392271Z return mod(**inputs) 2025-08-14T21:53:55.9392570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9392652Z outputs = self.mobilebert( 2025-08-14T21:53:55.9392958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9393037Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9393345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9393422Z layer_outputs = layer_module( 2025-08-14T21:53:55.9393736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9393836Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9394135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9394280Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9394581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9394724Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9394728Z 2025-08-14T21:53:55.9394840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9395051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9395132Z return mod(**inputs) 2025-08-14T21:53:55.9395435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9395529Z outputs = self.mobilebert( 2025-08-14T21:53:55.9395929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9396027Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9396326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9396406Z layer_outputs = layer_module( 2025-08-14T21:53:55.9396715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9396818Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9397133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9397265Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9397559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9397656Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9397661Z 2025-08-14T21:53:55.9397768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9397983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9398054Z return mod(**inputs) 2025-08-14T21:53:55.9398346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9398429Z outputs = self.mobilebert( 2025-08-14T21:53:55.9398730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9398806Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9399129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9399204Z layer_outputs = layer_module( 2025-08-14T21:53:55.9399511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9399625Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9399925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9400063Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9400369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9400503Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9400812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9400910Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9400914Z 2025-08-14T21:53:55.9401047Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9401260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9401340Z return mod(**inputs) 2025-08-14T21:53:55.9401663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9401739Z outputs = self.mobilebert( 2025-08-14T21:53:55.9402099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9402174Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9402476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9402559Z layer_outputs = layer_module( 2025-08-14T21:53:55.9402861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9402995Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9403298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9403385Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9403389Z 2025-08-14T21:53:55.9403499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9403714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9403789Z return mod(**inputs) 2025-08-14T21:53:55.9404082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9404156Z outputs = self.mobilebert( 2025-08-14T21:53:55.9404495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9404569Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9404872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9404953Z layer_outputs = layer_module( 2025-08-14T21:53:55.9405254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9405386Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9405684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9405817Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9405820Z 2025-08-14T21:53:55.9405935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9406155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9406232Z return mod(**inputs) 2025-08-14T21:53:55.9406540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9406615Z outputs = self.mobilebert( 2025-08-14T21:53:55.9406910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9406985Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9407280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9407360Z layer_outputs = layer_module( 2025-08-14T21:53:55.9407658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9407847Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9408152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9408276Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9408280Z 2025-08-14T21:53:55.9408393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9408609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9408945Z return mod(**inputs) 2025-08-14T21:53:55.9409368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9409446Z outputs = self.mobilebert( 2025-08-14T21:53:55.9409746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9409823Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9410111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9410197Z layer_outputs = layer_module( 2025-08-14T21:53:55.9410485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9410659Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9410961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9411092Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9411391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9411490Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9411494Z 2025-08-14T21:53:55.9411608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9411817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9411886Z return mod(**inputs) 2025-08-14T21:53:55.9412185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9412260Z outputs = self.mobilebert( 2025-08-14T21:53:55.9412549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9412701Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9412991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9413071Z layer_outputs = layer_module( 2025-08-14T21:53:55.9413372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9413562Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9413856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9413975Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9414251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9414335Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9414340Z 2025-08-14T21:53:55.9414441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9414641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9414734Z return mod(**inputs) 2025-08-14T21:53:55.9415016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9415108Z outputs = self.mobilebert( 2025-08-14T21:53:55.9415374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9415453Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9415719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9415788Z layer_outputs = layer_module( 2025-08-14T21:53:55.9416062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9416209Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9416481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9416599Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9416863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9416986Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9417249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9417345Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9417350Z 2025-08-14T21:53:55.9417445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9417632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9417703Z return mod(**inputs) 2025-08-14T21:53:55.9417969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9418037Z outputs = self.mobilebert( 2025-08-14T21:53:55.9418307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9418377Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9418647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9418715Z layer_outputs = layer_module( 2025-08-14T21:53:55.9418979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9419162Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9419432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9419559Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9419824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9419904Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9419907Z 2025-08-14T21:53:55.9420010Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9420201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9420269Z return mod(**inputs) 2025-08-14T21:53:55.9420539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9420605Z outputs = self.mobilebert( 2025-08-14T21:53:55.9420893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9420966Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9421253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9421332Z layer_outputs = layer_module( 2025-08-14T21:53:55.9421601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9421764Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9422036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9422144Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9422423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9422507Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9422788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9422877Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9422881Z 2025-08-14T21:53:55.9422979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9423179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9423244Z return mod(**inputs) 2025-08-14T21:53:55.9423528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9423604Z outputs = self.mobilebert( 2025-08-14T21:53:55.9423868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9423946Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9424210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9424277Z layer_outputs = layer_module( 2025-08-14T21:53:55.9424549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9424635Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9424912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9425002Z self_outputs = self.self( 2025-08-14T21:53:55.9425276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9425354Z self.query(query_tensor) 2025-08-14T21:53:55.9425360Z 2025-08-14T21:53:55.9425461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9425668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9425741Z return mod(**inputs) 2025-08-14T21:53:55.9426019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9426095Z outputs = self.mobilebert( 2025-08-14T21:53:55.9426369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9426440Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9426722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9426792Z layer_outputs = layer_module( 2025-08-14T21:53:55.9427099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9427184Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9427468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9427543Z self_outputs = self.self( 2025-08-14T21:53:55.9427811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9427875Z self.key(key_tensor) 2025-08-14T21:53:55.9427879Z 2025-08-14T21:53:55.9427983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9428172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9428242Z return mod(**inputs) 2025-08-14T21:53:55.9428513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9428582Z outputs = self.mobilebert( 2025-08-14T21:53:55.9428860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9428927Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9429193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9429269Z layer_outputs = layer_module( 2025-08-14T21:53:55.9429535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9429625Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9429893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9429961Z self_outputs = self.self( 2025-08-14T21:53:55.9430238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9430310Z self.value(value_tensor) 2025-08-14T21:53:55.9430314Z 2025-08-14T21:53:55.9430401Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9430477Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9430576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9430771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9430834Z return mod(**inputs) 2025-08-14T21:53:55.9431105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9431199Z outputs = self.mobilebert( 2025-08-14T21:53:55.9431470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9431548Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9431840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9431908Z layer_outputs = layer_module( 2025-08-14T21:53:55.9432196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9432277Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9432564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9432687Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9432969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9433083Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9433086Z 2025-08-14T21:53:55.9433196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9433416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9433495Z return mod(**inputs) 2025-08-14T21:53:55.9433791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9433869Z outputs = self.mobilebert( 2025-08-14T21:53:55.9434172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9434249Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9434549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9434623Z layer_outputs = layer_module( 2025-08-14T21:53:55.9434923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9435092Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9435381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9435502Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9435872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9435965Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9435978Z 2025-08-14T21:53:55.9436086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9436289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9436369Z return mod(**inputs) 2025-08-14T21:53:55.9436662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9436738Z outputs = self.mobilebert( 2025-08-14T21:53:55.9437033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9437108Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9437406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9437479Z layer_outputs = layer_module( 2025-08-14T21:53:55.9437801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9437893Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9438174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9438313Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9438593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9438717Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9439007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9439102Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9439107Z 2025-08-14T21:53:55.9439212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9439422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9439493Z return mod(**inputs) 2025-08-14T21:53:55.9439806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9439883Z outputs = self.mobilebert( 2025-08-14T21:53:55.9440187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9440280Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9440555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9440627Z layer_outputs = layer_module( 2025-08-14T21:53:55.9440916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9441012Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9441301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9441412Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9441701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9441798Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9441801Z 2025-08-14T21:53:55.9441907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9442121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9442193Z return mod(**inputs) 2025-08-14T21:53:55.9442490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9442574Z outputs = self.mobilebert( 2025-08-14T21:53:55.9442866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9442942Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9443244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9443319Z layer_outputs = layer_module( 2025-08-14T21:53:55.9443618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9443719Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9444006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9444157Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9444435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9444554Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9444557Z 2025-08-14T21:53:55.9444672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9444867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9444940Z return mod(**inputs) 2025-08-14T21:53:55.9445216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9445292Z outputs = self.mobilebert( 2025-08-14T21:53:55.9445565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9445636Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9445912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9445996Z layer_outputs = layer_module( 2025-08-14T21:53:55.9446268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9446385Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9446657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9446785Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9447059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9447143Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9447148Z 2025-08-14T21:53:55.9447257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9447460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9447536Z return mod(**inputs) 2025-08-14T21:53:55.9447828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9447906Z outputs = self.mobilebert( 2025-08-14T21:53:55.9448200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9448274Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9448562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9448645Z layer_outputs = layer_module( 2025-08-14T21:53:55.9448935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9449038Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9449328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9449457Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9449756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9449882Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9450175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9450271Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9450315Z 2025-08-14T21:53:55.9450422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9450634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9450703Z return mod(**inputs) 2025-08-14T21:53:55.9450998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9451096Z outputs = self.mobilebert( 2025-08-14T21:53:55.9451388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9451471Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9451761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9451834Z layer_outputs = layer_module( 2025-08-14T21:53:55.9452132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9452229Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9452544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9452661Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9452965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9453062Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9453066Z 2025-08-14T21:53:55.9453173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9453382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9453460Z return mod(**inputs) 2025-08-14T21:53:55.9453750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9453833Z outputs = self.mobilebert( 2025-08-14T21:53:55.9454136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9454209Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9454508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9454582Z layer_outputs = layer_module( 2025-08-14T21:53:55.9454876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9454973Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9455275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9455401Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9455690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9455807Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9455817Z 2025-08-14T21:53:55.9455925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9456131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9456208Z return mod(**inputs) 2025-08-14T21:53:55.9456500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9456573Z outputs = self.mobilebert( 2025-08-14T21:53:55.9456873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9456969Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9457266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9457343Z layer_outputs = layer_module( 2025-08-14T21:53:55.9457634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9457767Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9458056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9458184Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9458492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9458583Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9458588Z 2025-08-14T21:53:55.9458701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9458906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9458991Z return mod(**inputs) 2025-08-14T21:53:55.9459293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9459382Z outputs = self.mobilebert( 2025-08-14T21:53:55.9459681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9459755Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9460043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9460127Z layer_outputs = layer_module( 2025-08-14T21:53:55.9460427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9460523Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9460832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9460961Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9461257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9461383Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9461670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9461772Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9461778Z 2025-08-14T21:53:55.9461884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9462095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9462162Z return mod(**inputs) 2025-08-14T21:53:55.9462452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9462548Z outputs = self.mobilebert( 2025-08-14T21:53:55.9462849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9462931Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9463226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9463301Z layer_outputs = layer_module( 2025-08-14T21:53:55.9463604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9463724Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9464021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9464148Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9464467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9464574Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9464578Z 2025-08-14T21:53:55.9464677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9464875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9464948Z return mod(**inputs) 2025-08-14T21:53:55.9465226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9465304Z outputs = self.mobilebert( 2025-08-14T21:53:55.9465602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9465674Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9465968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9466041Z layer_outputs = layer_module( 2025-08-14T21:53:55.9466314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9466413Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9466703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9466829Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9467119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9467236Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9467240Z 2025-08-14T21:53:55.9467353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9467561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9467635Z return mod(**inputs) 2025-08-14T21:53:55.9467925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9467999Z outputs = self.mobilebert( 2025-08-14T21:53:55.9468307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9468381Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9468651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9468730Z layer_outputs = layer_module( 2025-08-14T21:53:55.9469010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9469116Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9469402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9469529Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9469823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9469911Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9469935Z 2025-08-14T21:53:55.9470049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9470254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9470323Z return mod(**inputs) 2025-08-14T21:53:55.9470621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9470712Z outputs = self.mobilebert( 2025-08-14T21:53:55.9471001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9471081Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9471370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9471448Z layer_outputs = layer_module( 2025-08-14T21:53:55.9471739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9471834Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9472149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9472278Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9472587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9472717Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9473002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9473103Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9473109Z 2025-08-14T21:53:55.9473214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9473431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9473501Z return mod(**inputs) 2025-08-14T21:53:55.9473801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9473886Z outputs = self.mobilebert( 2025-08-14T21:53:55.9474184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9474261Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9474564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9474639Z layer_outputs = layer_module( 2025-08-14T21:53:55.9474953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9475086Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9475395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9475492Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9475498Z 2025-08-14T21:53:55.9475607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9475912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9475989Z return mod(**inputs) 2025-08-14T21:53:55.9476294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9476380Z outputs = self.mobilebert( 2025-08-14T21:53:55.9476687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9476792Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9477115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9477192Z layer_outputs = layer_module( 2025-08-14T21:53:55.9477506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9477644Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9477915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9478034Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9478038Z 2025-08-14T21:53:55.9478138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9478345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9478411Z return mod(**inputs) 2025-08-14T21:53:55.9478703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9478784Z outputs = self.mobilebert( 2025-08-14T21:53:55.9479061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9479150Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9479430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9479499Z layer_outputs = layer_module( 2025-08-14T21:53:55.9479775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9479932Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9480202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9480303Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9480307Z 2025-08-14T21:53:55.9480405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9480608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9480673Z return mod(**inputs) 2025-08-14T21:53:55.9480952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9481033Z outputs = self.mobilebert( 2025-08-14T21:53:55.9481333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9481411Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9481720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9481792Z layer_outputs = layer_module( 2025-08-14T21:53:55.9482098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9482266Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9482555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9482684Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9482956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9483088Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9483092Z 2025-08-14T21:53:55.9483198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9483407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9483485Z return mod(**inputs) 2025-08-14T21:53:55.9483781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9483877Z outputs = self.mobilebert( 2025-08-14T21:53:55.9484179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9484254Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9484557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9484630Z layer_outputs = layer_module( 2025-08-14T21:53:55.9484935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9485096Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9485390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9485526Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9485833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9485922Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9485926Z 2025-08-14T21:53:55.9486037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9486241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9486318Z return mod(**inputs) 2025-08-14T21:53:55.9486609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9486684Z outputs = self.mobilebert( 2025-08-14T21:53:55.9486978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9487056Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9487343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9487423Z layer_outputs = layer_module( 2025-08-14T21:53:55.9487708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9487875Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9488162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9488285Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9488584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9488710Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9489005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9489102Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9489106Z 2025-08-14T21:53:55.9489210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9489422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9489510Z return mod(**inputs) 2025-08-14T21:53:55.9489814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9489895Z outputs = self.mobilebert( 2025-08-14T21:53:55.9490195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9490295Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9490594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9490667Z layer_outputs = layer_module( 2025-08-14T21:53:55.9490971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9491138Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9491445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9491561Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9491874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9491975Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9491979Z 2025-08-14T21:53:55.9492131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9492340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9492408Z return mod(**inputs) 2025-08-14T21:53:55.9492698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9492778Z outputs = self.mobilebert( 2025-08-14T21:53:55.9493065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9493143Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9493439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9493540Z layer_outputs = layer_module( 2025-08-14T21:53:55.9493967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9494183Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9494622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9494780Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9495225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9495355Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9495694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9495793Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9495798Z 2025-08-14T21:53:55.9495912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9496117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9496187Z return mod(**inputs) 2025-08-14T21:53:55.9496484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9496557Z outputs = self.mobilebert( 2025-08-14T21:53:55.9496860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9496971Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9497271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9497356Z layer_outputs = layer_module( 2025-08-14T21:53:55.9497648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9497768Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9498062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9498135Z self_outputs = self.self( 2025-08-14T21:53:55.9498441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9498517Z self.query(query_tensor) 2025-08-14T21:53:55.9498523Z 2025-08-14T21:53:55.9498630Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9498844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9498933Z return mod(**inputs) 2025-08-14T21:53:55.9499232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9499305Z outputs = self.mobilebert( 2025-08-14T21:53:55.9499607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9499693Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9499981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9500060Z layer_outputs = layer_module( 2025-08-14T21:53:55.9500366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9500461Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9500769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9500849Z self_outputs = self.self( 2025-08-14T21:53:55.9501143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9501226Z self.key(key_tensor) 2025-08-14T21:53:55.9501229Z 2025-08-14T21:53:55.9501339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9501551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9501623Z return mod(**inputs) 2025-08-14T21:53:55.9501919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9502007Z outputs = self.mobilebert( 2025-08-14T21:53:55.9502313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9502391Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9502697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9502774Z layer_outputs = layer_module( 2025-08-14T21:53:55.9503074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9503173Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9503453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9503547Z self_outputs = self.self( 2025-08-14T21:53:55.9503845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9503926Z self.value(value_tensor) 2025-08-14T21:53:55.9503932Z 2025-08-14T21:53:55.9504031Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9504116Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9504246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9504449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9504518Z return mod(**inputs) 2025-08-14T21:53:55.9504817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9504890Z outputs = self.mobilebert( 2025-08-14T21:53:55.9505185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9505262Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9505548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9505646Z layer_outputs = layer_module( 2025-08-14T21:53:55.9505933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9506048Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9506347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9506474Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9506773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9506864Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9506868Z 2025-08-14T21:53:55.9506973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9507186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9507256Z return mod(**inputs) 2025-08-14T21:53:55.9507558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9507634Z outputs = self.mobilebert( 2025-08-14T21:53:55.9507926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9508008Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9508300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9508376Z layer_outputs = layer_module( 2025-08-14T21:53:55.9508943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9509180Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9509482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9509602Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9509894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9509990Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9509994Z 2025-08-14T21:53:55.9510100Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9510314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9510439Z return mod(**inputs) 2025-08-14T21:53:55.9510731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9510815Z outputs = self.mobilebert( 2025-08-14T21:53:55.9511125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9511239Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9511540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9511617Z layer_outputs = layer_module( 2025-08-14T21:53:55.9511924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9512015Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9512321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9512472Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9512794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9512934Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9513250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9513350Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9513354Z 2025-08-14T21:53:55.9513470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9513678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9513757Z return mod(**inputs) 2025-08-14T21:53:55.9514060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9514136Z outputs = self.mobilebert( 2025-08-14T21:53:55.9514437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9514516Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9514814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9514898Z layer_outputs = layer_module( 2025-08-14T21:53:55.9515193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9515300Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9515608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9515781Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9516098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9516189Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9516194Z 2025-08-14T21:53:55.9516313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9516528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9516598Z return mod(**inputs) 2025-08-14T21:53:55.9516915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9516992Z outputs = self.mobilebert( 2025-08-14T21:53:55.9517292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9517412Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9517692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9517770Z layer_outputs = layer_module( 2025-08-14T21:53:55.9518046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9518158Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9518434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9518543Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9518828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9518940Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9518943Z 2025-08-14T21:53:55.9519043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9519258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9519324Z return mod(**inputs) 2025-08-14T21:53:55.9519630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9519704Z outputs = self.mobilebert( 2025-08-14T21:53:55.9519962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9520035Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9520292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9520359Z layer_outputs = layer_module( 2025-08-14T21:53:55.9520622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9520709Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9520974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9521096Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9521351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9521436Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9521440Z 2025-08-14T21:53:55.9521536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9521730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9521795Z return mod(**inputs) 2025-08-14T21:53:55.9522062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9522136Z outputs = self.mobilebert( 2025-08-14T21:53:55.9522406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9522477Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9522756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9522824Z layer_outputs = layer_module( 2025-08-14T21:53:55.9523100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9523190Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9523458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9523606Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9523884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9524029Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9524301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9524391Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9524394Z 2025-08-14T21:53:55.9524500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9524694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9524757Z return mod(**inputs) 2025-08-14T21:53:55.9525041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9525111Z outputs = self.mobilebert( 2025-08-14T21:53:55.9525410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9525484Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9525779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9525860Z layer_outputs = layer_module( 2025-08-14T21:53:55.9526136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9526235Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9526507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9526619Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9526903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9526986Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9526990Z 2025-08-14T21:53:55.9527090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9527295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9527360Z return mod(**inputs) 2025-08-14T21:53:55.9527651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9527720Z outputs = self.mobilebert( 2025-08-14T21:53:55.9527985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9528064Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9528331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9528409Z layer_outputs = layer_module( 2025-08-14T21:53:55.9528676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9528769Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9529041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9529147Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9529412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9529544Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9529547Z 2025-08-14T21:53:55.9529648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9529849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9529912Z return mod(**inputs) 2025-08-14T21:53:55.9530231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9530334Z outputs = self.mobilebert( 2025-08-14T21:53:55.9530634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9530721Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9531009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9531078Z layer_outputs = layer_module( 2025-08-14T21:53:55.9531359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9531449Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9531747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9531879Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9532176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9532265Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9532268Z 2025-08-14T21:53:55.9532367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9532555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9532627Z return mod(**inputs) 2025-08-14T21:53:55.9532892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9532966Z outputs = self.mobilebert( 2025-08-14T21:53:55.9533236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9533306Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9533584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9533653Z layer_outputs = layer_module( 2025-08-14T21:53:55.9533962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9534057Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9534323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9534446Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9534713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9534829Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9535107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9535194Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9535198Z 2025-08-14T21:53:55.9535302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9535490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9535553Z return mod(**inputs) 2025-08-14T21:53:55.9535846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9535914Z outputs = self.mobilebert( 2025-08-14T21:53:55.9536186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9536268Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9536539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9536615Z layer_outputs = layer_module( 2025-08-14T21:53:55.9536888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9536979Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9537286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9537404Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9537731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9537820Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9537825Z 2025-08-14T21:53:55.9537929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9538162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9538233Z return mod(**inputs) 2025-08-14T21:53:55.9538535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9538604Z outputs = self.mobilebert( 2025-08-14T21:53:55.9538875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9538963Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9539226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9539295Z layer_outputs = layer_module( 2025-08-14T21:53:55.9539565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9539656Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9539927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9540033Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9540299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9540413Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9540416Z 2025-08-14T21:53:55.9540513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9540708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9540771Z return mod(**inputs) 2025-08-14T21:53:55.9541040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9541121Z outputs = self.mobilebert( 2025-08-14T21:53:55.9541386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9541457Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9541732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9541799Z layer_outputs = layer_module( 2025-08-14T21:53:55.9542091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9542182Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9542452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9542596Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9542864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9542952Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9542955Z 2025-08-14T21:53:55.9543054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9543245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9543318Z return mod(**inputs) 2025-08-14T21:53:55.9543595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9543664Z outputs = self.mobilebert( 2025-08-14T21:53:55.9543985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9544059Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9544354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9544427Z layer_outputs = layer_module( 2025-08-14T21:53:55.9544697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9544797Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9545068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9545198Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9545477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9545596Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9545878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9545968Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9545971Z 2025-08-14T21:53:55.9546078Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9546271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9546335Z return mod(**inputs) 2025-08-14T21:53:55.9546620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9546689Z outputs = self.mobilebert( 2025-08-14T21:53:55.9546961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9547040Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9547313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9547391Z layer_outputs = layer_module( 2025-08-14T21:53:55.9547663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9547782Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9548064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9548164Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9548168Z 2025-08-14T21:53:55.9548274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9548469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9548549Z return mod(**inputs) 2025-08-14T21:53:55.9548838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9548908Z outputs = self.mobilebert( 2025-08-14T21:53:55.9549187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9549267Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9549542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9549620Z layer_outputs = layer_module( 2025-08-14T21:53:55.9549895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9550029Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9550311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9550433Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9550437Z 2025-08-14T21:53:55.9550543Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9550735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9550798Z return mod(**inputs) 2025-08-14T21:53:55.9551079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9551152Z outputs = self.mobilebert( 2025-08-14T21:53:55.9551437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9551519Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9551806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9551890Z layer_outputs = layer_module( 2025-08-14T21:53:55.9552179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9552343Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9552638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9552738Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9552742Z 2025-08-14T21:53:55.9552853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9553057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9553125Z return mod(**inputs) 2025-08-14T21:53:55.9553420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9553497Z outputs = self.mobilebert( 2025-08-14T21:53:55.9553784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9553867Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9554152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9554232Z layer_outputs = layer_module( 2025-08-14T21:53:55.9554548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9554714Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9555012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9555164Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9555464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9555561Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9555565Z 2025-08-14T21:53:55.9555745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9555975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9556052Z return mod(**inputs) 2025-08-14T21:53:55.9556353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9556436Z outputs = self.mobilebert( 2025-08-14T21:53:55.9556760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9556851Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9557171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9557248Z layer_outputs = layer_module( 2025-08-14T21:53:55.9557558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9557721Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9558031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9558160Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9558454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9558554Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9558558Z 2025-08-14T21:53:55.9558667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9558884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9558953Z return mod(**inputs) 2025-08-14T21:53:55.9559246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9559326Z outputs = self.mobilebert( 2025-08-14T21:53:55.9559629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9559705Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9560015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9560089Z layer_outputs = layer_module( 2025-08-14T21:53:55.9560385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9560549Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9560838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9560975Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9561294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9561428Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9561733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9561847Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9561851Z 2025-08-14T21:53:55.9561967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9562169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9562238Z return mod(**inputs) 2025-08-14T21:53:55.9562539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9562613Z outputs = self.mobilebert( 2025-08-14T21:53:55.9562917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9562993Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9563309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9563394Z layer_outputs = layer_module( 2025-08-14T21:53:55.9563703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9563880Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9564180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9564294Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9564600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9564688Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9564692Z 2025-08-14T21:53:55.9564803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9565006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9565074Z return mod(**inputs) 2025-08-14T21:53:55.9565377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9565447Z outputs = self.mobilebert( 2025-08-14T21:53:55.9565716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9565795Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9566072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9566149Z layer_outputs = layer_module( 2025-08-14T21:53:55.9566420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9566573Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9566856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9566964Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9567244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9567329Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9567608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9567722Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9567726Z 2025-08-14T21:53:55.9567826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9568020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9568091Z return mod(**inputs) 2025-08-14T21:53:55.9568384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9568464Z outputs = self.mobilebert( 2025-08-14T21:53:55.9568752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9568828Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9569125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9569200Z layer_outputs = layer_module( 2025-08-14T21:53:55.9569498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9569606Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9569893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9569999Z self_outputs = self.self( 2025-08-14T21:53:55.9570276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9570347Z self.query(query_tensor) 2025-08-14T21:53:55.9570350Z 2025-08-14T21:53:55.9570456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9570648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9570724Z return mod(**inputs) 2025-08-14T21:53:55.9571000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9571069Z outputs = self.mobilebert( 2025-08-14T21:53:55.9571350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9571423Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9571702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9571778Z layer_outputs = layer_module( 2025-08-14T21:53:55.9572054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9572145Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9572422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9572493Z self_outputs = self.self( 2025-08-14T21:53:55.9572774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9572841Z self.key(key_tensor) 2025-08-14T21:53:55.9572846Z 2025-08-14T21:53:55.9572951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9573144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9573207Z return mod(**inputs) 2025-08-14T21:53:55.9573494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9573565Z outputs = self.mobilebert( 2025-08-14T21:53:55.9573841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9573937Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9574208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9574289Z layer_outputs = layer_module( 2025-08-14T21:53:55.9574578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9574684Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9574981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9575053Z self_outputs = self.self( 2025-08-14T21:53:55.9575352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9575426Z self.value(value_tensor) 2025-08-14T21:53:55.9575431Z 2025-08-14T21:53:55.9575518Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9575608Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9575716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9575938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9576017Z return mod(**inputs) 2025-08-14T21:53:55.9576326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9576410Z outputs = self.mobilebert( 2025-08-14T21:53:55.9576696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9576771Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9577066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9577142Z layer_outputs = layer_module( 2025-08-14T21:53:55.9577429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9577528Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9577824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9577969Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9578268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9578360Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9578363Z 2025-08-14T21:53:55.9578479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9578686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9578765Z return mod(**inputs) 2025-08-14T21:53:55.9579066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9579142Z outputs = self.mobilebert( 2025-08-14T21:53:55.9579458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9579536Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9579825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9579905Z layer_outputs = layer_module( 2025-08-14T21:53:55.9580193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9580367Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9580693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9580809Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9581119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9581221Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9581226Z 2025-08-14T21:53:55.9581338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9581539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9581607Z return mod(**inputs) 2025-08-14T21:53:55.9581905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9581978Z outputs = self.mobilebert( 2025-08-14T21:53:55.9582274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9582350Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9582658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9582738Z layer_outputs = layer_module( 2025-08-14T21:53:55.9583038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9583126Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9583426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9583553Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9583866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9584001Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9584310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9584419Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9584422Z 2025-08-14T21:53:55.9584533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9584751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9584823Z return mod(**inputs) 2025-08-14T21:53:55.9585124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9585209Z outputs = self.mobilebert( 2025-08-14T21:53:55.9585574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9585649Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9585959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9586033Z layer_outputs = layer_module( 2025-08-14T21:53:55.9586332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9586431Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9586725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9586851Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9587152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9587267Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9587271Z 2025-08-14T21:53:55.9587377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9587583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9587660Z return mod(**inputs) 2025-08-14T21:53:55.9587967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9588041Z outputs = self.mobilebert( 2025-08-14T21:53:55.9588334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9588410Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9588705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9588780Z layer_outputs = layer_module( 2025-08-14T21:53:55.9589080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9589204Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9589494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9589638Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9589930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9590046Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9590050Z 2025-08-14T21:53:55.9590162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9590366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9590435Z return mod(**inputs) 2025-08-14T21:53:55.9590736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9590811Z outputs = self.mobilebert( 2025-08-14T21:53:55.9591106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9591187Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9591484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9591567Z layer_outputs = layer_module( 2025-08-14T21:53:55.9591866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9591974Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9592272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9592408Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9592713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9592806Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9592812Z 2025-08-14T21:53:55.9592929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9593138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9593208Z return mod(**inputs) 2025-08-14T21:53:55.9593519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9593594Z outputs = self.mobilebert( 2025-08-14T21:53:55.9593909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9593994Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9594291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9594391Z layer_outputs = layer_module( 2025-08-14T21:53:55.9594689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9594789Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9595092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9595225Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9595533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9595745Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9596541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9596661Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9596667Z 2025-08-14T21:53:55.9596799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9597014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9597095Z return mod(**inputs) 2025-08-14T21:53:55.9597396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9597479Z outputs = self.mobilebert( 2025-08-14T21:53:55.9597774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9597854Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9598182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9598258Z layer_outputs = layer_module( 2025-08-14T21:53:55.9598555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9598651Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9598939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9599062Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9599369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9599460Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9599471Z 2025-08-14T21:53:55.9599579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9599791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9599868Z return mod(**inputs) 2025-08-14T21:53:55.9600173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9600248Z outputs = self.mobilebert( 2025-08-14T21:53:55.9600550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9600627Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9600942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9601037Z layer_outputs = layer_module( 2025-08-14T21:53:55.9601341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9601451Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9601745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9601882Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9602186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9602306Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9602310Z 2025-08-14T21:53:55.9602425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9602638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9602709Z return mod(**inputs) 2025-08-14T21:53:55.9603014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9603106Z outputs = self.mobilebert( 2025-08-14T21:53:55.9603413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9603516Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9603816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9603898Z layer_outputs = layer_module( 2025-08-14T21:53:55.9604206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9604304Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9604608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9604738Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9605049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9605141Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9605145Z 2025-08-14T21:53:55.9605254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9605473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9605542Z return mod(**inputs) 2025-08-14T21:53:55.9605848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9605924Z outputs = self.mobilebert( 2025-08-14T21:53:55.9606233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9606318Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9606626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9606702Z layer_outputs = layer_module( 2025-08-14T21:53:55.9607005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9607103Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9607403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9607533Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9607839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9607996Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9608309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9608414Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9608433Z 2025-08-14T21:53:55.9608545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9609047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9609177Z return mod(**inputs) 2025-08-14T21:53:55.9609486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9609572Z outputs = self.mobilebert( 2025-08-14T21:53:55.9609883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9609966Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9610327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9610406Z layer_outputs = layer_module( 2025-08-14T21:53:55.9610732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9610841Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9611138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9611267Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9611563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9611656Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9611660Z 2025-08-14T21:53:55.9611777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9611991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9612070Z return mod(**inputs) 2025-08-14T21:53:55.9612381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9612453Z outputs = self.mobilebert( 2025-08-14T21:53:55.9612731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9612802Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9613071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9613152Z layer_outputs = layer_module( 2025-08-14T21:53:55.9613422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9613524Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9613797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9613909Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9614190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9614299Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9614303Z 2025-08-14T21:53:55.9614409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9614602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9614692Z return mod(**inputs) 2025-08-14T21:53:55.9614975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9615047Z outputs = self.mobilebert( 2025-08-14T21:53:55.9615319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9615428Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9615701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9615776Z layer_outputs = layer_module( 2025-08-14T21:53:55.9616051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9616143Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9616424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9616546Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9616843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9616927Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9616931Z 2025-08-14T21:53:55.9617044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9617244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9617309Z return mod(**inputs) 2025-08-14T21:53:55.9617584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9617662Z outputs = self.mobilebert( 2025-08-14T21:53:55.9617939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9618020Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9618293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9618366Z layer_outputs = layer_module( 2025-08-14T21:53:55.9618648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9618740Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9619018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9619139Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9619411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9619538Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9619811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9619902Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9619915Z 2025-08-14T21:53:55.9620018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9620211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9620281Z return mod(**inputs) 2025-08-14T21:53:55.9620561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9620635Z outputs = self.mobilebert( 2025-08-14T21:53:55.9620942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9621037Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9621332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9621402Z layer_outputs = layer_module( 2025-08-14T21:53:55.9621689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9621816Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9622086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9622168Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9622179Z 2025-08-14T21:53:55.9622280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9622475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9622546Z return mod(**inputs) 2025-08-14T21:53:55.9622837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9622909Z outputs = self.mobilebert( 2025-08-14T21:53:55.9623205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9623277Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9623558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9623628Z layer_outputs = layer_module( 2025-08-14T21:53:55.9623926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9624061Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9624351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9624474Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9624477Z 2025-08-14T21:53:55.9624584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9624794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9624869Z return mod(**inputs) 2025-08-14T21:53:55.9625164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9625238Z outputs = self.mobilebert( 2025-08-14T21:53:55.9625536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9625613Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9625922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9625995Z layer_outputs = layer_module( 2025-08-14T21:53:55.9626289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9626458Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9626733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9626834Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9626838Z 2025-08-14T21:53:55.9626943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9627151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9627249Z return mod(**inputs) 2025-08-14T21:53:55.9627551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9627623Z outputs = self.mobilebert( 2025-08-14T21:53:55.9627903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9627988Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9628273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9628344Z layer_outputs = layer_module( 2025-08-14T21:53:55.9628623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9628784Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9629065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9629191Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9629496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9629587Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9629605Z 2025-08-14T21:53:55.9629716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9629907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9629972Z return mod(**inputs) 2025-08-14T21:53:55.9630251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9630321Z outputs = self.mobilebert( 2025-08-14T21:53:55.9630601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9630672Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9630951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9631028Z layer_outputs = layer_module( 2025-08-14T21:53:55.9631299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9631457Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9631728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9631851Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9632147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9632236Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9632239Z 2025-08-14T21:53:55.9632353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9632557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9632627Z return mod(**inputs) 2025-08-14T21:53:55.9632925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9632999Z outputs = self.mobilebert( 2025-08-14T21:53:55.9633286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9633369Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9633658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9633758Z layer_outputs = layer_module( 2025-08-14T21:53:55.9634047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9634209Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9634525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9634652Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9634955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9635079Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9635372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9635475Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9635479Z 2025-08-14T21:53:55.9635600Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9635870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9635954Z return mod(**inputs) 2025-08-14T21:53:55.9636269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9636356Z outputs = self.mobilebert( 2025-08-14T21:53:55.9636655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9636733Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9637048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9637124Z layer_outputs = layer_module( 2025-08-14T21:53:55.9637422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9637590Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9637883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9638008Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9638297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9638384Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9638396Z 2025-08-14T21:53:55.9638503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9638710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9638788Z return mod(**inputs) 2025-08-14T21:53:55.9639085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9639159Z outputs = self.mobilebert( 2025-08-14T21:53:55.9639460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9639533Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9639829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9639903Z layer_outputs = layer_module( 2025-08-14T21:53:55.9640192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9640387Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9640682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9640801Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9641112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9641200Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9641500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9641593Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9641596Z 2025-08-14T21:53:55.9641702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9641919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9641985Z return mod(**inputs) 2025-08-14T21:53:55.9642308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9642382Z outputs = self.mobilebert( 2025-08-14T21:53:55.9642692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9642776Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9643066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9643147Z layer_outputs = layer_module( 2025-08-14T21:53:55.9643446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9643540Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9643838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9643913Z self_outputs = self.self( 2025-08-14T21:53:55.9644208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9644291Z self.query(query_tensor) 2025-08-14T21:53:55.9644294Z 2025-08-14T21:53:55.9644411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9644607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9644670Z return mod(**inputs) 2025-08-14T21:53:55.9644938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9645014Z outputs = self.mobilebert( 2025-08-14T21:53:55.9645289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9645361Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9645644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9645715Z layer_outputs = layer_module( 2025-08-14T21:53:55.9645997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9646082Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9646355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9646432Z self_outputs = self.self( 2025-08-14T21:53:55.9646708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9646803Z self.key(key_tensor) 2025-08-14T21:53:55.9646806Z 2025-08-14T21:53:55.9646917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9647107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9647179Z return mod(**inputs) 2025-08-14T21:53:55.9647471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9647539Z outputs = self.mobilebert( 2025-08-14T21:53:55.9647816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9647886Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9648163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9648233Z layer_outputs = layer_module( 2025-08-14T21:53:55.9648502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9648609Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9648880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9648958Z self_outputs = self.self( 2025-08-14T21:53:55.9649244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9649317Z self.value(value_tensor) 2025-08-14T21:53:55.9649321Z 2025-08-14T21:53:55.9649409Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9649487Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9649589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9649791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9649855Z return mod(**inputs) 2025-08-14T21:53:55.9650148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9650218Z outputs = self.mobilebert( 2025-08-14T21:53:55.9650493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9650572Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9650848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9650917Z layer_outputs = layer_module( 2025-08-14T21:53:55.9651189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9651272Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9651547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9651666Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9651932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9652025Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9652028Z 2025-08-14T21:53:55.9652126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9652326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9652389Z return mod(**inputs) 2025-08-14T21:53:55.9652657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9652752Z outputs = self.mobilebert( 2025-08-14T21:53:55.9653016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9653087Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9653365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9653461Z layer_outputs = layer_module( 2025-08-14T21:53:55.9653750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9653903Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9654171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9654286Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9654552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9654637Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9654641Z 2025-08-14T21:53:55.9654753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9654941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9655029Z return mod(**inputs) 2025-08-14T21:53:55.9655299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9655372Z outputs = self.mobilebert( 2025-08-14T21:53:55.9655637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9655707Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9655978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9656045Z layer_outputs = layer_module( 2025-08-14T21:53:55.9656307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9656395Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9656659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9656781Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9657042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9657158Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9657429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9657518Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9657522Z 2025-08-14T21:53:55.9657626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9657810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9657876Z return mod(**inputs) 2025-08-14T21:53:55.9658149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9658218Z outputs = self.mobilebert( 2025-08-14T21:53:55.9658480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9658559Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9658822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9658925Z layer_outputs = layer_module( 2025-08-14T21:53:55.9659196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9659288Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9659575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9659682Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9659953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9660034Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9660037Z 2025-08-14T21:53:55.9660133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9660330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9660392Z return mod(**inputs) 2025-08-14T21:53:55.9660674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9660752Z outputs = self.mobilebert( 2025-08-14T21:53:55.9661018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9661114Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9661384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9661455Z layer_outputs = layer_module( 2025-08-14T21:53:55.9661736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9661832Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9662112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9662223Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9662497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9662617Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9662620Z 2025-08-14T21:53:55.9662721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9662914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9662987Z return mod(**inputs) 2025-08-14T21:53:55.9663277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9663354Z outputs = self.mobilebert( 2025-08-14T21:53:55.9663622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9663689Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9663963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9664034Z layer_outputs = layer_module( 2025-08-14T21:53:55.9664319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9664408Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9664684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9664816Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9665113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9665196Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9665207Z 2025-08-14T21:53:55.9665307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9665499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9665589Z return mod(**inputs) 2025-08-14T21:53:55.9665864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9665934Z outputs = self.mobilebert( 2025-08-14T21:53:55.9666214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9666284Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9666572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9666642Z layer_outputs = layer_module( 2025-08-14T21:53:55.9666941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9667041Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9667325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9667447Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9667729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9667847Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9668127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9668229Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9668232Z 2025-08-14T21:53:55.9668330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9668527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9668591Z return mod(**inputs) 2025-08-14T21:53:55.9668867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9668935Z outputs = self.mobilebert( 2025-08-14T21:53:55.9669202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9669279Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9669556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9669634Z layer_outputs = layer_module( 2025-08-14T21:53:55.9669915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9670006Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9670291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9670403Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9670675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9670764Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9670767Z 2025-08-14T21:53:55.9670866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9671063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9671147Z return mod(**inputs) 2025-08-14T21:53:55.9671441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9671521Z outputs = self.mobilebert( 2025-08-14T21:53:55.9671807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9671905Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9672192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9672266Z layer_outputs = layer_module( 2025-08-14T21:53:55.9672563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9672661Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9672958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9673095Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9673393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9673534Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9673539Z 2025-08-14T21:53:55.9673644Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9673857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9673933Z return mod(**inputs) 2025-08-14T21:53:55.9674226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9674309Z outputs = self.mobilebert( 2025-08-14T21:53:55.9674607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9674681Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9674985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9675061Z layer_outputs = layer_module( 2025-08-14T21:53:55.9675363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9675467Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9676041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9676186Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9676503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9676593Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9676597Z 2025-08-14T21:53:55.9676716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9676929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9677010Z return mod(**inputs) 2025-08-14T21:53:55.9677314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9677389Z outputs = self.mobilebert( 2025-08-14T21:53:55.9677701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9677780Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9678098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9678219Z layer_outputs = layer_module( 2025-08-14T21:53:55.9678581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9678685Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9679005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9679133Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9679444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9679572Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9679882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9679981Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9679985Z 2025-08-14T21:53:55.9680091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9680323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9680394Z return mod(**inputs) 2025-08-14T21:53:55.9680708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9680785Z outputs = self.mobilebert( 2025-08-14T21:53:55.9681090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9681176Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9681483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9681560Z layer_outputs = layer_module( 2025-08-14T21:53:55.9681865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9681963Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9682276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9682395Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9682691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9682785Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9682788Z 2025-08-14T21:53:55.9682895Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9683109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9683178Z return mod(**inputs) 2025-08-14T21:53:55.9683473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9683556Z outputs = self.mobilebert( 2025-08-14T21:53:55.9683849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9683928Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9684227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9684300Z layer_outputs = layer_module( 2025-08-14T21:53:55.9684599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9684697Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9685013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9685135Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9685435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9685577Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9685580Z 2025-08-14T21:53:55.9685685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9685888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9685963Z return mod(**inputs) 2025-08-14T21:53:55.9686255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9686330Z outputs = self.mobilebert( 2025-08-14T21:53:55.9686637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9686711Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9687035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9687114Z layer_outputs = layer_module( 2025-08-14T21:53:55.9687418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9687523Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9687809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9687942Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9688232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9688319Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9688322Z 2025-08-14T21:53:55.9688435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9688637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9688707Z return mod(**inputs) 2025-08-14T21:53:55.9689004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9689076Z outputs = self.mobilebert( 2025-08-14T21:53:55.9689368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9689442Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9689741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9689823Z layer_outputs = layer_module( 2025-08-14T21:53:55.9690110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9690207Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9690479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9690599Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9690877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9690996Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9691263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9691380Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9691383Z 2025-08-14T21:53:55.9691482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9691684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9691765Z return mod(**inputs) 2025-08-14T21:53:55.9692046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9692123Z outputs = self.mobilebert( 2025-08-14T21:53:55.9692397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9692476Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9692753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9692826Z layer_outputs = layer_module( 2025-08-14T21:53:55.9693109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9693243Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9693531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9693624Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9693627Z 2025-08-14T21:53:55.9693727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9693929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9693993Z return mod(**inputs) 2025-08-14T21:53:55.9694272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9694351Z outputs = self.mobilebert( 2025-08-14T21:53:55.9694624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9694704Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9694976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9695048Z layer_outputs = layer_module( 2025-08-14T21:53:55.9695331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9695449Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9695723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9695841Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9695844Z 2025-08-14T21:53:55.9695944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9696146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9696213Z return mod(**inputs) 2025-08-14T21:53:55.9696491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9696570Z outputs = self.mobilebert( 2025-08-14T21:53:55.9696844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9696920Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9697194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9697265Z layer_outputs = layer_module( 2025-08-14T21:53:55.9697562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9697720Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9698002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9698144Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9698149Z 2025-08-14T21:53:55.9698249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9698447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9698512Z return mod(**inputs) 2025-08-14T21:53:55.9698803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9698886Z outputs = self.mobilebert( 2025-08-14T21:53:55.9699172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9699254Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9699559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9699635Z layer_outputs = layer_module( 2025-08-14T21:53:55.9699960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9700126Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9700428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9700548Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9700821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9700920Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9700923Z 2025-08-14T21:53:55.9701025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9701217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9701294Z return mod(**inputs) 2025-08-14T21:53:55.9701572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9701653Z outputs = self.mobilebert( 2025-08-14T21:53:55.9701939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9702016Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9702321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9702391Z layer_outputs = layer_module( 2025-08-14T21:53:55.9702668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9702821Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9703110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9703244Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9703531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9703618Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9703629Z 2025-08-14T21:53:55.9703755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9703958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9704035Z return mod(**inputs) 2025-08-14T21:53:55.9704326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9704416Z outputs = self.mobilebert( 2025-08-14T21:53:55.9704719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9704793Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9705091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9705165Z layer_outputs = layer_module( 2025-08-14T21:53:55.9705466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9705635Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9705952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9706084Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9706389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9706517Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9706809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9706904Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9706908Z 2025-08-14T21:53:55.9707019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9707233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9707303Z return mod(**inputs) 2025-08-14T21:53:55.9707605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9707679Z outputs = self.mobilebert( 2025-08-14T21:53:55.9707968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9708052Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9708339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9708421Z layer_outputs = layer_module( 2025-08-14T21:53:55.9708969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9709246Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9709554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9709669Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9709962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9710058Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9710062Z 2025-08-14T21:53:55.9710171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9710382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9710452Z return mod(**inputs) 2025-08-14T21:53:55.9710749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9710896Z outputs = self.mobilebert( 2025-08-14T21:53:55.9711217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9711304Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9711632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9711710Z layer_outputs = layer_module( 2025-08-14T21:53:55.9712010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9712179Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9712489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:53:55.9712617Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:53:55.9712926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:53:55.9713058Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:53:55.9713360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9713479Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9713484Z 2025-08-14T21:53:55.9713601Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9713811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9713890Z return mod(**inputs) 2025-08-14T21:53:55.9714189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9714266Z outputs = self.mobilebert( 2025-08-14T21:53:55.9714573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9714655Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9714962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9715043Z layer_outputs = layer_module( 2025-08-14T21:53:55.9715338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9715436Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9715814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9715897Z self_outputs = self.self( 2025-08-14T21:53:55.9716204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:53:55.9716281Z self.query(query_tensor) 2025-08-14T21:53:55.9716285Z 2025-08-14T21:53:55.9716402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9716616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9716688Z return mod(**inputs) 2025-08-14T21:53:55.9717006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9717081Z outputs = self.mobilebert( 2025-08-14T21:53:55.9717371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9717454Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9717742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9717857Z layer_outputs = layer_module( 2025-08-14T21:53:55.9718148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9718237Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9718551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9718625Z self_outputs = self.self( 2025-08-14T21:53:55.9718920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:53:55.9718989Z self.key(key_tensor) 2025-08-14T21:53:55.9718993Z 2025-08-14T21:53:55.9719098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9719307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9719377Z return mod(**inputs) 2025-08-14T21:53:55.9719667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9719764Z outputs = self.mobilebert( 2025-08-14T21:53:55.9720039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9720133Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9720411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9720481Z layer_outputs = layer_module( 2025-08-14T21:53:55.9720764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9720846Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9721132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:53:55.9721199Z self_outputs = self.self( 2025-08-14T21:53:55.9721467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:53:55.9721543Z self.value(value_tensor) 2025-08-14T21:53:55.9721546Z 2025-08-14T21:53:55.9721628Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9721703Z cudagraph partition due to non gpu ops 2025-08-14T21:53:55.9721808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9721994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9722063Z return mod(**inputs) 2025-08-14T21:53:55.9722332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9722400Z outputs = self.mobilebert( 2025-08-14T21:53:55.9722671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9722741Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9723005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9723083Z layer_outputs = layer_module( 2025-08-14T21:53:55.9723349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9723437Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9723701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9723818Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9724109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:53:55.9724189Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9724194Z 2025-08-14T21:53:55.9724299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9724504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9724572Z return mod(**inputs) 2025-08-14T21:53:55.9724852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9724921Z outputs = self.mobilebert( 2025-08-14T21:53:55.9725189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9725268Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9725538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9725615Z layer_outputs = layer_module( 2025-08-14T21:53:55.9725911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:53:55.9726071Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:53:55.9726363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:53:55.9726476Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:53:55.9726755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:53:55.9726837Z layer_input = self.dense(hidden_states) 2025-08-14T21:53:55.9726842Z 2025-08-14T21:53:55.9726943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9727144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9727208Z return mod(**inputs) 2025-08-14T21:53:55.9727491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9727563Z outputs = self.mobilebert( 2025-08-14T21:53:55.9727852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9727927Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9728204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9728274Z layer_outputs = layer_module( 2025-08-14T21:53:55.9728550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:53:55.9728634Z self_attention_outputs = self.attention( 2025-08-14T21:53:55.9728915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:53:55.9729033Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:53:55.9729301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:53:55.9729430Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9729699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9729795Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9729798Z 2025-08-14T21:53:55.9729896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9730099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9730170Z return mod(**inputs) 2025-08-14T21:53:55.9730444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9730514Z outputs = self.mobilebert( 2025-08-14T21:53:55.9730848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9730918Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9731190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9731259Z layer_outputs = layer_module( 2025-08-14T21:53:55.9731533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9731632Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9731896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9732028Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9732291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9732389Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9732392Z 2025-08-14T21:53:55.9732497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9732682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9732744Z return mod(**inputs) 2025-08-14T21:53:55.9733019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9733090Z outputs = self.mobilebert( 2025-08-14T21:53:55.9733363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9733433Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9733697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9733778Z layer_outputs = layer_module( 2025-08-14T21:53:55.9734040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9734138Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9734402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9734508Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9734780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9734886Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9734891Z 2025-08-14T21:53:55.9734990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9735185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9735250Z return mod(**inputs) 2025-08-14T21:53:55.9735524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9735592Z outputs = self.mobilebert( 2025-08-14T21:53:55.9735857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9735936Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9736226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9736301Z layer_outputs = layer_module( 2025-08-14T21:53:55.9736572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9736677Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9736953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9737071Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9737338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9737426Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9737431Z 2025-08-14T21:53:55.9737527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9737723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9737786Z return mod(**inputs) 2025-08-14T21:53:55.9738071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9738150Z outputs = self.mobilebert( 2025-08-14T21:53:55.9738434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9738510Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9738774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9738841Z layer_outputs = layer_module( 2025-08-14T21:53:55.9739111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9739201Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9739466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9739592Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9739858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9739978Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9740245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9740332Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9740335Z 2025-08-14T21:53:55.9740439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9740629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9740698Z return mod(**inputs) 2025-08-14T21:53:55.9740968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9741037Z outputs = self.mobilebert( 2025-08-14T21:53:55.9741320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9741390Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9741677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9741746Z layer_outputs = layer_module( 2025-08-14T21:53:55.9742009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9742128Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9742405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9742516Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9742800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9742899Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9742903Z 2025-08-14T21:53:55.9743011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9743206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9743270Z return mod(**inputs) 2025-08-14T21:53:55.9743553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9743624Z outputs = self.mobilebert( 2025-08-14T21:53:55.9743904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9743990Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9744265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9744361Z layer_outputs = layer_module( 2025-08-14T21:53:55.9744634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9744727Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9745011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9745120Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9745398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9745508Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9745512Z 2025-08-14T21:53:55.9745616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9745841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9745926Z return mod(**inputs) 2025-08-14T21:53:55.9746307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9746402Z outputs = self.mobilebert( 2025-08-14T21:53:55.9746783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9746862Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9747143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9747212Z layer_outputs = layer_module( 2025-08-14T21:53:55.9747499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9747594Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9747892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9748021Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9748316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9748408Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9748411Z 2025-08-14T21:53:55.9748537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9748742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9748807Z return mod(**inputs) 2025-08-14T21:53:55.9749089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9749195Z outputs = self.mobilebert( 2025-08-14T21:53:55.9749470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9749540Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9749822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9749894Z layer_outputs = layer_module( 2025-08-14T21:53:55.9750196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9750293Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9750610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9750749Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9751053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9751187Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9751477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9751575Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9751578Z 2025-08-14T21:53:55.9751696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9751903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9751972Z return mod(**inputs) 2025-08-14T21:53:55.9752281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9752357Z outputs = self.mobilebert( 2025-08-14T21:53:55.9752663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9752743Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9753042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9753127Z layer_outputs = layer_module( 2025-08-14T21:53:55.9753427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9753537Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9753847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9753967Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9754276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9754368Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9754372Z 2025-08-14T21:53:55.9754490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9754699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9754768Z return mod(**inputs) 2025-08-14T21:53:55.9755076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9755170Z outputs = self.mobilebert( 2025-08-14T21:53:55.9755479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9755563Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9755924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9756034Z layer_outputs = layer_module( 2025-08-14T21:53:55.9756330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9756427Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9756730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:53:55.9756847Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:53:55.9757152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9757276Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9757302Z 2025-08-14T21:53:55.9757404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9757608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9757689Z return mod(**inputs) 2025-08-14T21:53:55.9757967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9758046Z outputs = self.mobilebert( 2025-08-14T21:53:55.9758317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9758397Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9758671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9758742Z layer_outputs = layer_module( 2025-08-14T21:53:55.9759023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9759117Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9759397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9759520Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9759791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:53:55.9759882Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9759885Z 2025-08-14T21:53:55.9759988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9760182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9760255Z return mod(**inputs) 2025-08-14T21:53:55.9760531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9760612Z outputs = self.mobilebert( 2025-08-14T21:53:55.9760883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9760957Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9761253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9761325Z layer_outputs = layer_module( 2025-08-14T21:53:55.9761620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:53:55.9761771Z attention_output = ffn_module(attention_output) 2025-08-14T21:53:55.9762062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:53:55.9762196Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:53:55.9762503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:53:55.9762628Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9762926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9763016Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9763020Z 2025-08-14T21:53:55.9763126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9763320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9763384Z return mod(**inputs) 2025-08-14T21:53:55.9763684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9763756Z outputs = self.mobilebert( 2025-08-14T21:53:55.9764058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9764131Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9764406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9764487Z layer_outputs = layer_module( 2025-08-14T21:53:55.9764758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9764877Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9765159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:53:55.9765242Z hidden_states = self.dense(hidden_states) 2025-08-14T21:53:55.9765246Z 2025-08-14T21:53:55.9765352Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9765545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9765610Z return mod(**inputs) 2025-08-14T21:53:55.9765893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9765963Z outputs = self.mobilebert( 2025-08-14T21:53:55.9766241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9766314Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9766587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9766666Z layer_outputs = layer_module( 2025-08-14T21:53:55.9766937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:53:55.9767056Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:53:55.9767335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:53:55.9767445Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:53:55.9767448Z 2025-08-14T21:53:55.9767556Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9767750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9767837Z return mod(**inputs) 2025-08-14T21:53:55.9768118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9768190Z outputs = self.mobilebert( 2025-08-14T21:53:55.9768466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9768552Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9768824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9768902Z layer_outputs = layer_module( 2025-08-14T21:53:55.9769185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9769352Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9769647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:53:55.9769747Z layer_output = self.dense(intermediate_states) 2025-08-14T21:53:55.9769751Z 2025-08-14T21:53:55.9769879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9770082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9770169Z return mod(**inputs) 2025-08-14T21:53:55.9770467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9770540Z outputs = self.mobilebert( 2025-08-14T21:53:55.9770843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9770916Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9771217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9771296Z layer_outputs = layer_module( 2025-08-14T21:53:55.9771594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9771765Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9772055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:53:55.9772180Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:53:55.9772473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9772567Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9772572Z 2025-08-14T21:53:55.9772680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9772892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9772959Z return mod(**inputs) 2025-08-14T21:53:55.9773259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9773334Z outputs = self.mobilebert( 2025-08-14T21:53:55.9773625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9773707Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9773995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9774075Z layer_outputs = layer_module( 2025-08-14T21:53:55.9774373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9774562Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9774864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9774991Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9775301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:53:55.9775397Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:53:55.9775401Z 2025-08-14T21:53:55.9775504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9775713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9775783Z return mod(**inputs) 2025-08-14T21:53:55.9776076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:53:55.9776157Z outputs = self.mobilebert( 2025-08-14T21:53:55.9776472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:53:55.9776556Z encoder_outputs = self.encoder( 2025-08-14T21:53:55.9776861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:53:55.9776936Z layer_outputs = layer_module( 2025-08-14T21:53:55.9777228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:53:55.9777387Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:53:55.9777673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:53:55.9777810Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:53:55.9778098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:53:55.9778229Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:53:55.9778518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:53:55.9778614Z return input_tensor * self.weight + self.bias 2025-08-14T21:53:55.9778618Z 2025-08-14T21:53:55.9778730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9778931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9779008Z return mod(**inputs) 2025-08-14T21:53:55.9779299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1256, in forward 2025-08-14T21:53:55.9779390Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:53:55.9779393Z 2025-08-14T21:53:55.9779513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9779700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9779773Z return mod(**inputs) 2025-08-14T21:53:55.9780049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1274, in forward 2025-08-14T21:53:55.9780153Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:53:55.9780156Z 2025-08-14T21:53:55.9780262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:55.9780449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:55.9780513Z return mod(**inputs) 2025-08-14T21:53:55.9780823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1275, in forward 2025-08-14T21:53:55.9780916Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:53:55.9780919Z 2025-08-14T21:54:08.2891040Z Compilation time (from dynamo_timed): 38.671297292 2025-08-14T21:54:08.2891511Z pass 2025-08-14T21:54:08.2892601Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:08.2893620Z TIMING: _recursive_pre_grad_passes:0.02363 _recursive_joint_graph_passes:1.42656 _recursive_post_grad_passes:0.24634 async_compile.wait:0.32754 code_gen:9.05732 inductor_compile:13.7417 backend_compile:26.62642 gc:0.00112 entire_frame_compile:38.6713 total_wall_time:38.6713 2025-08-14T21:54:08.2894541Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:56761 | FakeTensor.__torch_dispatch__:16441 | ProxyTorchDispatchMode.__torch_dispatch__:21655 2025-08-14T21:54:08.2895083Z Dynamo produced 1 graphs covering 1453 ops with 0 graph breaks (0 unique) 2025-08-14T21:54:14.5493859Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:54:14.5494993Z from pkg_resources import resource_filename 2025-08-14T21:54:15.1444335Z 2025-08-14T21:54:17.0876493Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:54:17.0876808Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:54:17.0893760Z cpu eval OPTForCausalLM 2025-08-14T21:54:18.7802874Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:19.7595896Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:20.6970205Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:28.3294610Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3298992Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3304837Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3305282Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3305618Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3305941Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3306270Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3306578Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3306898Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3307207Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3307533Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3307807Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3308094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3308912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3309392Z return mod(**inputs) 2025-08-14T21:54:28.3309830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3310276Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3310711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3311142Z outputs = self.model.decoder( 2025-08-14T21:54:28.3311541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3311937Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3312357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3312780Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3313475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3313881Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3314297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3314755Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3315277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3315970Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3316166Z 2025-08-14T21:54:28.3316289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3316690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3317065Z return mod(**inputs) 2025-08-14T21:54:28.3317426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3317808Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3318284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3318711Z outputs = self.model.decoder( 2025-08-14T21:54:28.3319086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3319521Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3319928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3320380Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3320757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3321142Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3321535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3321930Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3322355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3322767Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3322914Z 2025-08-14T21:54:28.3323036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3323422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3323776Z return mod(**inputs) 2025-08-14T21:54:28.3324124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3324467Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3324840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3325283Z outputs = self.model.decoder( 2025-08-14T21:54:28.3325650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3326019Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3326415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3326821Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3327195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3327547Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3327924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3328318Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3328728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3329111Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3329257Z 2025-08-14T21:54:28.3329341Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3329554Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3330002Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3330210Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3330446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3330797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3331130Z return mod(**inputs) 2025-08-14T21:54:28.3331457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3331810Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3332177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3332550Z outputs = self.model.decoder( 2025-08-14T21:54:28.3332928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3333284Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3333710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3334111Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3334478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3334858Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3335251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3335682Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3336092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3336485Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3336964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3337475Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3337668Z 2025-08-14T21:54:28.3337779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3338154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3338497Z return mod(**inputs) 2025-08-14T21:54:28.3338840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3339211Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3339583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3339983Z outputs = self.model.decoder( 2025-08-14T21:54:28.3340340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3340709Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3341099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3341489Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3341847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3342236Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3342631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3343076Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3343483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3343907Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3344372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3344876Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3345055Z 2025-08-14T21:54:28.3345166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3345546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3345886Z return mod(**inputs) 2025-08-14T21:54:28.3346224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3346599Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3346991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3347379Z outputs = self.model.decoder( 2025-08-14T21:54:28.3347756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3348132Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3348544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3348928Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3349291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3349667Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3350058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3350472Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3350883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3351285Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3351427Z 2025-08-14T21:54:28.3351540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3351916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3352257Z return mod(**inputs) 2025-08-14T21:54:28.3352618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3352986Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3353366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3353759Z outputs = self.model.decoder( 2025-08-14T21:54:28.3354119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3354479Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3354871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3355265Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3355629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3356155Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3356566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3356986Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3357124Z 2025-08-14T21:54:28.3357236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3357618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3357941Z return mod(**inputs) 2025-08-14T21:54:28.3358271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3358617Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3359007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3359379Z outputs = self.model.decoder( 2025-08-14T21:54:28.3359718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3360058Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3360422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3360796Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3361133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3361488Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3361893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3362292Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3362444Z 2025-08-14T21:54:28.3362562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3362920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3363239Z return mod(**inputs) 2025-08-14T21:54:28.3363563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3363919Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3364296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3364680Z outputs = self.model.decoder( 2025-08-14T21:54:28.3365023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3365380Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3365757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3366135Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3366482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3366853Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3367236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3367618Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3367766Z 2025-08-14T21:54:28.3367884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3368238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3368563Z return mod(**inputs) 2025-08-14T21:54:28.3368886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3369243Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3369620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3369994Z outputs = self.model.decoder( 2025-08-14T21:54:28.3370348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3370697Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3371071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3371453Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3371799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3372162Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3372528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3372950Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3373347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3373757Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3373922Z 2025-08-14T21:54:28.3374035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3374381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3374695Z return mod(**inputs) 2025-08-14T21:54:28.3375008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3375359Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3375722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3376105Z outputs = self.model.decoder( 2025-08-14T21:54:28.3376463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3376818Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3377192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3377565Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3377910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3378262Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3378628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3379053Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3379452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3379834Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3379970Z 2025-08-14T21:54:28.3380081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3380433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3380756Z return mod(**inputs) 2025-08-14T21:54:28.3381084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3381443Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3381801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3382166Z outputs = self.model.decoder( 2025-08-14T21:54:28.3382503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3382841Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3383207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3383572Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3383912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3384255Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3384622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3385103Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3385486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3385868Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3386012Z 2025-08-14T21:54:28.3386114Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3386323Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3386523Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3386726Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3386953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3387292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3387605Z return mod(**inputs) 2025-08-14T21:54:28.3387923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3388270Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3388625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3389007Z outputs = self.model.decoder( 2025-08-14T21:54:28.3389342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3389693Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3390057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3390418Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3390752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3391097Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3391464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3391853Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3392233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3392631Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3393095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3393610Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3393802Z 2025-08-14T21:54:28.3393913Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3394292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3394646Z return mod(**inputs) 2025-08-14T21:54:28.3394996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3395368Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3395853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3396267Z outputs = self.model.decoder( 2025-08-14T21:54:28.3396630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3397017Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3397414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3397848Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3398189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3398552Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3398956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3399345Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3399740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3400156Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3400598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3401048Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3401218Z 2025-08-14T21:54:28.3401323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3401684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3402010Z return mod(**inputs) 2025-08-14T21:54:28.3402332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3402688Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3403075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3403448Z outputs = self.model.decoder( 2025-08-14T21:54:28.3403810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3404161Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3404531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3404892Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3405236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3405598Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3405964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3406362Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3406759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3407141Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3407276Z 2025-08-14T21:54:28.3407380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3407747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3408065Z return mod(**inputs) 2025-08-14T21:54:28.3408383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3408856Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3409224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3409592Z outputs = self.model.decoder( 2025-08-14T21:54:28.3409922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3410265Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3410629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3410992Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3411324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3411678Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3412041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3412471Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3412612Z 2025-08-14T21:54:28.3412714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3413065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3413385Z return mod(**inputs) 2025-08-14T21:54:28.3413703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3414095Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3414459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3414822Z outputs = self.model.decoder( 2025-08-14T21:54:28.3415143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3415485Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3415844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3416200Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3416570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3416923Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3417290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3417695Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3417851Z 2025-08-14T21:54:28.3417951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3418299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3418603Z return mod(**inputs) 2025-08-14T21:54:28.3418919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3419267Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3419629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3420009Z outputs = self.model.decoder( 2025-08-14T21:54:28.3420341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3420683Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3421032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3421389Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3421726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3422073Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3422428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3422802Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3422931Z 2025-08-14T21:54:28.3423048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3423383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3423682Z return mod(**inputs) 2025-08-14T21:54:28.3423993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3424323Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3424665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3425021Z outputs = self.model.decoder( 2025-08-14T21:54:28.3425340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3425701Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3426054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3426502Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3426886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3427251Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3427620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3428008Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3428395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3428805Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3428979Z 2025-08-14T21:54:28.3429087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3429443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3429775Z return mod(**inputs) 2025-08-14T21:54:28.3430103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3430447Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3430830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3431188Z outputs = self.model.decoder( 2025-08-14T21:54:28.3431524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3431865Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3432297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3432672Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3433015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3433375Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3433745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3434212Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3434649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3435067Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3435219Z 2025-08-14T21:54:28.3435323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3435739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3436078Z return mod(**inputs) 2025-08-14T21:54:28.3436403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3436785Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3437189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3437564Z outputs = self.model.decoder( 2025-08-14T21:54:28.3437898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3438249Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3438624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3439003Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3439333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3439720Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3440093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3440484Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3440882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3441284Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3441423Z 2025-08-14T21:54:28.3441517Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3441730Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3441946Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3442157Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3442388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3442753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3443087Z return mod(**inputs) 2025-08-14T21:54:28.3443411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3443769Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3444167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3444552Z outputs = self.model.decoder( 2025-08-14T21:54:28.3444936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3445308Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3445674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3446044Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3446383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3446742Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3447111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3447504Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3447897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3448295Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3448736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3449205Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3449396Z 2025-08-14T21:54:28.3449500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3449859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3450183Z return mod(**inputs) 2025-08-14T21:54:28.3450504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3450861Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3451236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3451609Z outputs = self.model.decoder( 2025-08-14T21:54:28.3451958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3452311Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3452686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3453056Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3453402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3453788Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3454157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3454560Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3454973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3455374Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3455841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3456337Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3456510Z 2025-08-14T21:54:28.3456631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3457024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3457356Z return mod(**inputs) 2025-08-14T21:54:28.3457690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3458069Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3458473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3458901Z outputs = self.model.decoder( 2025-08-14T21:54:28.3459265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3459645Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3460046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3460433Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3460800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3461175Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3461571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3461993Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3462411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3462802Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3462952Z 2025-08-14T21:54:28.3463061Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3463443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3463753Z return mod(**inputs) 2025-08-14T21:54:28.3464174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3492669Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3493125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3493554Z outputs = self.model.decoder( 2025-08-14T21:54:28.3493914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3494294Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3494669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3495046Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3495402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3495772Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3496148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3496630Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3496775Z 2025-08-14T21:54:28.3496896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3497273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3497659Z return mod(**inputs) 2025-08-14T21:54:28.3498002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3498365Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3498735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3499184Z outputs = self.model.decoder( 2025-08-14T21:54:28.3499527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3499872Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3500249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3500609Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3500979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3501330Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3501724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3502113Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3502264Z 2025-08-14T21:54:28.3502370Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3502716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3503035Z return mod(**inputs) 2025-08-14T21:54:28.3503356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3503701Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3504070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3504445Z outputs = self.model.decoder( 2025-08-14T21:54:28.3504810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3505190Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3505560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3505917Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3506249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3506599Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3506963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3507340Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3507477Z 2025-08-14T21:54:28.3507582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3507941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3508265Z return mod(**inputs) 2025-08-14T21:54:28.3508584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3509082Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3509452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3509822Z outputs = self.model.decoder( 2025-08-14T21:54:28.3510250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3510592Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3510960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3511319Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3511692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3512045Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3512411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:54:28.3512829Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:54:28.3513018Z 2025-08-14T21:54:28.3513121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3513472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3513794Z return mod(**inputs) 2025-08-14T21:54:28.3514106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3514473Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3514843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3515237Z outputs = self.model.decoder( 2025-08-14T21:54:28.3515580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3516022Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3516427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3516833Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3517173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3517530Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3517922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3518354Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3518779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3519213Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3519387Z 2025-08-14T21:54:28.3519493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3519861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3520191Z return mod(**inputs) 2025-08-14T21:54:28.3520534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3520900Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3521291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3521688Z outputs = self.model.decoder( 2025-08-14T21:54:28.3522044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3522417Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3522803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3523190Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3523539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3523909Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3524312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3524686Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3525060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3525419Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3525566Z 2025-08-14T21:54:28.3525670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3526003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3526309Z return mod(**inputs) 2025-08-14T21:54:28.3526617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3526961Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3527329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3527707Z outputs = self.model.decoder( 2025-08-14T21:54:28.3528049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3528414Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3528787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3529164Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3529504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3529846Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3530212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3530602Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3530984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3531362Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3531506Z 2025-08-14T21:54:28.3531590Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3531803Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3532001Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3532202Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3532434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3532775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3533092Z return mod(**inputs) 2025-08-14T21:54:28.3533417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3533763Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3534120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3534493Z outputs = self.model.decoder( 2025-08-14T21:54:28.3534834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3535187Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3535561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3535933Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3536271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3536621Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3536978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3537363Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3537771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3538173Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3538623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3539125Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3539311Z 2025-08-14T21:54:28.3539425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3539796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3540112Z return mod(**inputs) 2025-08-14T21:54:28.3540431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3540774Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3541134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3541501Z outputs = self.model.decoder( 2025-08-14T21:54:28.3541862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3542203Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3542573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3542936Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3543275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3543621Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3543980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3544359Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3544734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3545106Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3545538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3545985Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3546145Z 2025-08-14T21:54:28.3546255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3546598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3546929Z return mod(**inputs) 2025-08-14T21:54:28.3547238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3547568Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3547924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3548284Z outputs = self.model.decoder( 2025-08-14T21:54:28.3548621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3548963Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3549336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3549708Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3550050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3550443Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3550841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3551287Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3551706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3552124Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3552270Z 2025-08-14T21:54:28.3552389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3552785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3553099Z return mod(**inputs) 2025-08-14T21:54:28.3553423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3553777Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3554159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3554564Z outputs = self.model.decoder( 2025-08-14T21:54:28.3554934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3555307Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3555790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3556215Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3556655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3557032Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3557440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3557805Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3557936Z 2025-08-14T21:54:28.3558044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3558379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3558692Z return mod(**inputs) 2025-08-14T21:54:28.3559011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3559347Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3559714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3560081Z outputs = self.model.decoder( 2025-08-14T21:54:28.3560414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3560751Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3561113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3561473Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3561802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3562152Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3562526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3562932Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3563093Z 2025-08-14T21:54:28.3563202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3563586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3563938Z return mod(**inputs) 2025-08-14T21:54:28.3564283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3564645Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3565036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3565451Z outputs = self.model.decoder( 2025-08-14T21:54:28.3565783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3566124Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3566495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3566892Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3567231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3567592Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3567975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3568346Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3568480Z 2025-08-14T21:54:28.3568580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3568927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3569239Z return mod(**inputs) 2025-08-14T21:54:28.3569563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3569908Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3570285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3570650Z outputs = self.model.decoder( 2025-08-14T21:54:28.3570971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3571308Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3571668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3572021Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3572355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3572705Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3573067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3573445Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3573837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3574248Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3574412Z 2025-08-14T21:54:28.3574524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3574873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3575195Z return mod(**inputs) 2025-08-14T21:54:28.3575526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3575858Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3576220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3576584Z outputs = self.model.decoder( 2025-08-14T21:54:28.3576929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3577272Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3577644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3578013Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3578351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3578737Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3579103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3579494Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3579885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3580290Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3580426Z 2025-08-14T21:54:28.3580538Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3580897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3581219Z return mod(**inputs) 2025-08-14T21:54:28.3581543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3581898Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3582258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3582632Z outputs = self.model.decoder( 2025-08-14T21:54:28.3582995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3583349Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3583733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3584161Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3584537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3584923Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3585382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3585806Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3586208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3586594Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3586744Z 2025-08-14T21:54:28.3586828Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3587051Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3587262Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3587474Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3587721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3588082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3588406Z return mod(**inputs) 2025-08-14T21:54:28.3588743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3589101Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3589479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3589854Z outputs = self.model.decoder( 2025-08-14T21:54:28.3590203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3590563Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3590935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3591316Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3591665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3592035Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3592410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3592842Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3593257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3593668Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3594173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3594680Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3594872Z 2025-08-14T21:54:28.3594990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3595360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3595775Z return mod(**inputs) 2025-08-14T21:54:28.3596138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3596522Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3596910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3597347Z outputs = self.model.decoder( 2025-08-14T21:54:28.3597687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3598045Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3598414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3598786Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3599145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3599490Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3599869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3600278Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3600659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3601049Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3601488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3601936Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3602100Z 2025-08-14T21:54:28.3602206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3602563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3602889Z return mod(**inputs) 2025-08-14T21:54:28.3603221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3603567Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3603950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3604366Z outputs = self.model.decoder( 2025-08-14T21:54:28.3604722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3605077Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3605450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3605826Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3606167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3606538Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3606925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3607304Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3607695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3608108Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3608240Z 2025-08-14T21:54:28.3608354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3608830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3609165Z return mod(**inputs) 2025-08-14T21:54:28.3609501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3609846Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3610211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3610591Z outputs = self.model.decoder( 2025-08-14T21:54:28.3610988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3611485Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3611879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3612280Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3612630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3612984Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3613355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3613736Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3613875Z 2025-08-14T21:54:28.3613980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3614338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3614660Z return mod(**inputs) 2025-08-14T21:54:28.3614979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3615322Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3615694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3616064Z outputs = self.model.decoder( 2025-08-14T21:54:28.3616394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3616749Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3617119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3617489Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3617824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3618185Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3618561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3618965Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3619118Z 2025-08-14T21:54:28.3619221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3619578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3619906Z return mod(**inputs) 2025-08-14T21:54:28.3620222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3620608Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3620980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3621352Z outputs = self.model.decoder( 2025-08-14T21:54:28.3621693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3622077Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3622450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3622816Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3623165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3623527Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3623903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3624280Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3624422Z 2025-08-14T21:54:28.3624524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3624901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3625241Z return mod(**inputs) 2025-08-14T21:54:28.3625569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3625911Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3626271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3626658Z outputs = self.model.decoder( 2025-08-14T21:54:28.3626999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3627351Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3627725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3628091Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3628441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3628802Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3629181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:54:28.3629599Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:54:28.3629786Z 2025-08-14T21:54:28.3629885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3630234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3630547Z return mod(**inputs) 2025-08-14T21:54:28.3630875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3631228Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3631592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3631962Z outputs = self.model.decoder( 2025-08-14T21:54:28.3632308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3632661Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3633048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3633450Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3633818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3634210Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3634615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3635036Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3635457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3635975Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3636166Z 2025-08-14T21:54:28.3636285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3636673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3637023Z return mod(**inputs) 2025-08-14T21:54:28.3637371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3637739Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3638105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3638462Z outputs = self.model.decoder( 2025-08-14T21:54:28.3638821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3639163Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3639542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3639897Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3640236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3640590Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3640949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3641340Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3641725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3642097Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3642230Z 2025-08-14T21:54:28.3642330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3642681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3643005Z return mod(**inputs) 2025-08-14T21:54:28.3643330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3643676Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3644056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3644422Z outputs = self.model.decoder( 2025-08-14T21:54:28.3644749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3645082Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3645434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3645791Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3646119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3646472Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3646846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3647238Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3647643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3648049Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3648182Z 2025-08-14T21:54:28.3648267Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3648468Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3648673Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3648876Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3649096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3649468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3649781Z return mod(**inputs) 2025-08-14T21:54:28.3650094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3650428Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3650796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3651153Z outputs = self.model.decoder( 2025-08-14T21:54:28.3651470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3651803Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3652176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3652527Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3652866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3653212Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3653575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3653964Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3654349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3654726Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3655144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3655597Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3655778Z 2025-08-14T21:54:28.3655880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3656218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3656522Z return mod(**inputs) 2025-08-14T21:54:28.3656827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3657163Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3657519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3657872Z outputs = self.model.decoder( 2025-08-14T21:54:28.3658199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3658531Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3658889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3659239Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3659578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3659937Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3660296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3660663Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3661037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3661449Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3661881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3662319Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3662494Z 2025-08-14T21:54:28.3662592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3662935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3663243Z return mod(**inputs) 2025-08-14T21:54:28.3663559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3663899Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3664261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3664635Z outputs = self.model.decoder( 2025-08-14T21:54:28.3664979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3665349Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3665707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3666134Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3666482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3666838Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3667201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3667594Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3667982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3668351Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3668491Z 2025-08-14T21:54:28.3668593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3668946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3669263Z return mod(**inputs) 2025-08-14T21:54:28.3669578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3669923Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3670287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3670644Z outputs = self.model.decoder( 2025-08-14T21:54:28.3670981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3671322Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3671683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3672040Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3672382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3672729Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3673090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3673471Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3673614Z 2025-08-14T21:54:28.3673717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3674075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3674403Z return mod(**inputs) 2025-08-14T21:54:28.3674764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3675134Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3675530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3676014Z outputs = self.model.decoder( 2025-08-14T21:54:28.3676422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3676808Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3677163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3677525Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3677866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3678219Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3678574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3678977Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3679147Z 2025-08-14T21:54:28.3679256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3679601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3679938Z return mod(**inputs) 2025-08-14T21:54:28.3680265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3680627Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3680977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3681335Z outputs = self.model.decoder( 2025-08-14T21:54:28.3681663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3681997Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3682345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3682702Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3683045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3683392Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3683762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3684137Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3684271Z 2025-08-14T21:54:28.3684377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3684725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3685042Z return mod(**inputs) 2025-08-14T21:54:28.3685359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3685701Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3686066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3686436Z outputs = self.model.decoder( 2025-08-14T21:54:28.3686772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3687116Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3687506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3687892Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3688227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3688612Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3688988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3689386Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3689824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3690267Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3690441Z 2025-08-14T21:54:28.3690561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3690939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3691286Z return mod(**inputs) 2025-08-14T21:54:28.3691627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3691977Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3692333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3692749Z outputs = self.model.decoder( 2025-08-14T21:54:28.3693094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3693456Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3693871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3694259Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3694607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3694961Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3695340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3695740Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3696139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3696515Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3696658Z 2025-08-14T21:54:28.3696761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3697119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3697449Z return mod(**inputs) 2025-08-14T21:54:28.3697765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3698108Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3698479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3698855Z outputs = self.model.decoder( 2025-08-14T21:54:28.3699201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3699554Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3699926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3700307Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3700677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3701067Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3701464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3701859Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3702251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3702661Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3702803Z 2025-08-14T21:54:28.3702886Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3703115Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3703197Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3703307Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3703418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3703626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3703704Z return mod(**inputs) 2025-08-14T21:54:28.3703931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3704018Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3704267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3704348Z outputs = self.model.decoder( 2025-08-14T21:54:28.3704579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3704674Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3704935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3705032Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3705262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3705351Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3705605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3705708Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3705962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3706064Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3706369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3706511Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3706515Z 2025-08-14T21:54:28.3706625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3706840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3706910Z return mod(**inputs) 2025-08-14T21:54:28.3707143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3707220Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3707468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3707553Z outputs = self.model.decoder( 2025-08-14T21:54:28.3707780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3707857Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3708120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3708194Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3708429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3708512Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3708896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3709015Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3709316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3709418Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3709734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3709881Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3709887Z 2025-08-14T21:54:28.3710002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3710211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3710279Z return mod(**inputs) 2025-08-14T21:54:28.3710515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3710593Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3710848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3710927Z outputs = self.model.decoder( 2025-08-14T21:54:28.3711175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3711263Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3711540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3711617Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3711858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3711942Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3712211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3712317Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3712577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3712674Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3712678Z 2025-08-14T21:54:28.3712789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3713014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3713084Z return mod(**inputs) 2025-08-14T21:54:28.3713317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3713402Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3713665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3713746Z outputs = self.model.decoder( 2025-08-14T21:54:28.3713995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3714075Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3714344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3714422Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3714673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3714761Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3715013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3715098Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3715108Z 2025-08-14T21:54:28.3715217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3715427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3715533Z return mod(**inputs) 2025-08-14T21:54:28.3715832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3715920Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3716191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3716295Z outputs = self.model.decoder( 2025-08-14T21:54:28.3716536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3716616Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3716881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3716962Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3717194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3717275Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3717556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3717660Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3717666Z 2025-08-14T21:54:28.3717782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3718007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3718078Z return mod(**inputs) 2025-08-14T21:54:28.3718312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3718388Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3718640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3718726Z outputs = self.model.decoder( 2025-08-14T21:54:28.3718950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3719035Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3719282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3719351Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3719566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3719640Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3719871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3719947Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3719952Z 2025-08-14T21:54:28.3720047Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3720244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3720305Z return mod(**inputs) 2025-08-14T21:54:28.3720509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3720587Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3720810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3720884Z outputs = self.model.decoder( 2025-08-14T21:54:28.3721085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3721153Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3721386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3721470Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3721674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3721754Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3721978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:54:28.3722130Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:54:28.3722134Z 2025-08-14T21:54:28.3722231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3722419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3722491Z return mod(**inputs) 2025-08-14T21:54:28.3722700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3722779Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3723010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3723080Z outputs = self.model.decoder( 2025-08-14T21:54:28.3723320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3723404Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3723643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3723719Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3723922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3724002Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3724228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3724321Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3724549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3724657Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3724661Z 2025-08-14T21:54:28.3724764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3724951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3725011Z return mod(**inputs) 2025-08-14T21:54:28.3725219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3725290Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3725514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3725590Z outputs = self.model.decoder( 2025-08-14T21:54:28.3725791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3725867Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3726094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3726162Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3726379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3726455Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3726687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3726787Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3727014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3727119Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3727123Z 2025-08-14T21:54:28.3727221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3727416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3727485Z return mod(**inputs) 2025-08-14T21:54:28.3727717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3727793Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3728014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3728081Z outputs = self.model.decoder( 2025-08-14T21:54:28.3728289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3728357Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3728581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3728654Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3728872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3728953Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3729192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3729288Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3729526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3729609Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3729612Z 2025-08-14T21:54:28.3729691Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3729775Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3729851Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3729933Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3730032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3730232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3730306Z return mod(**inputs) 2025-08-14T21:54:28.3730528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3730600Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3730853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3730925Z outputs = self.model.decoder( 2025-08-14T21:54:28.3731150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3731226Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3731463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3731540Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3731763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3731845Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3732099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3732189Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3732424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3732517Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3732793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3732953Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3732956Z 2025-08-14T21:54:28.3733054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3733250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3733331Z return mod(**inputs) 2025-08-14T21:54:28.3733539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3733619Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3733849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3733920Z outputs = self.model.decoder( 2025-08-14T21:54:28.3734130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3734202Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3734434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3734519Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3734733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3734817Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3735060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3735173Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3735397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3735488Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3735771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3735877Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3735881Z 2025-08-14T21:54:28.3735982Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3736179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3736243Z return mod(**inputs) 2025-08-14T21:54:28.3736462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3736536Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3736770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3736850Z outputs = self.model.decoder( 2025-08-14T21:54:28.3737062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3737136Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3737373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3737445Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3737665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3737743Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3737973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3738074Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3738301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3738387Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3738408Z 2025-08-14T21:54:28.3738509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3738701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3738771Z return mod(**inputs) 2025-08-14T21:54:28.3738977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3739067Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3739307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3739378Z outputs = self.model.decoder( 2025-08-14T21:54:28.3739594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3739664Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3739892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3739971Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3740182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3740279Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3740515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3740609Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3740614Z 2025-08-14T21:54:28.3740718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3740908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3740970Z return mod(**inputs) 2025-08-14T21:54:28.3741184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3741257Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3741493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3741563Z outputs = self.model.decoder( 2025-08-14T21:54:28.3741770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3741851Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3742082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3742149Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3742367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3742443Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3742679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3742773Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3742777Z 2025-08-14T21:54:28.3742873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3743069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3743131Z return mod(**inputs) 2025-08-14T21:54:28.3743347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3743419Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3743651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3743729Z outputs = self.model.decoder( 2025-08-14T21:54:28.3743941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3744011Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3744265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3744336Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3744560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3744666Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3744893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3744980Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3744983Z 2025-08-14T21:54:28.3745081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3745270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3745340Z return mod(**inputs) 2025-08-14T21:54:28.3745545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3745626Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3745873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3745946Z outputs = self.model.decoder( 2025-08-14T21:54:28.3746161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3746246Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3746484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3746554Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3746761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3746844Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3747077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3747170Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3747409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3747518Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3747523Z 2025-08-14T21:54:28.3747629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3747818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3747881Z return mod(**inputs) 2025-08-14T21:54:28.3748094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3748165Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3748395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3748475Z outputs = self.model.decoder( 2025-08-14T21:54:28.3748683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3748761Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3748990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3749064Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3749286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3749365Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3749609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3749706Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3749962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3750047Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3750051Z 2025-08-14T21:54:28.3750151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3750346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3750439Z return mod(**inputs) 2025-08-14T21:54:28.3750652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3750730Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3750964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3751036Z outputs = self.model.decoder( 2025-08-14T21:54:28.3751254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3751327Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3751563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3751657Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3751871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3751973Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3752215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3752318Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3752572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3752660Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3752664Z 2025-08-14T21:54:28.3752756Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3752838Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3752919Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3753007Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3753113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3753323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3753400Z return mod(**inputs) 2025-08-14T21:54:28.3753623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3753705Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3753953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3754028Z outputs = self.model.decoder( 2025-08-14T21:54:28.3754258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3754334Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3754582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3754664Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3754895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3754983Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3755233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3755333Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3755593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3755793Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3756105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3756259Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3756263Z 2025-08-14T21:54:28.3756369Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3756612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3756684Z return mod(**inputs) 2025-08-14T21:54:28.3756918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3757005Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3757263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3757351Z outputs = self.model.decoder( 2025-08-14T21:54:28.3757590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3757664Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3757927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3758002Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3758233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3758323Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3758556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3758662Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3758898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3758993Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3759290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3759401Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3759405Z 2025-08-14T21:54:28.3759514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3759711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3759776Z return mod(**inputs) 2025-08-14T21:54:28.3759997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3760071Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3760314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3760396Z outputs = self.model.decoder( 2025-08-14T21:54:28.3760614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3760693Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3760934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3761006Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3761235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3761313Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3761552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3761656Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3761896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3762835Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3762838Z 2025-08-14T21:54:28.3762941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3763137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3763237Z return mod(**inputs) 2025-08-14T21:54:28.3763452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3763535Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3763777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3763849Z outputs = self.model.decoder( 2025-08-14T21:54:28.3764074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3764151Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3764391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3764469Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3764713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3764800Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3765051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3765131Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3765136Z 2025-08-14T21:54:28.3765383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3765582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3765653Z return mod(**inputs) 2025-08-14T21:54:28.3765865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3765937Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3766181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3766250Z outputs = self.model.decoder( 2025-08-14T21:54:28.3766451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3766531Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3766757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3766833Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3767043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3767119Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3767355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3767449Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3767453Z 2025-08-14T21:54:28.3767559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3767746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3767813Z return mod(**inputs) 2025-08-14T21:54:28.3768025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3768095Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3768320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3768400Z outputs = self.model.decoder( 2025-08-14T21:54:28.3768619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3768732Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3768958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3769025Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3769243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3769338Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3769570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3769654Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3769657Z 2025-08-14T21:54:28.3769753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3769950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3770014Z return mod(**inputs) 2025-08-14T21:54:28.3770222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3770300Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3770545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3770623Z outputs = self.model.decoder( 2025-08-14T21:54:28.3770845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3770917Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3771152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3771220Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3771427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3771509Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3771739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:54:28.3771878Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:54:28.3771883Z 2025-08-14T21:54:28.3771981Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3772173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3772245Z return mod(**inputs) 2025-08-14T21:54:28.3772451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3772521Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3772760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3772835Z outputs = self.model.decoder( 2025-08-14T21:54:28.3773054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3773127Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3773363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3773442Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3773664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3773747Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3773983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3774081Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3774324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3774459Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3774462Z 2025-08-14T21:54:28.3774561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3774768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3774855Z return mod(**inputs) 2025-08-14T21:54:28.3775077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3775152Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3775388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3775467Z outputs = self.model.decoder( 2025-08-14T21:54:28.3775679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3775754Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3776002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3776070Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3776303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3776388Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3776636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3776738Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3776962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3777049Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3777052Z 2025-08-14T21:54:28.3777151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3777340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3777412Z return mod(**inputs) 2025-08-14T21:54:28.3777621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3777692Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3777932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3778001Z outputs = self.model.decoder( 2025-08-14T21:54:28.3778217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3778288Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3778518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3778595Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3778818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3778893Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3779126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3779220Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3779451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3779530Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3779533Z 2025-08-14T21:54:28.3779607Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3779691Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3779765Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3779844Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3779960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3780146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3780215Z return mod(**inputs) 2025-08-14T21:54:28.3780421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3780508Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3780752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3780821Z outputs = self.model.decoder( 2025-08-14T21:54:28.3781028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3781095Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3781314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3781388Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3781589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3781678Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3781911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3782020Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3782257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3782351Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3782629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3782763Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3782768Z 2025-08-14T21:54:28.3782865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3783060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3783127Z return mod(**inputs) 2025-08-14T21:54:28.3783340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3783421Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3783669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3783739Z outputs = self.model.decoder( 2025-08-14T21:54:28.3783952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3784021Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3784256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3784328Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3784540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3784627Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3784859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3784957Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3785198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3785292Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3785581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3785712Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3785716Z 2025-08-14T21:54:28.3785815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3786019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3786085Z return mod(**inputs) 2025-08-14T21:54:28.3786304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3786399Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3786637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3786717Z outputs = self.model.decoder( 2025-08-14T21:54:28.3786933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3787007Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3787253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3787323Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3787565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3787646Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3787905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3788011Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3788247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3788326Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3788338Z 2025-08-14T21:54:28.3788440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3788640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3788719Z return mod(**inputs) 2025-08-14T21:54:28.3788943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3789023Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3789284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3789364Z outputs = self.model.decoder( 2025-08-14T21:54:28.3789595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3789668Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3789918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3789999Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3790228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3790311Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3790569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3790651Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3790656Z 2025-08-14T21:54:28.3790768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3790980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3791048Z return mod(**inputs) 2025-08-14T21:54:28.3791280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3791357Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3791615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3791713Z outputs = self.model.decoder( 2025-08-14T21:54:28.3791936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3792021Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3792269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3792359Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3792596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3792679Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3792937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3793038Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3793042Z 2025-08-14T21:54:28.3793150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3793363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3793431Z return mod(**inputs) 2025-08-14T21:54:28.3793672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3793763Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3794028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3794117Z outputs = self.model.decoder( 2025-08-14T21:54:28.3794343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3794419Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3794676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3794752Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3794992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3795074Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3795322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3795412Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3795416Z 2025-08-14T21:54:28.3795522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3795807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3795893Z return mod(**inputs) 2025-08-14T21:54:28.3796118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3796205Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3796455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3796530Z outputs = self.model.decoder( 2025-08-14T21:54:28.3796771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3796845Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3797082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3797161Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3797377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3797463Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3797696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3797818Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3798063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3798179Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3798183Z 2025-08-14T21:54:28.3798289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3798509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3798575Z return mod(**inputs) 2025-08-14T21:54:28.3798794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3798867Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3799099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3799180Z outputs = self.model.decoder( 2025-08-14T21:54:28.3799393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3799472Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3799724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3799795Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3800031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3800110Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3800347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3800450Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3800688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3800777Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3800780Z 2025-08-14T21:54:28.3800880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3801078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3801151Z return mod(**inputs) 2025-08-14T21:54:28.3801365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3801447Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3801687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3801760Z outputs = self.model.decoder( 2025-08-14T21:54:28.3801980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3802052Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3802293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3802372Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3802589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3802675Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3802914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3803008Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3803259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3803342Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3803345Z 2025-08-14T21:54:28.3803431Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3803508Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3803603Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3803685Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3803784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3803977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3804049Z return mod(**inputs) 2025-08-14T21:54:28.3804269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3804341Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3804575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3804644Z outputs = self.model.decoder( 2025-08-14T21:54:28.3804855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3804926Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3805154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3805232Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3805455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3805542Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3805788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3805886Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3806130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3806227Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3806516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3806657Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3806661Z 2025-08-14T21:54:28.3806763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3806967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3807035Z return mod(**inputs) 2025-08-14T21:54:28.3807252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3807332Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3807568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3807647Z outputs = self.model.decoder( 2025-08-14T21:54:28.3807862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3807935Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3808180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3808251Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3808469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3808556Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3808943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3809053Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3809288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3809384Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3809683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3809852Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3809855Z 2025-08-14T21:54:28.3809964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3810160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3810259Z return mod(**inputs) 2025-08-14T21:54:28.3810485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3810560Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3810807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3810888Z outputs = self.model.decoder( 2025-08-14T21:54:28.3811098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3811178Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3811411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3811516Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3811744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3811821Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3812086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3812194Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3812428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3812514Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3812518Z 2025-08-14T21:54:28.3812619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3812815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3812887Z return mod(**inputs) 2025-08-14T21:54:28.3813098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3813171Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3813413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3813486Z outputs = self.model.decoder( 2025-08-14T21:54:28.3813706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3813777Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3814019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3814095Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3814305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3814388Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3814616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3814696Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3814701Z 2025-08-14T21:54:28.3814804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3814995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3815059Z return mod(**inputs) 2025-08-14T21:54:28.3815272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3815343Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3815601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3815673Z outputs = self.model.decoder( 2025-08-14T21:54:28.3815887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3815968Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3816224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3816304Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3816524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3816598Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3816835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3816931Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3816935Z 2025-08-14T21:54:28.3817033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3817249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3817316Z return mod(**inputs) 2025-08-14T21:54:28.3817534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3817623Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3817856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3817935Z outputs = self.model.decoder( 2025-08-14T21:54:28.3818147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3818219Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3818468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3818538Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3818767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3818849Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3819105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3819196Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3819200Z 2025-08-14T21:54:28.3819307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3819524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3819592Z return mod(**inputs) 2025-08-14T21:54:28.3819820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3819906Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3820158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3820235Z outputs = self.model.decoder( 2025-08-14T21:54:28.3820471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3820550Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3820810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3820883Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3821113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3821202Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3821452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:54:28.3821614Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:54:28.3821626Z 2025-08-14T21:54:28.3821734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3821939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3822037Z return mod(**inputs) 2025-08-14T21:54:28.3822266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3822344Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3822603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3822679Z outputs = self.model.decoder( 2025-08-14T21:54:28.3822910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3822987Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3823241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3823342Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3823571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3823668Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3823932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3824036Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3824293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:54:28.3824411Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:54:28.3824416Z 2025-08-14T21:54:28.3824522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3824755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3824824Z return mod(**inputs) 2025-08-14T21:54:28.3825050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3825136Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3825389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3825470Z outputs = self.model.decoder( 2025-08-14T21:54:28.3825696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3825772Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3826027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3826102Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3826336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3826418Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3826668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3826777Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3827025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:54:28.3827108Z key_states = self.k_proj(hidden_states) 2025-08-14T21:54:28.3827112Z 2025-08-14T21:54:28.3827224Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3827433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3827538Z return mod(**inputs) 2025-08-14T21:54:28.3827762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3827839Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3828094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3828188Z outputs = self.model.decoder( 2025-08-14T21:54:28.3828422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3828507Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3828814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3828895Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3829123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3829207Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3829484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3829600Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3829872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:54:28.3829976Z value_states = self.v_proj(hidden_states) 2025-08-14T21:54:28.3829980Z 2025-08-14T21:54:28.3830065Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3830157Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3830238Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3830316Z cudagraph partition due to non gpu ops 2025-08-14T21:54:28.3830433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3830648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3830726Z return mod(**inputs) 2025-08-14T21:54:28.3830960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3831040Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3831307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3831380Z outputs = self.model.decoder( 2025-08-14T21:54:28.3831595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3831676Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3831930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3832013Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3832245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3832329Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3832593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3832696Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3832949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3833061Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3833373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:28.3833526Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:28.3833530Z 2025-08-14T21:54:28.3833636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3833860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3833938Z return mod(**inputs) 2025-08-14T21:54:28.3834165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3834250Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3834506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3834616Z outputs = self.model.decoder( 2025-08-14T21:54:28.3834858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3834935Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3835205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3835288Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3835533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3835622Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3835987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3836095Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3836372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:54:28.3836481Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:28.3836808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:28.3836928Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:28.3836933Z 2025-08-14T21:54:28.3837043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3837268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3837339Z return mod(**inputs) 2025-08-14T21:54:28.3837572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3837671Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3837911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3837995Z outputs = self.model.decoder( 2025-08-14T21:54:28.3838209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3838281Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3838526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3838597Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3838815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3838902Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3839136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:54:28.3839241Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:28.3839477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:54:28.3839557Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:28.3839561Z 2025-08-14T21:54:28.3839670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3839867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3839938Z return mod(**inputs) 2025-08-14T21:54:28.3840166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3840239Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3840481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3840555Z outputs = self.model.decoder( 2025-08-14T21:54:28.3840788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3840870Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3841108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3841187Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3841405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3841483Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3841728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:54:28.3841809Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:54:28.3841812Z 2025-08-14T21:54:28.3841940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3842133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3842198Z return mod(**inputs) 2025-08-14T21:54:28.3842427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3842500Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3842736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3842820Z outputs = self.model.decoder( 2025-08-14T21:54:28.3843044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3843131Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3843379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3843457Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3843700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3843788Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3844042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:54:28.3844154Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:54:28.3844157Z 2025-08-14T21:54:28.3844264Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3844482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3844555Z return mod(**inputs) 2025-08-14T21:54:28.3844784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3844870Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3845126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:54:28.3845212Z outputs = self.model.decoder( 2025-08-14T21:54:28.3845443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3845523Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3845783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:54:28.3845859Z layer_outputs = decoder_layer( 2025-08-14T21:54:28.3846094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:28.3846224Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:28.3846472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:54:28.3846564Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:28.3846568Z 2025-08-14T21:54:28.3846672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3846900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3846974Z return mod(**inputs) 2025-08-14T21:54:28.3847200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3847275Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3847532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 841, in forward 2025-08-14T21:54:28.3847631Z logits = self.lm_head(outputs[0]).contiguous() 2025-08-14T21:54:28.3847635Z 2025-08-14T21:54:28.3847747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:28.3847956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:28.3848037Z return mod(**inputs) 2025-08-14T21:54:28.3848279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:54:28.3848373Z output = func(self, *args, **kwargs) 2025-08-14T21:54:28.3848641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 847, in forward 2025-08-14T21:54:28.3848721Z loss = self.loss_function( 2025-08-14T21:54:28.3848973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:54:28.3849153Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:54:28.3849414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:54:28.3849619Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:54:28.3849631Z 2025-08-14T21:54:38.9627625Z Compilation time (from dynamo_timed): 15.999078338 2025-08-14T21:54:39.0193019Z pass 2025-08-14T21:54:39.0193492Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:39.0194345Z TIMING: _recursive_pre_grad_passes:0.00839 _recursive_joint_graph_passes:0.63505 _recursive_post_grad_passes:0.10075 async_compile.wait:0.86864 code_gen:9.24716 inductor_compile:10.48717 backend_compile:13.67746 gc:0.00116 entire_frame_compile:15.99908 total_wall_time:15.99908 2025-08-14T21:54:39.0195317Z STATS: call_* op count: 415 | FakeTensorMode.__torch_dispatch__:12797 | FakeTensor.__torch_dispatch__:4472 | ProxyTorchDispatchMode.__torch_dispatch__:4707 2025-08-14T21:54:39.0196049Z Dynamo produced 1 graphs covering 415 ops with 0 graph breaks (0 unique) 2025-08-14T21:54:44.4191444Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:54:44.4192368Z from pkg_resources import resource_filename 2025-08-14T21:54:45.0086064Z 2025-08-14T21:54:46.4639227Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:54:46.4639683Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:54:46.4649848Z cpu eval PLBartForCausalLM 2025-08-14T21:54:47.1650297Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:47.4675243Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:47.8225064Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:52.8182292Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8182795Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8183080Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8183590Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8183798Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8184014Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8184254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8184677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8185062Z return mod(**inputs) 2025-08-14T21:54:52.8185474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8185894Z outputs = self.model.decoder( 2025-08-14T21:54:52.8186438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8186874Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8188296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8188711Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8189214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8189781Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8190282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:54:52.8191042Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:52.8191379Z 2025-08-14T21:54:52.8191549Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8192123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8192632Z return mod(**inputs) 2025-08-14T21:54:52.8193163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8193849Z outputs = self.model.decoder( 2025-08-14T21:54:52.8194515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8195154Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8195803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8196393Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8197048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8197620Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8198151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:54:52.8198709Z key_states = self.k_proj(current_states) 2025-08-14T21:54:52.8198866Z 2025-08-14T21:54:52.8198983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8199373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8199725Z return mod(**inputs) 2025-08-14T21:54:52.8200111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8200528Z outputs = self.model.decoder( 2025-08-14T21:54:52.8200943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8201432Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8201806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8202191Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8202614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8203081Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8203534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:54:52.8203974Z value_states = self.v_proj(current_states) 2025-08-14T21:54:52.8204124Z 2025-08-14T21:54:52.8204218Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8204440Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8204662Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8204880Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8205121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8205500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8205873Z return mod(**inputs) 2025-08-14T21:54:52.8206254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8206668Z outputs = self.model.decoder( 2025-08-14T21:54:52.8207097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8207514Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8207882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8208269Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8208863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8209324Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8209763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8210184Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8210641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:52.8211124Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:52.8211318Z 2025-08-14T21:54:52.8211455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8211822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8212148Z return mod(**inputs) 2025-08-14T21:54:52.8212514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8212914Z outputs = self.model.decoder( 2025-08-14T21:54:52.8213302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8213685Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8214040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8214409Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8214827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8215272Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8215714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8216185Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8216650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:52.8217103Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:52.8217275Z 2025-08-14T21:54:52.8217379Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8217766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8218081Z return mod(**inputs) 2025-08-14T21:54:52.8218447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8218840Z outputs = self.model.decoder( 2025-08-14T21:54:52.8219223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8219602Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8219952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8220329Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8220774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8221207Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8221671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:54:52.8222097Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:52.8222233Z 2025-08-14T21:54:52.8222338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8222695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8223020Z return mod(**inputs) 2025-08-14T21:54:52.8223390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8223778Z outputs = self.model.decoder( 2025-08-14T21:54:52.8224184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8224643Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8225009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8225391Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8225808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8226271Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8226457Z 2025-08-14T21:54:52.8226566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8226945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8227287Z return mod(**inputs) 2025-08-14T21:54:52.8227675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8228080Z outputs = self.model.decoder( 2025-08-14T21:54:52.8228490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8228899Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8229257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8229636Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8230050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8230570Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8230978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:52.8231338Z return self.act(input) 2025-08-14T21:54:52.8231455Z 2025-08-14T21:54:52.8231575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8231964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8232303Z return mod(**inputs) 2025-08-14T21:54:52.8232690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8233115Z outputs = self.model.decoder( 2025-08-14T21:54:52.8233536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8233952Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8234324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8234712Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8235135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:54:52.8235570Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:52.8236011Z 2025-08-14T21:54:52.8236149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8236559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8236912Z return mod(**inputs) 2025-08-14T21:54:52.8237297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8237724Z outputs = self.model.decoder( 2025-08-14T21:54:52.8238133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8238551Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8238924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8239306Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8239779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8240239Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8240681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:54:52.8241232Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:52.8241463Z 2025-08-14T21:54:52.8241581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8241939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8242303Z return mod(**inputs) 2025-08-14T21:54:52.8242814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8243225Z outputs = self.model.decoder( 2025-08-14T21:54:52.8243620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8244077Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8244460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8244855Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8245294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8245759Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8246239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:54:52.8246671Z key_states = self.k_proj(current_states) 2025-08-14T21:54:52.8246814Z 2025-08-14T21:54:52.8246935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8247305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8247697Z return mod(**inputs) 2025-08-14T21:54:52.8248088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8248512Z outputs = self.model.decoder( 2025-08-14T21:54:52.8248924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8249349Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8249698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8250061Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8250476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8250890Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8251312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:54:52.8251720Z value_states = self.v_proj(current_states) 2025-08-14T21:54:52.8251870Z 2025-08-14T21:54:52.8251951Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8252167Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8252371Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8252582Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8252829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8253207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8253550Z return mod(**inputs) 2025-08-14T21:54:52.8253940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8254358Z outputs = self.model.decoder( 2025-08-14T21:54:52.8254763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8255174Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8255543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8255925Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8256335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8256756Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8257168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8257581Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8258044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:52.8258564Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:52.8258759Z 2025-08-14T21:54:52.8258875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8259248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8259591Z return mod(**inputs) 2025-08-14T21:54:52.8259984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8260416Z outputs = self.model.decoder( 2025-08-14T21:54:52.8260823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8261231Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8261613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8261988Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8262386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8262798Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8263208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8263614Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8264054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:52.8264534Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:52.8264702Z 2025-08-14T21:54:52.8264839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8265198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8265526Z return mod(**inputs) 2025-08-14T21:54:52.8265959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8266341Z outputs = self.model.decoder( 2025-08-14T21:54:52.8266739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8267151Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8267518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8267892Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8268307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8268743Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8269147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:54:52.8269580Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:52.8269732Z 2025-08-14T21:54:52.8269841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8270217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8270550Z return mod(**inputs) 2025-08-14T21:54:52.8270933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8271356Z outputs = self.model.decoder( 2025-08-14T21:54:52.8271757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8272178Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8272543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8272927Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8273345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8273813Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8273997Z 2025-08-14T21:54:52.8274115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8274503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8274866Z return mod(**inputs) 2025-08-14T21:54:52.8275281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8275802Z outputs = self.model.decoder( 2025-08-14T21:54:52.8276228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8276678Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8277065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8277459Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8277907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8278387Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8278811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:52.8279182Z return self.act(input) 2025-08-14T21:54:52.8279313Z 2025-08-14T21:54:52.8279427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8279842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8280194Z return mod(**inputs) 2025-08-14T21:54:52.8280622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8281051Z outputs = self.model.decoder( 2025-08-14T21:54:52.8281467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8281887Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8282256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8282646Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8283073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:54:52.8283507Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:52.8283653Z 2025-08-14T21:54:52.8283759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8284116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8284451Z return mod(**inputs) 2025-08-14T21:54:52.8284854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8285256Z outputs = self.model.decoder( 2025-08-14T21:54:52.8285635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8286012Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8286359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8286715Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8287114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8287526Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8287937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:54:52.8288399Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:52.8288600Z 2025-08-14T21:54:52.8288710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8289058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8289379Z return mod(**inputs) 2025-08-14T21:54:52.8289769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8290155Z outputs = self.model.decoder( 2025-08-14T21:54:52.8290551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8290957Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8291318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8291690Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8292110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8292557Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8292994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:54:52.8293404Z key_states = self.k_proj(current_states) 2025-08-14T21:54:52.8293546Z 2025-08-14T21:54:52.8293648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8294026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8294345Z return mod(**inputs) 2025-08-14T21:54:52.8294731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8295123Z outputs = self.model.decoder( 2025-08-14T21:54:52.8295504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8295889Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8296246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8296638Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8297034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8297448Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8297861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:54:52.8298266Z value_states = self.v_proj(current_states) 2025-08-14T21:54:52.8298410Z 2025-08-14T21:54:52.8298491Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8298706Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8298918Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8299120Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8299357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8299715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8300039Z return mod(**inputs) 2025-08-14T21:54:52.8300397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8300786Z outputs = self.model.decoder( 2025-08-14T21:54:52.8301169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8301551Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8301904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8302263Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8302658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8303066Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8303477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8303916Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8304360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:52.8304833Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:52.8305040Z 2025-08-14T21:54:52.8305146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8305500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8305817Z return mod(**inputs) 2025-08-14T21:54:52.8306183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8306579Z outputs = self.model.decoder( 2025-08-14T21:54:52.8306963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8307346Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8307702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8308068Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8308446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8309075Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8309475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8309876Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8310303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:52.8310758Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:52.8310927Z 2025-08-14T21:54:52.8311030Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8311395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8311700Z return mod(**inputs) 2025-08-14T21:54:52.8312061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8312530Z outputs = self.model.decoder( 2025-08-14T21:54:52.8312935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8313359Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8313730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8314112Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8314519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8314938Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8315352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:54:52.8315834Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:52.8315985Z 2025-08-14T21:54:52.8316100Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8316502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8316887Z return mod(**inputs) 2025-08-14T21:54:52.8317266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8317686Z outputs = self.model.decoder( 2025-08-14T21:54:52.8318170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8318561Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8318900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8319262Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8319687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8320169Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8320339Z 2025-08-14T21:54:52.8320442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8320798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8321124Z return mod(**inputs) 2025-08-14T21:54:52.8321488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8321888Z outputs = self.model.decoder( 2025-08-14T21:54:52.8322303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8322697Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8323033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8323409Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8323804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8324235Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8324643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:52.8325005Z return self.act(input) 2025-08-14T21:54:52.8325117Z 2025-08-14T21:54:52.8325226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8325578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8325908Z return mod(**inputs) 2025-08-14T21:54:52.8326279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8326670Z outputs = self.model.decoder( 2025-08-14T21:54:52.8327049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8327436Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8327781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8328132Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8328528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:54:52.8328929Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:52.8329063Z 2025-08-14T21:54:52.8329172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8329525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8329850Z return mod(**inputs) 2025-08-14T21:54:52.8330274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8330658Z outputs = self.model.decoder( 2025-08-14T21:54:52.8331042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8331429Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8331770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8332149Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8332545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8332968Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8333379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:54:52.8333867Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:52.8334077Z 2025-08-14T21:54:52.8334184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8334540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8334853Z return mod(**inputs) 2025-08-14T21:54:52.8335217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8335610Z outputs = self.model.decoder( 2025-08-14T21:54:52.8335990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8336403Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8336750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8337109Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8337519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8337936Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8338346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:54:52.8338750Z key_states = self.k_proj(current_states) 2025-08-14T21:54:52.8338895Z 2025-08-14T21:54:52.8338996Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8339342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8339661Z return mod(**inputs) 2025-08-14T21:54:52.8340025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8340415Z outputs = self.model.decoder( 2025-08-14T21:54:52.8340787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8341163Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8341488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8341837Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8342220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8342627Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8343022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:54:52.8343418Z value_states = self.v_proj(current_states) 2025-08-14T21:54:52.8343557Z 2025-08-14T21:54:52.8343645Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8343851Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8344057Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8344260Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8344486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8344830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8345143Z return mod(**inputs) 2025-08-14T21:54:52.8345499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8345907Z outputs = self.model.decoder( 2025-08-14T21:54:52.8346296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8346678Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8347011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8347385Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8347764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8348172Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8348572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8348986Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8349452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:52.8349980Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:52.8350175Z 2025-08-14T21:54:52.8350285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8350687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8351032Z return mod(**inputs) 2025-08-14T21:54:52.8351418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8351831Z outputs = self.model.decoder( 2025-08-14T21:54:52.8352236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8352649Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8353011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8353391Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8353809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8354249Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8354687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8355126Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8355596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:52.8356175Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:52.8356352Z 2025-08-14T21:54:52.8356468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8356855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8357207Z return mod(**inputs) 2025-08-14T21:54:52.8357616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8358043Z outputs = self.model.decoder( 2025-08-14T21:54:52.8358482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8358917Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8359287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8359677Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8360102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8360581Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8361011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:54:52.8361441Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:52.8361585Z 2025-08-14T21:54:52.8361768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8362143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8362489Z return mod(**inputs) 2025-08-14T21:54:52.8362876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8363300Z outputs = self.model.decoder( 2025-08-14T21:54:52.8363698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8364107Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8364475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8364846Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8365271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8365695Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8365890Z 2025-08-14T21:54:52.8365995Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8366324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8366633Z return mod(**inputs) 2025-08-14T21:54:52.8366976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8367358Z outputs = self.model.decoder( 2025-08-14T21:54:52.8367727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8368103Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8368444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8368787Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8369175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8369588Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8369953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:52.8370267Z return self.act(input) 2025-08-14T21:54:52.8370380Z 2025-08-14T21:54:52.8370477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8370816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8371117Z return mod(**inputs) 2025-08-14T21:54:52.8371466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8371833Z outputs = self.model.decoder( 2025-08-14T21:54:52.8372196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8372560Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8372887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8373224Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8373592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:54:52.8373991Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:52.8374127Z 2025-08-14T21:54:52.8374223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8374560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8374865Z return mod(**inputs) 2025-08-14T21:54:52.8375215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8375605Z outputs = self.model.decoder( 2025-08-14T21:54:52.8375971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8376333Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8376664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8377021Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8377399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8377806Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8378228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:54:52.8378694Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:52.8378891Z 2025-08-14T21:54:52.8379003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8379341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8379645Z return mod(**inputs) 2025-08-14T21:54:52.8379998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8380377Z outputs = self.model.decoder( 2025-08-14T21:54:52.8380740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8381116Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8381447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8381797Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8382179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8382578Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8382968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:54:52.8383354Z key_states = self.k_proj(current_states) 2025-08-14T21:54:52.8383486Z 2025-08-14T21:54:52.8383594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8383943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8384250Z return mod(**inputs) 2025-08-14T21:54:52.8384604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8384984Z outputs = self.model.decoder( 2025-08-14T21:54:52.8385350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8385737Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8386075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8386428Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8386803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8387205Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8387624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:54:52.8388012Z value_states = self.v_proj(current_states) 2025-08-14T21:54:52.8388156Z 2025-08-14T21:54:52.8388237Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8388448Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8388683Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8388878Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8389107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8389455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8389761Z return mod(**inputs) 2025-08-14T21:54:52.8390117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8390499Z outputs = self.model.decoder( 2025-08-14T21:54:52.8390872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8391245Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8391602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8391959Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8392350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8392759Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8393164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8393572Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8393999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:52.8394471Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:52.8394656Z 2025-08-14T21:54:52.8394757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8395108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8395419Z return mod(**inputs) 2025-08-14T21:54:52.8395867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8396275Z outputs = self.model.decoder( 2025-08-14T21:54:52.8396657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8397050Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8397398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8397763Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8398147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8398564Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8398986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8399389Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8399819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:52.8400276Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:52.8400436Z 2025-08-14T21:54:52.8400547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8400897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8401252Z return mod(**inputs) 2025-08-14T21:54:52.8401617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8402214Z outputs = self.model.decoder( 2025-08-14T21:54:52.8402815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8403473Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8404019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8404581Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8405102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8405541Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8405975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:54:52.8406389Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:52.8406541Z 2025-08-14T21:54:52.8406675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8407053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8407400Z return mod(**inputs) 2025-08-14T21:54:52.8407809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8408245Z outputs = self.model.decoder( 2025-08-14T21:54:52.8408925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8409485Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8409852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8410239Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8410648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8411076Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8411258Z 2025-08-14T21:54:52.8411363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8411725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8412050Z return mod(**inputs) 2025-08-14T21:54:52.8412410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8412805Z outputs = self.model.decoder( 2025-08-14T21:54:52.8413209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8413621Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8413968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8414331Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8414723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8415177Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8415583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:52.8415920Z return self.act(input) 2025-08-14T21:54:52.8416029Z 2025-08-14T21:54:52.8416132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8416488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8416888Z return mod(**inputs) 2025-08-14T21:54:52.8417250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8417634Z outputs = self.model.decoder( 2025-08-14T21:54:52.8418015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8418439Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8418797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8419177Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8419593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:54:52.8420010Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:52.8420148Z 2025-08-14T21:54:52.8420249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8420605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8420927Z return mod(**inputs) 2025-08-14T21:54:52.8421321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8421696Z outputs = self.model.decoder( 2025-08-14T21:54:52.8422094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8422642Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8423086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8423575Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8424014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8424509Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8425000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:54:52.8425469Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:52.8425774Z 2025-08-14T21:54:52.8425897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8426289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8426693Z return mod(**inputs) 2025-08-14T21:54:52.8427062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8427543Z outputs = self.model.decoder( 2025-08-14T21:54:52.8427958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8428354Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8428757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8429116Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8429590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8430006Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8430423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:54:52.8430817Z key_states = self.k_proj(current_states) 2025-08-14T21:54:52.8430959Z 2025-08-14T21:54:52.8431067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8431448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8431822Z return mod(**inputs) 2025-08-14T21:54:52.8432202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8432625Z outputs = self.model.decoder( 2025-08-14T21:54:52.8433032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8433466Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8433826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8434209Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8434623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8435054Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8435494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:54:52.8436031Z value_states = self.v_proj(current_states) 2025-08-14T21:54:52.8436183Z 2025-08-14T21:54:52.8436280Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8436503Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8436759Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8436991Z cudagraph partition due to non gpu ops 2025-08-14T21:54:52.8437242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8437652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8437998Z return mod(**inputs) 2025-08-14T21:54:52.8438387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8438803Z outputs = self.model.decoder( 2025-08-14T21:54:52.8439209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8439625Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8439986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8440379Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8440796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8441239Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8441666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8442107Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8442579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:52.8443086Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:52.8443282Z 2025-08-14T21:54:52.8443391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8443772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8444118Z return mod(**inputs) 2025-08-14T21:54:52.8444469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8444855Z outputs = self.model.decoder( 2025-08-14T21:54:52.8445239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8445624Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8445963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8446321Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8446769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8447268Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8447681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:54:52.8448104Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:52.8448540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:52.8448992Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:52.8449172Z 2025-08-14T21:54:52.8449281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8449664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8450005Z return mod(**inputs) 2025-08-14T21:54:52.8450388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8450773Z outputs = self.model.decoder( 2025-08-14T21:54:52.8451160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8451531Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8451895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8452250Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8452632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:54:52.8453028Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:52.8453429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:54:52.8453816Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:52.8453948Z 2025-08-14T21:54:52.8454054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8454395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8454710Z return mod(**inputs) 2025-08-14T21:54:52.8455067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8455447Z outputs = self.model.decoder( 2025-08-14T21:54:52.8455830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8456215Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8456559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8456912Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8457307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8457731Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8457897Z 2025-08-14T21:54:52.8457998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8458344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8458661Z return mod(**inputs) 2025-08-14T21:54:52.8459025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8459409Z outputs = self.model.decoder( 2025-08-14T21:54:52.8459788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8460179Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8460544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8460899Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8461290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:54:52.8461720Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:52.8462122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:52.8462461Z return self.act(input) 2025-08-14T21:54:52.8462579Z 2025-08-14T21:54:52.8462683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8463036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8463350Z return mod(**inputs) 2025-08-14T21:54:52.8463715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:54:52.8464108Z outputs = self.model.decoder( 2025-08-14T21:54:52.8464483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:54:52.8464888Z layer_outputs = decoder_layer( 2025-08-14T21:54:52.8465235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:52.8465612Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:52.8465998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:54:52.8466394Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:52.8466531Z 2025-08-14T21:54:52.8466642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8466992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8467307Z return mod(**inputs) 2025-08-14T21:54:52.8467674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1694, in forward 2025-08-14T21:54:52.8468074Z logits = self.lm_head(outputs[0]) 2025-08-14T21:54:52.8468203Z 2025-08-14T21:54:52.8468304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:52.8468659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:52.8468975Z return mod(**inputs) 2025-08-14T21:54:52.8469337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1700, in forward 2025-08-14T21:54:52.8469791Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:54:52.8469995Z 2025-08-14T21:55:00.7996854Z Compilation time (from dynamo_timed): 11.443264836 2025-08-14T21:55:00.8298456Z pass 2025-08-14T21:55:00.8303274Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:00.8304156Z TIMING: _recursive_pre_grad_passes:0.00574 _recursive_joint_graph_passes:0.25142 _recursive_post_grad_passes:0.05794 async_compile.wait:0.75418 code_gen:7.34666 inductor_compile:8.35266 backend_compile:10.16923 gc:0.00088 entire_frame_compile:11.44326 total_wall_time:11.44326 2025-08-14T21:55:00.8305157Z STATS: call_* op count: 198 | FakeTensorMode.__torch_dispatch__:7102 | FakeTensor.__torch_dispatch__:2588 | ProxyTorchDispatchMode.__torch_dispatch__:2533 2025-08-14T21:55:00.8305698Z Dynamo produced 1 graphs covering 198 ops with 0 graph breaks (0 unique) 2025-08-14T21:55:06.1300191Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:55:06.1301440Z from pkg_resources import resource_filename 2025-08-14T21:55:06.7131728Z 2025-08-14T21:55:09.2344963Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:55:09.2348101Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:55:09.2354572Z cpu eval PLBartForConditionalGeneration 2025-08-14T21:55:10.5434037Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:11.0866567Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:11.6268404Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:21.4776279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4780938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4783876Z return mod(**inputs) 2025-08-14T21:55:21.4784414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1357, in forward 2025-08-14T21:55:21.4785106Z decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id) 2025-08-14T21:55:21.4785993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1084, in shift_tokens_right 2025-08-14T21:55:21.4786546Z index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1) 2025-08-14T21:55:21.4786786Z 2025-08-14T21:55:21.4786956Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4787184Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4787404Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4787622Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4787837Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4788045Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4788298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4788731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4789072Z return mod(**inputs) 2025-08-14T21:55:21.4789485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4789882Z outputs = self.model( 2025-08-14T21:55:21.4790297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4790724Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4791153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4791577Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4791965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4792380Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4792819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4793262Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4793719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.4794292Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.4794522Z 2025-08-14T21:55:21.4794648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4795043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4795398Z return mod(**inputs) 2025-08-14T21:55:21.4795997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4796508Z outputs = self.model( 2025-08-14T21:55:21.4796927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4797350Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4797742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4798182Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4798539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4798907Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4799297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4799714Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4800128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.4800572Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.4800716Z 2025-08-14T21:55:21.4800832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4802187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4802704Z return mod(**inputs) 2025-08-14T21:55:21.4803210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4803664Z outputs = self.model( 2025-08-14T21:55:21.4804078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4804529Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4804963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4805404Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4805795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4806200Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4806629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4807078Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4807524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.4807981Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.4808174Z 2025-08-14T21:55:21.4808299Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4808641Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4809072Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4809298Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4809564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4810143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4810468Z return mod(**inputs) 2025-08-14T21:55:21.4810844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4811316Z outputs = self.model( 2025-08-14T21:55:21.4811712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4812100Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4812491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4812906Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4813278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4813840Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4814290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4814810Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4815419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.4816053Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.4816720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.4817246Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.4817457Z 2025-08-14T21:55:21.4817569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4818057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4818567Z return mod(**inputs) 2025-08-14T21:55:21.4819150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4819614Z outputs = self.model( 2025-08-14T21:55:21.4820021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4820476Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4820988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4821508Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4821884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4822269Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4822701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4823144Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4823581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.4824025Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.4824701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.4825429Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.4825600Z 2025-08-14T21:55:21.4825719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4826093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4826435Z return mod(**inputs) 2025-08-14T21:55:21.4826824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4827268Z outputs = self.model( 2025-08-14T21:55:21.4827839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4828272Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4828697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4829118Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4829489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4829876Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4830302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4830969Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4831416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.4831866Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.4832096Z 2025-08-14T21:55:21.4832262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4832732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4833239Z return mod(**inputs) 2025-08-14T21:55:21.4833845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4834294Z outputs = self.model( 2025-08-14T21:55:21.4834841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4835332Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4836183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4836627Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4837056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4837441Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4837879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.4838358Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.4838556Z 2025-08-14T21:55:21.4838673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4839067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4839417Z return mod(**inputs) 2025-08-14T21:55:21.4839825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4840253Z outputs = self.model( 2025-08-14T21:55:21.4840651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4841077Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4841504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4841929Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4842298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4842689Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4843116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.4843591Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.4844004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.4844377Z return self.act(input) 2025-08-14T21:55:21.4844497Z 2025-08-14T21:55:21.4844621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4845005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4845364Z return mod(**inputs) 2025-08-14T21:55:21.4845755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4846175Z outputs = self.model( 2025-08-14T21:55:21.4846556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4846976Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4847404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4847820Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4848196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4848587Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4849040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:55:21.4849452Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.4849606Z 2025-08-14T21:55:21.4849720Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4850100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4850441Z return mod(**inputs) 2025-08-14T21:55:21.4850832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4851244Z outputs = self.model( 2025-08-14T21:55:21.4851636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4852081Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4852666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4854260Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4854645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4855027Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4855464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4855894Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4856324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.4856821Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.4857047Z 2025-08-14T21:55:21.4857162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4857547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4857893Z return mod(**inputs) 2025-08-14T21:55:21.4858282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4858693Z outputs = self.model( 2025-08-14T21:55:21.4859073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4859489Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4859898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4860305Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4860665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4861045Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4861462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4861909Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4862711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.4863140Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.4863284Z 2025-08-14T21:55:21.4863404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4863786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4864162Z return mod(**inputs) 2025-08-14T21:55:21.4864551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4864963Z outputs = self.model( 2025-08-14T21:55:21.4865344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4865778Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4866184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4866602Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4866962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4867348Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4867766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4868202Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4868655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.4869086Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.4869238Z 2025-08-14T21:55:21.4869351Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4869572Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4869794Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4870010Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4870244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4870618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4870961Z return mod(**inputs) 2025-08-14T21:55:21.4871349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4871748Z outputs = self.model( 2025-08-14T21:55:21.4872137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4872546Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4872949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4873355Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4873721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4874098Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4874503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4874934Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4875367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.4875972Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.4876471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.4877002Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.4877205Z 2025-08-14T21:55:21.4877325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4877713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4878067Z return mod(**inputs) 2025-08-14T21:55:21.4878487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4878959Z outputs = self.model( 2025-08-14T21:55:21.4879364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4879788Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4880211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4880654Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4881043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4881440Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4881873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4882339Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4882837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.4883297Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.4883807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.4884305Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.4884490Z 2025-08-14T21:55:21.4884627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4885020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4885364Z return mod(**inputs) 2025-08-14T21:55:21.4885855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4886274Z outputs = self.model( 2025-08-14T21:55:21.4886676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4887110Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4887530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4887953Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4888322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4888722Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4889134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4889540Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4889961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.4890393Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.4890545Z 2025-08-14T21:55:21.4890653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4891027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4891365Z return mod(**inputs) 2025-08-14T21:55:21.4891745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4892133Z outputs = self.model( 2025-08-14T21:55:21.4892491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4892898Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4893304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4893723Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4894085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4894508Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4894905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.4895342Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.4895534Z 2025-08-14T21:55:21.4895638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4895998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4896321Z return mod(**inputs) 2025-08-14T21:55:21.4896679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4897067Z outputs = self.model( 2025-08-14T21:55:21.4897433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4897856Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4898258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4898694Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4899064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4899461Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4899878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.4900333Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.4900741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.4901094Z return self.act(input) 2025-08-14T21:55:21.4901220Z 2025-08-14T21:55:21.4901330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4901707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4902046Z return mod(**inputs) 2025-08-14T21:55:21.4902435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4902845Z outputs = self.model( 2025-08-14T21:55:21.4903231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4903635Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4904043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4904453Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4904820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4905192Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4905605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:55:21.4906025Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.4906170Z 2025-08-14T21:55:21.4906282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4906661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4907005Z return mod(**inputs) 2025-08-14T21:55:21.4907394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4907792Z outputs = self.model( 2025-08-14T21:55:21.4908177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4908639Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4909275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4909693Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4910068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4910534Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4910947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4911383Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4911814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.4912312Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.4912534Z 2025-08-14T21:55:21.4912647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4913032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4913382Z return mod(**inputs) 2025-08-14T21:55:21.4913795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4914207Z outputs = self.model( 2025-08-14T21:55:21.4914620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4915039Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4915441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4915946Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4916342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4916745Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4917189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4917642Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4918085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.4918518Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.4918674Z 2025-08-14T21:55:21.4918784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4919167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4919516Z return mod(**inputs) 2025-08-14T21:55:21.4919908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4920330Z outputs = self.model( 2025-08-14T21:55:21.4920727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4921150Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4921569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4921992Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4922370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4922750Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4923178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4923604Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4924009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.4924434Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.4924579Z 2025-08-14T21:55:21.4924663Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4924879Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4925081Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4925307Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4925541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4925897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4926236Z return mod(**inputs) 2025-08-14T21:55:21.4926626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4927013Z outputs = self.model( 2025-08-14T21:55:21.4927374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4927773Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4928171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4928544Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4928884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4929251Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4929634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4930021Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4930416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.4930827Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.4931262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.4931727Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.4931915Z 2025-08-14T21:55:21.4932016Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4932373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4932680Z return mod(**inputs) 2025-08-14T21:55:21.4933039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4933415Z outputs = self.model( 2025-08-14T21:55:21.4933769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4934144Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4934537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4934919Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4935258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4935609Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4936003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4936408Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4936814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.4937227Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.4937670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.4938138Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.4938292Z 2025-08-14T21:55:21.4938394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4938749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4939088Z return mod(**inputs) 2025-08-14T21:55:21.4939447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4939815Z outputs = self.model( 2025-08-14T21:55:21.4940210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4940596Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4940972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4941365Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4941762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4942275Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4942665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4943079Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4943542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.4943964Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.4944123Z 2025-08-14T21:55:21.4944229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4944588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4944932Z return mod(**inputs) 2025-08-14T21:55:21.4945309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4945713Z outputs = self.model( 2025-08-14T21:55:21.4946116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4946538Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4946931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4947317Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4947666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4948020Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4948411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.4948854Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.4949025Z 2025-08-14T21:55:21.4949138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4949492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4949816Z return mod(**inputs) 2025-08-14T21:55:21.4950184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4950557Z outputs = self.model( 2025-08-14T21:55:21.4950925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4951316Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4951701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4952105Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4952450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4952814Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4953210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.4953685Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.4954096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.4954460Z return self.act(input) 2025-08-14T21:55:21.4954575Z 2025-08-14T21:55:21.4954683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4955062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4955401Z return mod(**inputs) 2025-08-14T21:55:21.4955915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4956340Z outputs = self.model( 2025-08-14T21:55:21.4956797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4957233Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4957673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4958085Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4958433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4958796Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4959198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:55:21.4959624Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.4959770Z 2025-08-14T21:55:21.4959886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4960271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4960612Z return mod(**inputs) 2025-08-14T21:55:21.4960999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4961414Z outputs = self.model( 2025-08-14T21:55:21.4961797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4962211Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4962621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4963033Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4963397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4963780Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4964198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4964628Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4965064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.4965559Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.4965775Z 2025-08-14T21:55:21.4965893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4966273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4966622Z return mod(**inputs) 2025-08-14T21:55:21.4967040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4967452Z outputs = self.model( 2025-08-14T21:55:21.4967836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4968253Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4968692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4969098Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4969466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4969857Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4970269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4970719Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4971128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.4971553Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.4971692Z 2025-08-14T21:55:21.4971794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4972173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4972507Z return mod(**inputs) 2025-08-14T21:55:21.4972878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4973271Z outputs = self.model( 2025-08-14T21:55:21.4973663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4974094Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4974506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4974907Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4975261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4975631Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4976026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4976444Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4976885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.4977328Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.4977482Z 2025-08-14T21:55:21.4977572Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4977807Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4978038Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4978254Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.4978508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.4978894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.4979247Z return mod(**inputs) 2025-08-14T21:55:21.4979648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.4980038Z outputs = self.model( 2025-08-14T21:55:21.4980412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.4980801Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.4981193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.4981612Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.4981958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.4982315Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.4982707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.4983137Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.4983537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.4983950Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.4984394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5011691Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5011938Z 2025-08-14T21:55:21.5012070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5012446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5012936Z return mod(**inputs) 2025-08-14T21:55:21.5013359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5013779Z outputs = self.model( 2025-08-14T21:55:21.5014187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5014582Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5014971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5015358Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5015698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5016059Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5016455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5016852Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5017258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5017677Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5018130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5018586Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5018760Z 2025-08-14T21:55:21.5018872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5019245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5019569Z return mod(**inputs) 2025-08-14T21:55:21.5019941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5020326Z outputs = self.model( 2025-08-14T21:55:21.5020705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5021085Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5021463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5021849Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5022196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5022554Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5022989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5023399Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5023800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5024240Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5024389Z 2025-08-14T21:55:21.5024499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5024869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5025191Z return mod(**inputs) 2025-08-14T21:55:21.5025563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5025952Z outputs = self.model( 2025-08-14T21:55:21.5026319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5026710Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5027117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5027512Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5027877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5028239Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5028630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.5029066Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5029241Z 2025-08-14T21:55:21.5029349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5029710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5030034Z return mod(**inputs) 2025-08-14T21:55:21.5030394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5030782Z outputs = self.model( 2025-08-14T21:55:21.5031151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5031543Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5031918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5032308Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5032657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5033007Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5033409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.5033874Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5034276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.5034613Z return self.act(input) 2025-08-14T21:55:21.5034731Z 2025-08-14T21:55:21.5034840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5035210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5035550Z return mod(**inputs) 2025-08-14T21:55:21.5036025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5036452Z outputs = self.model( 2025-08-14T21:55:21.5036856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5037300Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5037697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5038119Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5038496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5038902Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5039340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:55:21.5039793Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.5039945Z 2025-08-14T21:55:21.5040060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5040460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5040824Z return mod(**inputs) 2025-08-14T21:55:21.5041233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5041675Z outputs = self.model( 2025-08-14T21:55:21.5042075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5042499Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5042944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5043371Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5043757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5044158Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5044579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5045051Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5045494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5045989Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5046195Z 2025-08-14T21:55:21.5046305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5046667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5046991Z return mod(**inputs) 2025-08-14T21:55:21.5047351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5047738Z outputs = self.model( 2025-08-14T21:55:21.5048105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5048507Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5048889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5049279Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5049627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5050021Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5050441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5050851Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5051257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5051649Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5051817Z 2025-08-14T21:55:21.5051920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5052270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5052592Z return mod(**inputs) 2025-08-14T21:55:21.5052943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5053351Z outputs = self.model( 2025-08-14T21:55:21.5053740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5054145Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5054557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5054977Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5055327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5055685Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5056086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5056493Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5056919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5057303Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5057446Z 2025-08-14T21:55:21.5057531Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5057740Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5057938Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5058132Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5058358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5058709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5059023Z return mod(**inputs) 2025-08-14T21:55:21.5059375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5059743Z outputs = self.model( 2025-08-14T21:55:21.5060097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5060462Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5060830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5061198Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5061528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5061863Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5062235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5062621Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5063004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5063394Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5063824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5064282Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5064459Z 2025-08-14T21:55:21.5064558Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5064901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5065240Z return mod(**inputs) 2025-08-14T21:55:21.5065585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5065950Z outputs = self.model( 2025-08-14T21:55:21.5066313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5066712Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5067089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5067464Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5067792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5068131Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5068504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5068902Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5069293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5069712Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5070148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5070611Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5070775Z 2025-08-14T21:55:21.5070884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5071257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5071582Z return mod(**inputs) 2025-08-14T21:55:21.5071961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5072350Z outputs = self.model( 2025-08-14T21:55:21.5072701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5073081Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5073475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5073876Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5074228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5074597Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5074999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5075417Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5075931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5076361Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5076508Z 2025-08-14T21:55:21.5076628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5077015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5077352Z return mod(**inputs) 2025-08-14T21:55:21.5077735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5078114Z outputs = self.model( 2025-08-14T21:55:21.5078468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5078854Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5079233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5079627Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5079968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5080318Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5080700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.5081138Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5081311Z 2025-08-14T21:55:21.5081414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5081763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5082076Z return mod(**inputs) 2025-08-14T21:55:21.5082439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5082824Z outputs = self.model( 2025-08-14T21:55:21.5083182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5083601Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5083987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5084362Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5084712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5085054Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5085425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.5085845Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5086222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.5086554Z return self.act(input) 2025-08-14T21:55:21.5086660Z 2025-08-14T21:55:21.5086765Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5087118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5087429Z return mod(**inputs) 2025-08-14T21:55:21.5087788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5088154Z outputs = self.model( 2025-08-14T21:55:21.5088498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5088876Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5089253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5089628Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5089956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5090313Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5090710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:55:21.5091102Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.5091250Z 2025-08-14T21:55:21.5091354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5091715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5092038Z return mod(**inputs) 2025-08-14T21:55:21.5092388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5092781Z outputs = self.model( 2025-08-14T21:55:21.5093137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5093514Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5093883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5094280Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5094619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5094964Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5095348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5095748Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5096141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5096605Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5096823Z 2025-08-14T21:55:21.5096925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5097290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5097604Z return mod(**inputs) 2025-08-14T21:55:21.5097962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5098335Z outputs = self.model( 2025-08-14T21:55:21.5098691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5099062Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5099433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5099810Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5100146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5100486Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5100867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5101267Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5101653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5102036Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5102175Z 2025-08-14T21:55:21.5102277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5102622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5102931Z return mod(**inputs) 2025-08-14T21:55:21.5103286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5103665Z outputs = self.model( 2025-08-14T21:55:21.5104005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5104383Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5104759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5105136Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5105468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5105816Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5106210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5106632Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5107025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5107430Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5107569Z 2025-08-14T21:55:21.5107681Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5107897Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5108104Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5108308Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5108539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5109122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5109448Z return mod(**inputs) 2025-08-14T21:55:21.5109815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5110204Z outputs = self.model( 2025-08-14T21:55:21.5110582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5111045Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5111431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5111847Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5112200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5112565Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5112954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5113392Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5113826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5114242Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5114689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5115172Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5115358Z 2025-08-14T21:55:21.5115470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5115895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5116239Z return mod(**inputs) 2025-08-14T21:55:21.5116625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5117048Z outputs = self.model( 2025-08-14T21:55:21.5117431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5117883Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5118270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5118663Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5119005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5119370Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5119770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5120179Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5120593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5121045Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5121488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5121940Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5122113Z 2025-08-14T21:55:21.5122218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5122615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5122941Z return mod(**inputs) 2025-08-14T21:55:21.5123299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5123686Z outputs = self.model( 2025-08-14T21:55:21.5124051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5124437Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5124826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5125217Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5125583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5125942Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5126361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:55:21.5126803Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:55:21.5127224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5127618Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5127770Z 2025-08-14T21:55:21.5127881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5128266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5128600Z return mod(**inputs) 2025-08-14T21:55:21.5128970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5129354Z outputs = self.model( 2025-08-14T21:55:21.5129723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5130106Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5130492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5130881Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5131229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5131586Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5131981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.5132420Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5132592Z 2025-08-14T21:55:21.5132696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5133059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5133382Z return mod(**inputs) 2025-08-14T21:55:21.5133777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5134154Z outputs = self.model( 2025-08-14T21:55:21.5134522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5134931Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5135307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5135733Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5136071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5136444Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5136831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:55:21.5137276Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5137669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.5138009Z return self.act(input) 2025-08-14T21:55:21.5138118Z 2025-08-14T21:55:21.5138231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5138585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5138905Z return mod(**inputs) 2025-08-14T21:55:21.5139281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5139668Z outputs = self.model( 2025-08-14T21:55:21.5140051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:55:21.5140438Z encoder_outputs = self.encoder( 2025-08-14T21:55:21.5140812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:55:21.5141275Z layer_outputs = encoder_layer( 2025-08-14T21:55:21.5141609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5141952Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5142340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:55:21.5142728Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.5142863Z 2025-08-14T21:55:21.5142973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5143317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5143641Z return mod(**inputs) 2025-08-14T21:55:21.5144010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5144386Z outputs = self.model( 2025-08-14T21:55:21.5144750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5145149Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5145521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5145895Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5146235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5146586Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5146979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5147391Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5147800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5148266Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5148461Z 2025-08-14T21:55:21.5148561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5148937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5149258Z return mod(**inputs) 2025-08-14T21:55:21.5149618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5150004Z outputs = self.model( 2025-08-14T21:55:21.5150378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5150756Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5151137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5151507Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5151844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5152195Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5152588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5152998Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5153425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5153847Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5153987Z 2025-08-14T21:55:21.5154116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5154493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5154833Z return mod(**inputs) 2025-08-14T21:55:21.5155216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5155621Z outputs = self.model( 2025-08-14T21:55:21.5156100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5156526Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5156939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5157347Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5157718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5158109Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5158491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5158906Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5159320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5159724Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5159864Z 2025-08-14T21:55:21.5159946Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5160160Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5160368Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5160567Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5160801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5161160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5161472Z return mod(**inputs) 2025-08-14T21:55:21.5161836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5162216Z outputs = self.model( 2025-08-14T21:55:21.5162579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5162987Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5163375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5163767Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5164118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5164491Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5164888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5165306Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5165712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5166127Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5166579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5167060Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5167245Z 2025-08-14T21:55:21.5167380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5167742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5168064Z return mod(**inputs) 2025-08-14T21:55:21.5168436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5168824Z outputs = self.model( 2025-08-14T21:55:21.5169191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5169592Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5169960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5170340Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5170684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5171034Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5171409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5171814Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5172213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5172608Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5173046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5173505Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5173667Z 2025-08-14T21:55:21.5173780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5174132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5174459Z return mod(**inputs) 2025-08-14T21:55:21.5174827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5175209Z outputs = self.model( 2025-08-14T21:55:21.5175569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5175958Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5176342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5176722Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5177091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5177452Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5177864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5178312Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5178746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5179169Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5179312Z 2025-08-14T21:55:21.5179429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5179788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5180106Z return mod(**inputs) 2025-08-14T21:55:21.5180471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5180854Z outputs = self.model( 2025-08-14T21:55:21.5181242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5181633Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5182033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5182416Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5182763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5183120Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5183523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5183974Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5184395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5184861Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5185065Z 2025-08-14T21:55:21.5185172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5185533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5185878Z return mod(**inputs) 2025-08-14T21:55:21.5186270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5186668Z outputs = self.model( 2025-08-14T21:55:21.5187051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5187443Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5187819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5188208Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5188555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5188914Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5189299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5189721Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5190154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5190569Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5190718Z 2025-08-14T21:55:21.5190851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5191230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5191571Z return mod(**inputs) 2025-08-14T21:55:21.5191955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5192377Z outputs = self.model( 2025-08-14T21:55:21.5192762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5193172Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5193580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5194000Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5194374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5194757Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5195178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5195657Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5196196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5196656Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5196823Z 2025-08-14T21:55:21.5196913Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5197144Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5197349Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5197559Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5197798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5198166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5198487Z return mod(**inputs) 2025-08-14T21:55:21.5198857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5199245Z outputs = self.model( 2025-08-14T21:55:21.5199610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5200007Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5200398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5200789Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5201133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5201513Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5201928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5202374Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5202823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5203261Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5203736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5204235Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5204441Z 2025-08-14T21:55:21.5204551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5204933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5205276Z return mod(**inputs) 2025-08-14T21:55:21.5205679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5206089Z outputs = self.model( 2025-08-14T21:55:21.5206480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5206883Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5207310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5207724Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5208094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5208467Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5209106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5209589Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5210051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5210203Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5210519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5210664Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5210670Z 2025-08-14T21:55:21.5210780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5210987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5211064Z return mod(**inputs) 2025-08-14T21:55:21.5211337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5211418Z outputs = self.model( 2025-08-14T21:55:21.5211700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5211776Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5212054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5212133Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5212383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5212470Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5212748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5212868Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5213145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5213235Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5213239Z 2025-08-14T21:55:21.5213356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5213566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5213642Z return mod(**inputs) 2025-08-14T21:55:21.5213917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5213989Z outputs = self.model( 2025-08-14T21:55:21.5214272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5214350Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5214630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5214737Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5214981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5215074Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5215345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5215521Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5215525Z 2025-08-14T21:55:21.5215642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5215851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5215927Z return mod(**inputs) 2025-08-14T21:55:21.5216205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5216278Z outputs = self.model( 2025-08-14T21:55:21.5216550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5216626Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5216923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5217017Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5217256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5217346Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5217599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5217714Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5217933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.5218006Z return self.act(input) 2025-08-14T21:55:21.5218010Z 2025-08-14T21:55:21.5218123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5218331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5218402Z return mod(**inputs) 2025-08-14T21:55:21.5218695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5218765Z outputs = self.model( 2025-08-14T21:55:21.5219038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5219122Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5219389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5219475Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5219706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5219784Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5220046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:55:21.5220131Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.5220135Z 2025-08-14T21:55:21.5220245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5220459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5220529Z return mod(**inputs) 2025-08-14T21:55:21.5220814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5220884Z outputs = self.model( 2025-08-14T21:55:21.5221173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5221257Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5221528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5221604Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5221842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5221919Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5222180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5222280Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5222536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5222702Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5222706Z 2025-08-14T21:55:21.5222812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5223048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5223118Z return mod(**inputs) 2025-08-14T21:55:21.5223408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5223487Z outputs = self.model( 2025-08-14T21:55:21.5223757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5223841Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5224112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5224191Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5224428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5224512Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5224781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5224895Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5225172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5225261Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5225265Z 2025-08-14T21:55:21.5225366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5225563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5225636Z return mod(**inputs) 2025-08-14T21:55:21.5225900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5225975Z outputs = self.model( 2025-08-14T21:55:21.5226235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5226313Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5226593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5226667Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5226896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5226985Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5227255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5227388Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5227662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5227759Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5227763Z 2025-08-14T21:55:21.5227859Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5227971Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5228052Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5228138Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5228244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5228460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5228528Z return mod(**inputs) 2025-08-14T21:55:21.5228816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5228890Z outputs = self.model( 2025-08-14T21:55:21.5229151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5229242Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5229503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5229576Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5229815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5229894Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5230155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5230263Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5230530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5230642Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5230947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5231087Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5231093Z 2025-08-14T21:55:21.5231207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5231413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5231486Z return mod(**inputs) 2025-08-14T21:55:21.5231768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5231842Z outputs = self.model( 2025-08-14T21:55:21.5232120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5232201Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5232476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5232566Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5232809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5232898Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5233184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5233293Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5233582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5233709Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5234020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5234148Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5234152Z 2025-08-14T21:55:21.5234260Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5234497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5234569Z return mod(**inputs) 2025-08-14T21:55:21.5234843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5234922Z outputs = self.model( 2025-08-14T21:55:21.5235200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5235280Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5235553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5235632Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5235974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5236072Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5236371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5236487Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5236761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5236861Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5236865Z 2025-08-14T21:55:21.5236975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5237190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5237274Z return mod(**inputs) 2025-08-14T21:55:21.5237553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5237627Z outputs = self.model( 2025-08-14T21:55:21.5237914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5237993Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5238278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5238354Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5238592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5238685Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5238959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5239084Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5239358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5239528Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5239532Z 2025-08-14T21:55:21.5239649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5239860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5239932Z return mod(**inputs) 2025-08-14T21:55:21.5240212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5240320Z outputs = self.model( 2025-08-14T21:55:21.5240603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5240681Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5240956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5241063Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5241302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5241395Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5241673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5241789Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5242074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5242162Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5242166Z 2025-08-14T21:55:21.5242276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5242511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5242586Z return mod(**inputs) 2025-08-14T21:55:21.5242890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5242963Z outputs = self.model( 2025-08-14T21:55:21.5243239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5243325Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5243603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5243680Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5243923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5244009Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5244291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5244419Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5244682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5244780Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5244784Z 2025-08-14T21:55:21.5244868Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5244956Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5245035Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5245114Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5245225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5245434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5245504Z return mod(**inputs) 2025-08-14T21:55:21.5245781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5245853Z outputs = self.model( 2025-08-14T21:55:21.5246115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5246188Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5246440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5246520Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5246770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5246853Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5247127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5247238Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5247534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5247636Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5247942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5248091Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5248095Z 2025-08-14T21:55:21.5248201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5248417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5248485Z return mod(**inputs) 2025-08-14T21:55:21.5248776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5248856Z outputs = self.model( 2025-08-14T21:55:21.5249161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5249239Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5249504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5249576Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5249806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5249885Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5250141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5250254Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5250510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5250616Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5250909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5251023Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5251027Z 2025-08-14T21:55:21.5251132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5251324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5251390Z return mod(**inputs) 2025-08-14T21:55:21.5251656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5251724Z outputs = self.model( 2025-08-14T21:55:21.5251989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5252065Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5252321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5252398Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5252618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5252696Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5252961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5253085Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5253342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5253425Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5253447Z 2025-08-14T21:55:21.5253554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5253771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5253841Z return mod(**inputs) 2025-08-14T21:55:21.5254130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5254197Z outputs = self.model( 2025-08-14T21:55:21.5254454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5254535Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5254791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5254879Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5255107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5255191Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5255479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5255595Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5255599Z 2025-08-14T21:55:21.5255696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5255894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5255958Z return mod(**inputs) 2025-08-14T21:55:21.5256219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5256286Z outputs = self.model( 2025-08-14T21:55:21.5256542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5256622Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5256876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5256945Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5257171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5257248Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5257505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5257624Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5257834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.5257911Z return self.act(input) 2025-08-14T21:55:21.5257915Z 2025-08-14T21:55:21.5258018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5258222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5258286Z return mod(**inputs) 2025-08-14T21:55:21.5258538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5258614Z outputs = self.model( 2025-08-14T21:55:21.5258865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5258954Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5259213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5259283Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5259505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5259600Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5259862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:55:21.5259948Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.5259952Z 2025-08-14T21:55:21.5260050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5260237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5260306Z return mod(**inputs) 2025-08-14T21:55:21.5260553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5260627Z outputs = self.model( 2025-08-14T21:55:21.5260894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5260966Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5261252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5261325Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5261550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5261626Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5261880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5261988Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5262238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5262389Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5262399Z 2025-08-14T21:55:21.5262503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5262699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5262772Z return mod(**inputs) 2025-08-14T21:55:21.5263028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5263095Z outputs = self.model( 2025-08-14T21:55:21.5263358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5263435Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5263708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5263784Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5264014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5264105Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5264375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5264479Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5264755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5264839Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5264843Z 2025-08-14T21:55:21.5264982Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5265186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5265255Z return mod(**inputs) 2025-08-14T21:55:21.5265531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5265619Z outputs = self.model( 2025-08-14T21:55:21.5265875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5265956Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5266211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5266291Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5266511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5266591Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5266858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5266976Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5267235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5267338Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5267341Z 2025-08-14T21:55:21.5267423Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5267507Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5267582Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5267654Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5267762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5267954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5268026Z return mod(**inputs) 2025-08-14T21:55:21.5268278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5268346Z outputs = self.model( 2025-08-14T21:55:21.5268604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5268681Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5268935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5269014Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5269232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5269315Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5269567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5269667Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5269927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5270024Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5270317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5270451Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5270454Z 2025-08-14T21:55:21.5270556Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5270757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5270823Z return mod(**inputs) 2025-08-14T21:55:21.5271099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5271174Z outputs = self.model( 2025-08-14T21:55:21.5271428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5271508Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5271779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5271850Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5272074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5272150Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5272403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5272507Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5272781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5272906Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5273208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5273342Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5273346Z 2025-08-14T21:55:21.5273462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5273666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5273742Z return mod(**inputs) 2025-08-14T21:55:21.5274023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5274099Z outputs = self.model( 2025-08-14T21:55:21.5274382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5274457Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5274734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5274816Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5275046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5275135Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5275412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5275512Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5275871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5275968Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5275971Z 2025-08-14T21:55:21.5276093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5276310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5276382Z return mod(**inputs) 2025-08-14T21:55:21.5276676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5276748Z outputs = self.model( 2025-08-14T21:55:21.5277039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5277118Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5277378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5277483Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5277713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5277799Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5278080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5278219Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5278518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5278684Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5278688Z 2025-08-14T21:55:21.5278798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5279021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5279093Z return mod(**inputs) 2025-08-14T21:55:21.5279382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5279463Z outputs = self.model( 2025-08-14T21:55:21.5279770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5279857Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5280165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5280242Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5280487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5280572Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5280859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5280986Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5281266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5281359Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5281365Z 2025-08-14T21:55:21.5281475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5281690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5281769Z return mod(**inputs) 2025-08-14T21:55:21.5282046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5282126Z outputs = self.model( 2025-08-14T21:55:21.5282402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5282485Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5282780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5282859Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5283095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5283189Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5283470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5283589Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5283871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5283969Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5283990Z 2025-08-14T21:55:21.5284076Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5284154Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5284234Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5284310Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5284411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5284629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5284694Z return mod(**inputs) 2025-08-14T21:55:21.5284947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5285022Z outputs = self.model( 2025-08-14T21:55:21.5285284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5285368Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5285636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5285709Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5285973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5286055Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5286344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5286462Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5286736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5286841Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5287127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5287261Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5287265Z 2025-08-14T21:55:21.5287373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5287570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5287638Z return mod(**inputs) 2025-08-14T21:55:21.5287895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5287964Z outputs = self.model( 2025-08-14T21:55:21.5288224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5288296Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5288550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5288629Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5288847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5288929Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5289181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5289286Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5289547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5289645Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5289932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5290042Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5290066Z 2025-08-14T21:55:21.5290166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5290367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5290435Z return mod(**inputs) 2025-08-14T21:55:21.5290689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5290780Z outputs = self.model( 2025-08-14T21:55:21.5291031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5291107Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5291360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5291432Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5291655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5291734Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5292005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5292121Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5292424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5292518Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5292522Z 2025-08-14T21:55:21.5292629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5292833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5292909Z return mod(**inputs) 2025-08-14T21:55:21.5293229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5293337Z outputs = self.model( 2025-08-14T21:55:21.5293765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5293878Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5294314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5294402Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5294717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5294831Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5295111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5295241Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5295249Z 2025-08-14T21:55:21.5295356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5295572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5295648Z return mod(**inputs) 2025-08-14T21:55:21.5295928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5296010Z outputs = self.model( 2025-08-14T21:55:21.5296292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5296369Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5296656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5296731Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5296975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5297107Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5297393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5297527Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5297779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.5297854Z return self.act(input) 2025-08-14T21:55:21.5297858Z 2025-08-14T21:55:21.5297975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5298198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5298267Z return mod(**inputs) 2025-08-14T21:55:21.5298554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5298626Z outputs = self.model( 2025-08-14T21:55:21.5298914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5298989Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5299283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5299368Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5299625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5299718Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5300000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:55:21.5300085Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.5300088Z 2025-08-14T21:55:21.5300204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5300423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5300489Z return mod(**inputs) 2025-08-14T21:55:21.5300776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5300847Z outputs = self.model( 2025-08-14T21:55:21.5301135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5301209Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5301487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5301566Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5301804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5301889Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5302182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5302293Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5302580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5302749Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5302753Z 2025-08-14T21:55:21.5302863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5303084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5303152Z return mod(**inputs) 2025-08-14T21:55:21.5303528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5303643Z outputs = self.model( 2025-08-14T21:55:21.5304070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5304177Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5304627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5304740Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5305074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5305184Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5305459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5305563Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5305952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5306077Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5306082Z 2025-08-14T21:55:21.5306192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5306427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5306500Z return mod(**inputs) 2025-08-14T21:55:21.5306798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5306881Z outputs = self.model( 2025-08-14T21:55:21.5307156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5307234Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5307518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5307597Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5307840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5307926Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5308206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5308336Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5308787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5308948Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5308954Z 2025-08-14T21:55:21.5309061Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5309147Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5309237Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5309323Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5309434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5309657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5309728Z return mod(**inputs) 2025-08-14T21:55:21.5310017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5310094Z outputs = self.model( 2025-08-14T21:55:21.5310386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5310471Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5310743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5310819Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5311114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5311195Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5311471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5311577Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5311885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5311996Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5312305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5312450Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5312463Z 2025-08-14T21:55:21.5312574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5312788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5312867Z return mod(**inputs) 2025-08-14T21:55:21.5314031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5314118Z outputs = self.model( 2025-08-14T21:55:21.5314432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5314513Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5314794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5314869Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5315104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5315199Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5315473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5315580Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5315924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5316043Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5316373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5316488Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5316493Z 2025-08-14T21:55:21.5316601Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5316822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5316896Z return mod(**inputs) 2025-08-14T21:55:21.5317180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5317253Z outputs = self.model( 2025-08-14T21:55:21.5317543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5317633Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5317907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5317982Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5318219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5318301Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5318581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5318711Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5318989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5319086Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5319111Z 2025-08-14T21:55:21.5319224Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5319448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5319519Z return mod(**inputs) 2025-08-14T21:55:21.5319798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5319880Z outputs = self.model( 2025-08-14T21:55:21.5320155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5320235Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5320521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5320615Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5320859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5320945Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5321240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5321368Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5321646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5321816Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5321822Z 2025-08-14T21:55:21.5321933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5322147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5322226Z return mod(**inputs) 2025-08-14T21:55:21.5322503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5322575Z outputs = self.model( 2025-08-14T21:55:21.5322863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5322941Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5323224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5323299Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5323535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5323626Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5323900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5324014Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5324298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5324386Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5324390Z 2025-08-14T21:55:21.5324505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5324716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5324787Z return mod(**inputs) 2025-08-14T21:55:21.5325072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5325164Z outputs = self.model( 2025-08-14T21:55:21.5325450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5325530Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5325809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5325905Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5326126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5326201Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5326471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5326574Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5326829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5326911Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5326915Z 2025-08-14T21:55:21.5327007Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5327092Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5327167Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5327253Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5327361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5327551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5327621Z return mod(**inputs) 2025-08-14T21:55:21.5327868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5327932Z outputs = self.model( 2025-08-14T21:55:21.5328188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5328258Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5328513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5328580Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5328793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5328878Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5329123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5329225Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5329481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5329580Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5329867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5329996Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5330000Z 2025-08-14T21:55:21.5330098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5330297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5330361Z return mod(**inputs) 2025-08-14T21:55:21.5330615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5330679Z outputs = self.model( 2025-08-14T21:55:21.5330927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5331023Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5331268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5331337Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5331553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5331650Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5331908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5332011Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5332254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5332354Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5332631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5332734Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5332744Z 2025-08-14T21:55:21.5332859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5333053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5333126Z return mod(**inputs) 2025-08-14T21:55:21.5333389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5333458Z outputs = self.model( 2025-08-14T21:55:21.5333718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5333790Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5334049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5334120Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5334337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5334425Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5334682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5334787Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5335042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5335124Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5335127Z 2025-08-14T21:55:21.5335235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5335428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5335494Z return mod(**inputs) 2025-08-14T21:55:21.5335756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5335824Z outputs = self.model( 2025-08-14T21:55:21.5336084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5336158Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5336409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5336487Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5336702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5336778Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5337065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5337186Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5337189Z 2025-08-14T21:55:21.5337299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5337496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5337581Z return mod(**inputs) 2025-08-14T21:55:21.5337843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5337910Z outputs = self.model( 2025-08-14T21:55:21.5338170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5338243Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5338506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5338583Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5338809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5338887Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5339147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5339278Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5339497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.5339567Z return self.act(input) 2025-08-14T21:55:21.5339570Z 2025-08-14T21:55:21.5339669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5339869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5339933Z return mod(**inputs) 2025-08-14T21:55:21.5340187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5340261Z outputs = self.model( 2025-08-14T21:55:21.5340513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5340591Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5340846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5340916Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5341138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5341213Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5341486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:55:21.5341567Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.5341571Z 2025-08-14T21:55:21.5341670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5341875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5341941Z return mod(**inputs) 2025-08-14T21:55:21.5342204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5342279Z outputs = self.model( 2025-08-14T21:55:21.5342543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5342625Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5342888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5342980Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5343205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5343288Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5343555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5343707Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5343980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5344137Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5344141Z 2025-08-14T21:55:21.5344244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5344441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5344516Z return mod(**inputs) 2025-08-14T21:55:21.5344769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5344861Z outputs = self.model( 2025-08-14T21:55:21.5345115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5345192Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5345472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5345545Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5345769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5345859Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5346129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5346243Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5346522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5346602Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5346607Z 2025-08-14T21:55:21.5346727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5346925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5346998Z return mod(**inputs) 2025-08-14T21:55:21.5347265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5347335Z outputs = self.model( 2025-08-14T21:55:21.5347615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5347692Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5347961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5348043Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5348275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5348367Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5348638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5348740Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5349025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5349113Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5349134Z 2025-08-14T21:55:21.5349223Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5349302Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5349379Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5349461Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5349562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5349776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5349852Z return mod(**inputs) 2025-08-14T21:55:21.5350106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5350173Z outputs = self.model( 2025-08-14T21:55:21.5350435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5350509Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5350771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5350846Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5351092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5351186Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5351473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5351583Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5351854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5351956Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5352268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5352411Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5352414Z 2025-08-14T21:55:21.5352527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5352733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5352802Z return mod(**inputs) 2025-08-14T21:55:21.5353081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5353151Z outputs = self.model( 2025-08-14T21:55:21.5353421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5353505Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5353773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5353858Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5354087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5354168Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5354445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5354549Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5354820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5354930Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5355242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5355362Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5355386Z 2025-08-14T21:55:21.5355493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5355761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5355859Z return mod(**inputs) 2025-08-14T21:55:21.5356146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5356255Z outputs = self.model( 2025-08-14T21:55:21.5356547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5356625Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5356919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5356996Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5357248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5357342Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5357644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5357763Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5358078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5358165Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5358169Z 2025-08-14T21:55:21.5358284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5358530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5358607Z return mod(**inputs) 2025-08-14T21:55:21.5358887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5358959Z outputs = self.model( 2025-08-14T21:55:21.5359244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5359321Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5359606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5359692Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5359922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5360010Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5360290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5360403Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5360693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5360851Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5360855Z 2025-08-14T21:55:21.5360969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5361175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5361251Z return mod(**inputs) 2025-08-14T21:55:21.5361536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5361607Z outputs = self.model( 2025-08-14T21:55:21.5361885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5361967Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5362245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5362351Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5362588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5362670Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5362970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5363080Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5363360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5363450Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5363453Z 2025-08-14T21:55:21.5363561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5363774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5363844Z return mod(**inputs) 2025-08-14T21:55:21.5364142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5364222Z outputs = self.model( 2025-08-14T21:55:21.5364503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5364606Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5364888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5364963Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5365204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5365286Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5365573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5365690Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5365962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5366061Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5366065Z 2025-08-14T21:55:21.5366148Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5366230Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5366319Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5366397Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5366506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5366719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5366789Z return mod(**inputs) 2025-08-14T21:55:21.5367071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5367141Z outputs = self.model( 2025-08-14T21:55:21.5367439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5367527Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5367796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5367865Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5368089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5368166Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5368428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5368551Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5368805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5368911Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5369199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5369355Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5369359Z 2025-08-14T21:55:21.5369458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5369652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5369724Z return mod(**inputs) 2025-08-14T21:55:21.5369978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5370046Z outputs = self.model( 2025-08-14T21:55:21.5370305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5370393Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5370657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5370744Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5370961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5371045Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5371296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5371405Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5371658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5371754Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5372048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5372154Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5372158Z 2025-08-14T21:55:21.5372265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5372461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5372528Z return mod(**inputs) 2025-08-14T21:55:21.5372791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5372856Z outputs = self.model( 2025-08-14T21:55:21.5373108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5373189Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5373451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5373530Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5373751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5373830Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5374087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5374192Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5374444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5374562Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5374565Z 2025-08-14T21:55:21.5374668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5374873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5374939Z return mod(**inputs) 2025-08-14T21:55:21.5375193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5375289Z outputs = self.model( 2025-08-14T21:55:21.5375542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5375621Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5375875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5375945Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5376170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5376248Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5376523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5376659Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5376665Z 2025-08-14T21:55:21.5376789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5377011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5377078Z return mod(**inputs) 2025-08-14T21:55:21.5377333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5377410Z outputs = self.model( 2025-08-14T21:55:21.5377666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5377741Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5378000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5378069Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5378293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5378372Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5378624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5378746Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5378962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.5379043Z return self.act(input) 2025-08-14T21:55:21.5379048Z 2025-08-14T21:55:21.5379155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5379361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5379437Z return mod(**inputs) 2025-08-14T21:55:21.5379703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5379774Z outputs = self.model( 2025-08-14T21:55:21.5380049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5380125Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5380401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5380475Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5380705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5380824Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5381083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:55:21.5381172Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.5381193Z 2025-08-14T21:55:21.5381295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5381491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5381564Z return mod(**inputs) 2025-08-14T21:55:21.5381818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5381884Z outputs = self.model( 2025-08-14T21:55:21.5382144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5382219Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5382481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5382579Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5382798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5382901Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5383154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5383252Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5383513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5383660Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5383666Z 2025-08-14T21:55:21.5383777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5383984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5384054Z return mod(**inputs) 2025-08-14T21:55:21.5384334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5384407Z outputs = self.model( 2025-08-14T21:55:21.5384681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5384757Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5385026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5385109Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5385338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5385420Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5385695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5385798Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5386076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5386159Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5386162Z 2025-08-14T21:55:21.5386268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5386482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5386550Z return mod(**inputs) 2025-08-14T21:55:21.5386825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5386914Z outputs = self.model( 2025-08-14T21:55:21.5387183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5387268Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5387537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5387630Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5387869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5387949Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5388224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5388326Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5388598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5388698Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5388701Z 2025-08-14T21:55:21.5388800Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5388891Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5388975Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5389071Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5389186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5389390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5389459Z return mod(**inputs) 2025-08-14T21:55:21.5389733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5389804Z outputs = self.model( 2025-08-14T21:55:21.5390073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5390157Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5390426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5390512Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5390742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5390824Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5391097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5391200Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5391473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5391579Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5391878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5392028Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5392033Z 2025-08-14T21:55:21.5392142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5392345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5392422Z return mod(**inputs) 2025-08-14T21:55:21.5392689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5392766Z outputs = self.model( 2025-08-14T21:55:21.5393035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5393131Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5393408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5393485Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5393721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5393821Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5394092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5394201Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5394470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5394570Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5394884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5394999Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5395003Z 2025-08-14T21:55:21.5395136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5395344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5395416Z return mod(**inputs) 2025-08-14T21:55:21.5395955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5396040Z outputs = self.model( 2025-08-14T21:55:21.5396328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5396408Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5396686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5396775Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5397022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5397104Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5397388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:55:21.5397491Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:21.5397771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5397857Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5397861Z 2025-08-14T21:55:21.5397967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5398181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5398251Z return mod(**inputs) 2025-08-14T21:55:21.5398532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5398606Z outputs = self.model( 2025-08-14T21:55:21.5398874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5398960Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5399232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5399306Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5399544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5399626Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5399919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5400031Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5400300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:55:21.5400486Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:21.5400490Z 2025-08-14T21:55:21.5400598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5400810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5400878Z return mod(**inputs) 2025-08-14T21:55:21.5401148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5401225Z outputs = self.model( 2025-08-14T21:55:21.5401506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5401583Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5401895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5401971Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5402224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5402308Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5402582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5402703Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5402980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:55:21.5403069Z key_states = self.k_proj(current_states) 2025-08-14T21:55:21.5403079Z 2025-08-14T21:55:21.5403188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5403402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5403479Z return mod(**inputs) 2025-08-14T21:55:21.5403757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5403830Z outputs = self.model( 2025-08-14T21:55:21.5404116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5404194Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5404478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5404558Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5404798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5404890Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5405167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5405282Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5405570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:55:21.5405664Z value_states = self.v_proj(current_states) 2025-08-14T21:55:21.5405668Z 2025-08-14T21:55:21.5405762Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5405846Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5405930Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5406020Z cudagraph partition due to non gpu ops 2025-08-14T21:55:21.5406152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5406365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5406443Z return mod(**inputs) 2025-08-14T21:55:21.5406722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5406831Z outputs = self.model( 2025-08-14T21:55:21.5407113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5407192Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5407480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5407556Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5407796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5407888Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5408165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5408305Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5408580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5408874Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5409245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:21.5409392Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:21.5409397Z 2025-08-14T21:55:21.5409515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5409728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5409803Z return mod(**inputs) 2025-08-14T21:55:21.5410089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5410165Z outputs = self.model( 2025-08-14T21:55:21.5410444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5410535Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5410813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5410901Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5411138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5411223Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5411509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5411622Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5411904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:55:21.5412008Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:21.5412321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:21.5412444Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:21.5412447Z 2025-08-14T21:55:21.5412556Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5412766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5412844Z return mod(**inputs) 2025-08-14T21:55:21.5413162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5413242Z outputs = self.model( 2025-08-14T21:55:21.5413521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5413601Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5413913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5413991Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5414238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5414314Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5414565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:55:21.5414679Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:55:21.5414934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:55:21.5415036Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:21.5415048Z 2025-08-14T21:55:21.5415150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5415369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5415443Z return mod(**inputs) 2025-08-14T21:55:21.5415701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5415767Z outputs = self.model( 2025-08-14T21:55:21.5416034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5416105Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5416368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5416439Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5416658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5416744Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5417001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5417119Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5417123Z 2025-08-14T21:55:21.5417230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5417425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5417497Z return mod(**inputs) 2025-08-14T21:55:21.5417753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5417821Z outputs = self.model( 2025-08-14T21:55:21.5418084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5418155Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5418414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5418493Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5418712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5418798Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5419081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:55:21.5419219Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:21.5419435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:21.5419503Z return self.act(input) 2025-08-14T21:55:21.5419507Z 2025-08-14T21:55:21.5419615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5419827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5419892Z return mod(**inputs) 2025-08-14T21:55:21.5420151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:55:21.5420218Z outputs = self.model( 2025-08-14T21:55:21.5420470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:55:21.5420549Z decoder_outputs = self.decoder( 2025-08-14T21:55:21.5420804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:55:21.5420882Z layer_outputs = decoder_layer( 2025-08-14T21:55:21.5421113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:21.5421193Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:21.5421482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:55:21.5421567Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:21.5421571Z 2025-08-14T21:55:21.5421678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5421873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5421938Z return mod(**inputs) 2025-08-14T21:55:21.5422216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1377, in forward 2025-08-14T21:55:21.5422299Z lm_logits = self.lm_head(outputs[0]) 2025-08-14T21:55:21.5422302Z 2025-08-14T21:55:21.5422402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:21.5422607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:21.5422672Z return mod(**inputs) 2025-08-14T21:55:21.5422937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1383, in forward 2025-08-14T21:55:21.5423107Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:55:21.5423111Z 2025-08-14T21:55:31.1459822Z Compilation time (from dynamo_timed): 17.767136513 2025-08-14T21:55:31.1712639Z pass 2025-08-14T21:55:31.1713094Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:31.1713974Z TIMING: _recursive_pre_grad_passes:0.00913 _recursive_joint_graph_passes:0.47856 _recursive_post_grad_passes:0.11615 async_compile.wait:0.76674 code_gen:8.91801 inductor_compile:10.57503 backend_compile:14.68636 gc:0.00228 entire_frame_compile:17.76714 total_wall_time:17.76714 2025-08-14T21:55:31.1715151Z STATS: call_* op count: 517 | FakeTensorMode.__torch_dispatch__:17501 | FakeTensor.__torch_dispatch__:6218 | ProxyTorchDispatchMode.__torch_dispatch__:6406 2025-08-14T21:55:31.1715864Z Dynamo produced 1 graphs covering 517 ops with 0 graph breaks (0 unique) 2025-08-14T21:55:36.6718009Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:55:36.6722563Z from pkg_resources import resource_filename 2025-08-14T21:55:37.2617917Z 2025-08-14T21:55:41.1761208Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:55:41.1761565Z loading model: 0it [00:03, ?it/s] 2025-08-14T21:55:41.1770605Z cpu eval PegasusForCausalLM 2025-08-14T21:55:41.5621954Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:41.7155020Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:41.8586385Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:49.7555366Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7555840Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7556134Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7556362Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7556602Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7556869Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7557161Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7557444Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7557665Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7557873Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7558074Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7558608Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7558875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7559341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7559714Z return mod(**inputs) 2025-08-14T21:55:49.7560156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7560605Z outputs = self.model.decoder( 2025-08-14T21:55:49.7561022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7561436Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7561797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7562160Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7562571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7563006Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7563434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7563928Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7564156Z 2025-08-14T21:55:49.7564271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7564660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7565019Z return mod(**inputs) 2025-08-14T21:55:49.7565422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7565851Z outputs = self.model.decoder( 2025-08-14T21:55:49.7566334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7566732Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7567091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7567458Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7567881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7568379Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7568797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7569254Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7569392Z 2025-08-14T21:55:49.7569506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7569889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7570281Z return mod(**inputs) 2025-08-14T21:55:49.7570742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7571158Z outputs = self.model.decoder( 2025-08-14T21:55:49.7571571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7572003Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7572383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7572769Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7573198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7574430Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7574907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7575378Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7575538Z 2025-08-14T21:55:49.7575626Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7575855Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7576071Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7576294Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7576543Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7576917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7577262Z return mod(**inputs) 2025-08-14T21:55:49.7577667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7578097Z outputs = self.model.decoder( 2025-08-14T21:55:49.7578512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7578932Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7579363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7579750Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7580173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7580625Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7581076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7581530Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7582013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7582574Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7582774Z 2025-08-14T21:55:49.7582895Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7583275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7583622Z return mod(**inputs) 2025-08-14T21:55:49.7584024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7584494Z outputs = self.model.decoder( 2025-08-14T21:55:49.7584941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7585366Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7585738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7586121Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7586574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7587017Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7587454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7587893Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7588363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7588852Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7589023Z 2025-08-14T21:55:49.7589132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7589523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7589868Z return mod(**inputs) 2025-08-14T21:55:49.7590285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7590720Z outputs = self.model.decoder( 2025-08-14T21:55:49.7591148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7591590Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7591980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7592377Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7592822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7593290Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7593741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.7594190Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.7594354Z 2025-08-14T21:55:49.7594470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7594866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7595215Z return mod(**inputs) 2025-08-14T21:55:49.7595625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7596379Z outputs = self.model.decoder( 2025-08-14T21:55:49.7596809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7597233Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7597619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7598018Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7598447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7598983Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7599184Z 2025-08-14T21:55:49.7599299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7599688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7600060Z return mod(**inputs) 2025-08-14T21:55:49.7600482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7600922Z outputs = self.model.decoder( 2025-08-14T21:55:49.7601353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7601805Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7602184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7602576Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7603002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7603480Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7603904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.7604275Z return self.act(input) 2025-08-14T21:55:49.7604394Z 2025-08-14T21:55:49.7604507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7604915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7605288Z return mod(**inputs) 2025-08-14T21:55:49.7605702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7606132Z outputs = self.model.decoder( 2025-08-14T21:55:49.7606561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7607005Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7607362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7607744Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7608167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.7608587Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.7608996Z 2025-08-14T21:55:49.7609108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7609497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7609844Z return mod(**inputs) 2025-08-14T21:55:49.7610228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7610656Z outputs = self.model.decoder( 2025-08-14T21:55:49.7611067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7611492Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7611857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7612245Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7612673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7613108Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7613559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7614065Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7614283Z 2025-08-14T21:55:49.7614402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7614776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7615120Z return mod(**inputs) 2025-08-14T21:55:49.7615589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7615988Z outputs = self.model.decoder( 2025-08-14T21:55:49.7616369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7616763Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7617150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7617522Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7617958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7618409Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7618855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7619288Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7619433Z 2025-08-14T21:55:49.7619542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7619932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7620251Z return mod(**inputs) 2025-08-14T21:55:49.7620670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7621092Z outputs = self.model.decoder( 2025-08-14T21:55:49.7621504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7621916Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7622262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7622626Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7623028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7623463Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7623909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7624318Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7624460Z 2025-08-14T21:55:49.7624539Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7624754Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7624965Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7625169Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7625684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7626042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7626370Z return mod(**inputs) 2025-08-14T21:55:49.7626734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7627139Z outputs = self.model.decoder( 2025-08-14T21:55:49.7627533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7627931Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7628286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7628667Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7629083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7629515Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7629950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7630421Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7630893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7631399Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7631625Z 2025-08-14T21:55:49.7631736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7632136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7632503Z return mod(**inputs) 2025-08-14T21:55:49.7632912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7633351Z outputs = self.model.decoder( 2025-08-14T21:55:49.7633780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7634217Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7634619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7635015Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7635456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7636009Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7636479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7636942Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7637425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7637933Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7638118Z 2025-08-14T21:55:49.7638233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7638624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7638968Z return mod(**inputs) 2025-08-14T21:55:49.7639384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7639827Z outputs = self.model.decoder( 2025-08-14T21:55:49.7640259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7640689Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7641068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7641462Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7641861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7642275Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7642706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.7643096Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.7643228Z 2025-08-14T21:55:49.7643330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7643678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7643993Z return mod(**inputs) 2025-08-14T21:55:49.7644363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7644751Z outputs = self.model.decoder( 2025-08-14T21:55:49.7645165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7645566Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7645915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7646265Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7646676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7647127Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7647311Z 2025-08-14T21:55:49.7647419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7647802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7648147Z return mod(**inputs) 2025-08-14T21:55:49.7648543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7648969Z outputs = self.model.decoder( 2025-08-14T21:55:49.7649383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7649784Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7650133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7650490Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7650879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7651310Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7651684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.7652020Z return self.act(input) 2025-08-14T21:55:49.7652131Z 2025-08-14T21:55:49.7652239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7652596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7652914Z return mod(**inputs) 2025-08-14T21:55:49.7653286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7653709Z outputs = self.model.decoder( 2025-08-14T21:55:49.7654094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7654477Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7654818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7655169Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7655552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.7655949Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.7656083Z 2025-08-14T21:55:49.7656195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7656537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7656857Z return mod(**inputs) 2025-08-14T21:55:49.7657230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7657630Z outputs = self.model.decoder( 2025-08-14T21:55:49.7658016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7658413Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7658771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7659150Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7659533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7659950Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7660402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7660876Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7661094Z 2025-08-14T21:55:49.7661199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7661562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7661889Z return mod(**inputs) 2025-08-14T21:55:49.7662262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7662667Z outputs = self.model.decoder( 2025-08-14T21:55:49.7663084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7663492Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7663839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7664243Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7664649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7665068Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7665491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7665902Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7666039Z 2025-08-14T21:55:49.7666153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7666512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7666842Z return mod(**inputs) 2025-08-14T21:55:49.7667226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7667640Z outputs = self.model.decoder( 2025-08-14T21:55:49.7668048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7668468Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7668821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7669190Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7669606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7670046Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7670479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7670905Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7671059Z 2025-08-14T21:55:49.7671145Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7671370Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7671582Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7671799Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7672047Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7672416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7672735Z return mod(**inputs) 2025-08-14T21:55:49.7673128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7673533Z outputs = self.model.decoder( 2025-08-14T21:55:49.7673947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7674397Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7674776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7675163Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7675580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7676138Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7676602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7677072Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7677561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7678077Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7678265Z 2025-08-14T21:55:49.7678375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7678745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7679074Z return mod(**inputs) 2025-08-14T21:55:49.7679448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7679845Z outputs = self.model.decoder( 2025-08-14T21:55:49.7680229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7680624Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7680973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7681329Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7681729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7682155Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7682573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7682985Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7683429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7683886Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7684048Z 2025-08-14T21:55:49.7684160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7684514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7684840Z return mod(**inputs) 2025-08-14T21:55:49.7685216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7685615Z outputs = self.model.decoder( 2025-08-14T21:55:49.7686024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7686504Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7686880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7687262Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7687681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7688097Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7688520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.7688946Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.7689114Z 2025-08-14T21:55:49.7689225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7689602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7689932Z return mod(**inputs) 2025-08-14T21:55:49.7690324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7690753Z outputs = self.model.decoder( 2025-08-14T21:55:49.7691161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7691568Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7691949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7692340Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7692751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7693235Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7693426Z 2025-08-14T21:55:49.7693534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7693908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7694237Z return mod(**inputs) 2025-08-14T21:55:49.7694628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7695048Z outputs = self.model.decoder( 2025-08-14T21:55:49.7695459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7695865Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7696236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7696599Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7696988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7697451Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7697854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.7698210Z return self.act(input) 2025-08-14T21:55:49.7698327Z 2025-08-14T21:55:49.7698436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7698813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7699148Z return mod(**inputs) 2025-08-14T21:55:49.7699507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7699908Z outputs = self.model.decoder( 2025-08-14T21:55:49.7700298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7700695Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7701029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7701388Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7701782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.7702226Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.7702374Z 2025-08-14T21:55:49.7702491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7702850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7703197Z return mod(**inputs) 2025-08-14T21:55:49.7703584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7704007Z outputs = self.model.decoder( 2025-08-14T21:55:49.7704430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7704828Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7705179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7705562Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7705984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:55:49.7706427Z hidden_states = residual + hidden_states 2025-08-14T21:55:49.7706579Z 2025-08-14T21:55:49.7706687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7707086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7707408Z return mod(**inputs) 2025-08-14T21:55:49.7707785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7708205Z outputs = self.model.decoder( 2025-08-14T21:55:49.7708613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7709218Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7709579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7709958Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7710370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7710788Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7711231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7711729Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7711948Z 2025-08-14T21:55:49.7712067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7712434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7712778Z return mod(**inputs) 2025-08-14T21:55:49.7713169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7713586Z outputs = self.model.decoder( 2025-08-14T21:55:49.7713990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7714403Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7714768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7715136Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7715556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7716054Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7716501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7716967Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7717116Z 2025-08-14T21:55:49.7717226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7717600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7717956Z return mod(**inputs) 2025-08-14T21:55:49.7718348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7718766Z outputs = self.model.decoder( 2025-08-14T21:55:49.7719179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7719587Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7719954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7720337Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7720757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7721233Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7721674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7722126Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7722277Z 2025-08-14T21:55:49.7722361Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7722595Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7722826Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7723055Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7723305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7723691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7724041Z return mod(**inputs) 2025-08-14T21:55:49.7724432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7724862Z outputs = self.model.decoder( 2025-08-14T21:55:49.7725304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7725740Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7726117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7726502Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7726927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7727366Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7727811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7728268Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7728741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7729253Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7729458Z 2025-08-14T21:55:49.7729571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7729954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7730304Z return mod(**inputs) 2025-08-14T21:55:49.7730696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7731124Z outputs = self.model.decoder( 2025-08-14T21:55:49.7731555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7731971Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7732337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7732711Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7733150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7733585Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7733997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7734407Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7734840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7735295Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7735471Z 2025-08-14T21:55:49.7735578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7735976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7736317Z return mod(**inputs) 2025-08-14T21:55:49.7736739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7737156Z outputs = self.model.decoder( 2025-08-14T21:55:49.7737575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7737994Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7738361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7738750Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7739163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7739606Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7740041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.7740466Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.7740610Z 2025-08-14T21:55:49.7740718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7741096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7741434Z return mod(**inputs) 2025-08-14T21:55:49.7741830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7742261Z outputs = self.model.decoder( 2025-08-14T21:55:49.7742677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7743101Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7743463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7743859Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7744278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7744739Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7744919Z 2025-08-14T21:55:49.7745027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7745413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7745766Z return mod(**inputs) 2025-08-14T21:55:49.7746183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7746632Z outputs = self.model.decoder( 2025-08-14T21:55:49.7747062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7747496Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7747857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7748240Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7748655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7749118Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7749512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.7749874Z return self.act(input) 2025-08-14T21:55:49.7749991Z 2025-08-14T21:55:49.7750109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7750501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7750846Z return mod(**inputs) 2025-08-14T21:55:49.7751239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7751689Z outputs = self.model.decoder( 2025-08-14T21:55:49.7752097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7752516Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7752899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7753292Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7753735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.7754297Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.7754449Z 2025-08-14T21:55:49.7754575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7754960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7755315Z return mod(**inputs) 2025-08-14T21:55:49.7755868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7756371Z outputs = self.model.decoder( 2025-08-14T21:55:49.7756805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7757238Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7757617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7757997Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7758427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7758882Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7759336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7759838Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7760108Z 2025-08-14T21:55:49.7760225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7760604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7760949Z return mod(**inputs) 2025-08-14T21:55:49.7761377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7761798Z outputs = self.model.decoder( 2025-08-14T21:55:49.7762211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7762621Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7763057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7763446Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7763863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7764299Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7764749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7765186Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7765372Z 2025-08-14T21:55:49.7765500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7765904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7766269Z return mod(**inputs) 2025-08-14T21:55:49.7766667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7767109Z outputs = self.model.decoder( 2025-08-14T21:55:49.7767533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7767957Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7768332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7768714Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7769146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7769600Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7770044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7770482Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7770640Z 2025-08-14T21:55:49.7770729Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7770964Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7771248Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7771475Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7771727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7772109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7772470Z return mod(**inputs) 2025-08-14T21:55:49.7772874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7773361Z outputs = self.model.decoder( 2025-08-14T21:55:49.7773781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7774210Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7774589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7774973Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7775402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7775864Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7776312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7776785Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7777270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7777797Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7778033Z 2025-08-14T21:55:49.7778152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7778534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7778885Z return mod(**inputs) 2025-08-14T21:55:49.7779288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7779726Z outputs = self.model.decoder( 2025-08-14T21:55:49.7780163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7780599Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7780979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7781444Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7781883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7782357Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7782811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7783265Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7783759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7784272Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7784465Z 2025-08-14T21:55:49.7784579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7784987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7785344Z return mod(**inputs) 2025-08-14T21:55:49.7785752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7786193Z outputs = self.model.decoder( 2025-08-14T21:55:49.7786632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7787073Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7787460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7787856Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7788291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7788746Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7789189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.7789629Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.7789783Z 2025-08-14T21:55:49.7789896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7790283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7790631Z return mod(**inputs) 2025-08-14T21:55:49.7791032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7791463Z outputs = self.model.decoder( 2025-08-14T21:55:49.7791902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7792332Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7792708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7793095Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7793538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7794011Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7794196Z 2025-08-14T21:55:49.7794317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7794704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7795044Z return mod(**inputs) 2025-08-14T21:55:49.7795445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7795945Z outputs = self.model.decoder( 2025-08-14T21:55:49.7796397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7796833Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7797218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7797633Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7798069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7798550Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7798978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.7799353Z return self.act(input) 2025-08-14T21:55:49.7799474Z 2025-08-14T21:55:49.7799585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7799978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7800332Z return mod(**inputs) 2025-08-14T21:55:49.7800732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7801182Z outputs = self.model.decoder( 2025-08-14T21:55:49.7801606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7802037Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7802395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7802774Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7803193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.7803612Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.7803766Z 2025-08-14T21:55:49.7803878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7804256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7804603Z return mod(**inputs) 2025-08-14T21:55:49.7804992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7805410Z outputs = self.model.decoder( 2025-08-14T21:55:49.7805820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7806231Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7806595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7806993Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7807409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:55:49.7807823Z hidden_states = residual + hidden_states 2025-08-14T21:55:49.7807973Z 2025-08-14T21:55:49.7808101Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7808474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7808942Z return mod(**inputs) 2025-08-14T21:55:49.7809351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7809777Z outputs = self.model.decoder( 2025-08-14T21:55:49.7810201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7810615Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7810981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7811370Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7811841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7812280Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7812744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7813240Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7813457Z 2025-08-14T21:55:49.7813575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7813946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7814289Z return mod(**inputs) 2025-08-14T21:55:49.7814679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7815099Z outputs = self.model.decoder( 2025-08-14T21:55:49.7815512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7815931Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7816299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7816664Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7817068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7817509Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7817939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7818362Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7818509Z 2025-08-14T21:55:49.7818622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7818996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7819332Z return mod(**inputs) 2025-08-14T21:55:49.7819727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7820144Z outputs = self.model.decoder( 2025-08-14T21:55:49.7820554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7820964Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7821328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7821735Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7822149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7822597Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7823040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7823499Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7823648Z 2025-08-14T21:55:49.7823734Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7823963Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7824185Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7824398Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7824649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7825026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7825372Z return mod(**inputs) 2025-08-14T21:55:49.7825781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7826204Z outputs = self.model.decoder( 2025-08-14T21:55:49.7826618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7827052Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7827422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7827804Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7828224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7828659Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7829102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7829541Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7830013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7830519Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7830725Z 2025-08-14T21:55:49.7830837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7831225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7831574Z return mod(**inputs) 2025-08-14T21:55:49.7831999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7832442Z outputs = self.model.decoder( 2025-08-14T21:55:49.7832870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7833304Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7833683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7834083Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7834516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7834966Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7835423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7835953Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7836442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7836980Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7837166Z 2025-08-14T21:55:49.7837281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7837681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7838040Z return mod(**inputs) 2025-08-14T21:55:49.7838458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7838882Z outputs = self.model.decoder( 2025-08-14T21:55:49.7839362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7839770Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7840135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7840516Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7840926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7841387Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7841834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.7842280Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.7842427Z 2025-08-14T21:55:49.7842541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7842926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7843277Z return mod(**inputs) 2025-08-14T21:55:49.7843667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7844094Z outputs = self.model.decoder( 2025-08-14T21:55:49.7844504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7844925Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7845287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7845674Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7846101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7846569Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7846752Z 2025-08-14T21:55:49.7846864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7847249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7847596Z return mod(**inputs) 2025-08-14T21:55:49.7847987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7848410Z outputs = self.model.decoder( 2025-08-14T21:55:49.7848824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7849245Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7849609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7849992Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7850417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7850875Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7851286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.7851672Z return self.act(input) 2025-08-14T21:55:49.7851787Z 2025-08-14T21:55:49.7851903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7852278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7852617Z return mod(**inputs) 2025-08-14T21:55:49.7853035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7853453Z outputs = self.model.decoder( 2025-08-14T21:55:49.7853874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7854293Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7854655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7855028Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7855449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.7855872Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.7856034Z 2025-08-14T21:55:49.7856155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7856528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7856888Z return mod(**inputs) 2025-08-14T21:55:49.7857282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7857698Z outputs = self.model.decoder( 2025-08-14T21:55:49.7858105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7858518Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7858883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7859257Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7859681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7860126Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7860565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7861051Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7861273Z 2025-08-14T21:55:49.7861384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7861762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7862089Z return mod(**inputs) 2025-08-14T21:55:49.7862484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7862902Z outputs = self.model.decoder( 2025-08-14T21:55:49.7863311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7863713Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7864076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7864454Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7864874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7865306Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7865745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7866189Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7866332Z 2025-08-14T21:55:49.7866441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7866828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7867172Z return mod(**inputs) 2025-08-14T21:55:49.7867582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7867992Z outputs = self.model.decoder( 2025-08-14T21:55:49.7868398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7868811Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7869166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7869544Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7869963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7870521Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7870968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7871411Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7871597Z 2025-08-14T21:55:49.7871687Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7871921Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7872144Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7872377Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7872638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7873028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7873389Z return mod(**inputs) 2025-08-14T21:55:49.7873802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7874253Z outputs = self.model.decoder( 2025-08-14T21:55:49.7874674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7875113Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7875493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7875965Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7876426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7876894Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7877345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7877800Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7878286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7878815Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7879021Z 2025-08-14T21:55:49.7879147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7879533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7879886Z return mod(**inputs) 2025-08-14T21:55:49.7880303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7880740Z outputs = self.model.decoder( 2025-08-14T21:55:49.7881182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7881648Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7882032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7882419Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7882881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7883321Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7883750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7884189Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7884653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7885133Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7885302Z 2025-08-14T21:55:49.7885412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7885809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7886151Z return mod(**inputs) 2025-08-14T21:55:49.7886575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7886989Z outputs = self.model.decoder( 2025-08-14T21:55:49.7887404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7887829Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7888196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7888584Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7889013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7889459Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7889897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.7890334Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.7890484Z 2025-08-14T21:55:49.7890607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7890993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7891346Z return mod(**inputs) 2025-08-14T21:55:49.7891743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7892172Z outputs = self.model.decoder( 2025-08-14T21:55:49.7892584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7893016Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7893390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7893779Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7894201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7894676Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7894861Z 2025-08-14T21:55:49.7894981Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7895360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7895707Z return mod(**inputs) 2025-08-14T21:55:49.7896120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7896550Z outputs = self.model.decoder( 2025-08-14T21:55:49.7896978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7897399Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7897786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7898168Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7898587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7899056Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7899471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.7899832Z return self.act(input) 2025-08-14T21:55:49.7899959Z 2025-08-14T21:55:49.7900071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7900475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7900817Z return mod(**inputs) 2025-08-14T21:55:49.7901202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7901641Z outputs = self.model.decoder( 2025-08-14T21:55:49.7902049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7902464Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7902839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7903236Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7903656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.7904068Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.7904219Z 2025-08-14T21:55:49.7904328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7904706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7905046Z return mod(**inputs) 2025-08-14T21:55:49.7905428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7905845Z outputs = self.model.decoder( 2025-08-14T21:55:49.7906247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7906658Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7907024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7907404Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7907821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:55:49.7908238Z hidden_states = residual + hidden_states 2025-08-14T21:55:49.7908390Z 2025-08-14T21:55:49.7908500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7909029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7909375Z return mod(**inputs) 2025-08-14T21:55:49.7909779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7910210Z outputs = self.model.decoder( 2025-08-14T21:55:49.7910768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7911243Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7911618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7912015Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7912445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7912949Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7913407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7913922Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7914144Z 2025-08-14T21:55:49.7914259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7914653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7915005Z return mod(**inputs) 2025-08-14T21:55:49.7915410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7915937Z outputs = self.model.decoder( 2025-08-14T21:55:49.7916371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7916840Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7917218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7917601Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7918029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7918486Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7918932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7919376Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7919530Z 2025-08-14T21:55:49.7919650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7920043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7920387Z return mod(**inputs) 2025-08-14T21:55:49.7920792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7921223Z outputs = self.model.decoder( 2025-08-14T21:55:49.7921636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7922059Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7922432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7922822Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7923244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7923692Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7924143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7924585Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7924738Z 2025-08-14T21:55:49.7924826Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7925059Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7925287Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7925503Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7925757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7926173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7926530Z return mod(**inputs) 2025-08-14T21:55:49.7926940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7927373Z outputs = self.model.decoder( 2025-08-14T21:55:49.7927824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7928253Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7928641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7929043Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7929486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7933475Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7934021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7934540Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7935023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7935552Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7935754Z 2025-08-14T21:55:49.7935875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7936268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7936620Z return mod(**inputs) 2025-08-14T21:55:49.7937024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7937457Z outputs = self.model.decoder( 2025-08-14T21:55:49.7937914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7938342Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7938717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7939121Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7939538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7939973Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7940406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7940857Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7941341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7941828Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7942013Z 2025-08-14T21:55:49.7942128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7942514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7942865Z return mod(**inputs) 2025-08-14T21:55:49.7943277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7943709Z outputs = self.model.decoder( 2025-08-14T21:55:49.7944131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7944549Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7944927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7945346Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7945784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7946233Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7946705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.7947140Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.7947289Z 2025-08-14T21:55:49.7947408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7947785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7948137Z return mod(**inputs) 2025-08-14T21:55:49.7948536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7949020Z outputs = self.model.decoder( 2025-08-14T21:55:49.7949445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7949889Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7950271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7950664Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7951103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7951593Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7951784Z 2025-08-14T21:55:49.7951907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7952296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7952662Z return mod(**inputs) 2025-08-14T21:55:49.7953080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7953516Z outputs = self.model.decoder( 2025-08-14T21:55:49.7953947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7954390Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7954771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7955174Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7955606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7956174Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7956606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.7956981Z return self.act(input) 2025-08-14T21:55:49.7957102Z 2025-08-14T21:55:49.7957214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7957608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7957965Z return mod(**inputs) 2025-08-14T21:55:49.7958364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7958794Z outputs = self.model.decoder( 2025-08-14T21:55:49.7959218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7959648Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7960021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7960443Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7960873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.7961310Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.7961458Z 2025-08-14T21:55:49.7961570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7961974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7962340Z return mod(**inputs) 2025-08-14T21:55:49.7962722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7963135Z outputs = self.model.decoder( 2025-08-14T21:55:49.7963539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7963951Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7964358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7964741Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7965181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7965618Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7966055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7966220Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7966225Z 2025-08-14T21:55:49.7966335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7966543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7966621Z return mod(**inputs) 2025-08-14T21:55:49.7966898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7966983Z outputs = self.model.decoder( 2025-08-14T21:55:49.7967258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7967333Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7967570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7967653Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7967926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7968035Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7968306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7968400Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7968403Z 2025-08-14T21:55:49.7968510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7968718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7968797Z return mod(**inputs) 2025-08-14T21:55:49.7969073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7969158Z outputs = self.model.decoder( 2025-08-14T21:55:49.7969431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7969505Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7969744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7969849Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7970124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7970233Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7970502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7970620Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7970623Z 2025-08-14T21:55:49.7970709Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7970791Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7970882Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7970961Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7971068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7971306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7971379Z return mod(**inputs) 2025-08-14T21:55:49.7971664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7971757Z outputs = self.model.decoder( 2025-08-14T21:55:49.7972032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7972120Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7972350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7972431Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7972711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7972812Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7973098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7973200Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7973505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7973655Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7973659Z 2025-08-14T21:55:49.7973766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7973980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7974049Z return mod(**inputs) 2025-08-14T21:55:49.7974331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7974420Z outputs = self.model.decoder( 2025-08-14T21:55:49.7974707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7974785Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7975040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7975121Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7975400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7975501Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7975772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7975882Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7976194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7976334Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7976339Z 2025-08-14T21:55:49.7976445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7976652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7976746Z return mod(**inputs) 2025-08-14T21:55:49.7977025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7977110Z outputs = self.model.decoder( 2025-08-14T21:55:49.7977386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7977461Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7977699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7977802Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7978074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7978202Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7978475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.7978568Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.7978572Z 2025-08-14T21:55:49.7978678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7978883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7978960Z return mod(**inputs) 2025-08-14T21:55:49.7979237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7979314Z outputs = self.model.decoder( 2025-08-14T21:55:49.7979599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7979675Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7979918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7980001Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7980273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7980406Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7980410Z 2025-08-14T21:55:49.7980518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7980730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7980801Z return mod(**inputs) 2025-08-14T21:55:49.7981076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7981163Z outputs = self.model.decoder( 2025-08-14T21:55:49.7981445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7981524Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7981764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7981850Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7982139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.7982267Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.7982496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.7982602Z return self.act(input) 2025-08-14T21:55:49.7982606Z 2025-08-14T21:55:49.7982715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7982948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7983017Z return mod(**inputs) 2025-08-14T21:55:49.7983313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7983396Z outputs = self.model.decoder( 2025-08-14T21:55:49.7983673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7983747Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7983989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7984072Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7984375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.7984465Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.7984484Z 2025-08-14T21:55:49.7984592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7984809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7984884Z return mod(**inputs) 2025-08-14T21:55:49.7985177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7985260Z outputs = self.model.decoder( 2025-08-14T21:55:49.7985546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7985637Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7985880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7985970Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7986264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:55:49.7986356Z hidden_states = residual + hidden_states 2025-08-14T21:55:49.7986361Z 2025-08-14T21:55:49.7986482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7986699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7986774Z return mod(**inputs) 2025-08-14T21:55:49.7987067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7987149Z outputs = self.model.decoder( 2025-08-14T21:55:49.7987435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7987523Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7987773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7987868Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7988152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7988263Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7988558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.7988724Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.7988728Z 2025-08-14T21:55:49.7988849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7989083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7989154Z return mod(**inputs) 2025-08-14T21:55:49.7989444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7989522Z outputs = self.model.decoder( 2025-08-14T21:55:49.7989859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7989944Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7990178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7990270Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7990554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7990659Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7990965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.7991053Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.7991077Z 2025-08-14T21:55:49.7991194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7991407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7991477Z return mod(**inputs) 2025-08-14T21:55:49.7991772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7991850Z outputs = self.model.decoder( 2025-08-14T21:55:49.7992137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7992220Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7992462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7992555Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7992840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7992945Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7993238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.7993329Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.7993332Z 2025-08-14T21:55:49.7993424Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7993510Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7993594Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7993683Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.7993792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7994008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7994087Z return mod(**inputs) 2025-08-14T21:55:49.7994373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7994458Z outputs = self.model.decoder( 2025-08-14T21:55:49.7994741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7994815Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7995064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7995147Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7995428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7995563Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7995951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7996073Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7996390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.7996567Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.7996571Z 2025-08-14T21:55:49.7996693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7996909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7996990Z return mod(**inputs) 2025-08-14T21:55:49.7997276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.7997381Z outputs = self.model.decoder( 2025-08-14T21:55:49.7997678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.7997773Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.7998073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.7998169Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.7998452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.7998562Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.7998845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.7998950Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.7999283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.7999405Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.7999409Z 2025-08-14T21:55:49.7999532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.7999747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.7999820Z return mod(**inputs) 2025-08-14T21:55:49.8000112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8000192Z outputs = self.model.decoder( 2025-08-14T21:55:49.8000480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8000566Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8000817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8000908Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8001194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8001296Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8001582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.8001667Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.8001671Z 2025-08-14T21:55:49.8001785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8001993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8002061Z return mod(**inputs) 2025-08-14T21:55:49.8002346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8002442Z outputs = self.model.decoder( 2025-08-14T21:55:49.8002715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8002797Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8003049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8003137Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8003413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.8003539Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.8003543Z 2025-08-14T21:55:49.8003656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8003877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8003957Z return mod(**inputs) 2025-08-14T21:55:49.8004237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8004328Z outputs = self.model.decoder( 2025-08-14T21:55:49.8004613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8004689Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8004927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8005017Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8005291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.8005421Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.8005645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.8005719Z return self.act(input) 2025-08-14T21:55:49.8005722Z 2025-08-14T21:55:49.8005836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8006051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8006121Z return mod(**inputs) 2025-08-14T21:55:49.8006406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8006477Z outputs = self.model.decoder( 2025-08-14T21:55:49.8006743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8006813Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8007035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8007121Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8007382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.8007473Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.8007479Z 2025-08-14T21:55:49.8007585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8007804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8007880Z return mod(**inputs) 2025-08-14T21:55:49.8008156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8008232Z outputs = self.model.decoder( 2025-08-14T21:55:49.8008516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8008614Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8009009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8009099Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8009374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8009532Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8009804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.8009969Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.8009974Z 2025-08-14T21:55:49.8010078Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8010282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8010395Z return mod(**inputs) 2025-08-14T21:55:49.8010673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8010776Z outputs = self.model.decoder( 2025-08-14T21:55:49.8011062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8011139Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8011376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8011460Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8011731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8011840Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8012114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.8012206Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.8012210Z 2025-08-14T21:55:49.8012316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8012518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8012595Z return mod(**inputs) 2025-08-14T21:55:49.8012869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8012944Z outputs = self.model.decoder( 2025-08-14T21:55:49.8013224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8013298Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8013536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8013618Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8013890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8014001Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8014275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.8014372Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.8014376Z 2025-08-14T21:55:49.8014460Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.8014542Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.8014629Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.8014708Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.8014814Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8015069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8015137Z return mod(**inputs) 2025-08-14T21:55:49.8015418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8015495Z outputs = self.model.decoder( 2025-08-14T21:55:49.8015788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8015870Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8016106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8016188Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8016471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8016572Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8016870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.8017336Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.8017833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.8018354Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.8018560Z 2025-08-14T21:55:49.8018753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8019166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8019521Z return mod(**inputs) 2025-08-14T21:55:49.8019936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8020357Z outputs = self.model.decoder( 2025-08-14T21:55:49.8020783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8021208Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8021598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8021988Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8022428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8053342Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8053940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.8054428Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.8054953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.8055471Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.8055659Z 2025-08-14T21:55:49.8055790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8056194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8056561Z return mod(**inputs) 2025-08-14T21:55:49.8056983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8057429Z outputs = self.model.decoder( 2025-08-14T21:55:49.8057876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8058319Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8058710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8059322Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8059805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8060290Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8060787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.8061247Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.8061404Z 2025-08-14T21:55:49.8061520Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8061918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8062309Z return mod(**inputs) 2025-08-14T21:55:49.8062716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8063213Z outputs = self.model.decoder( 2025-08-14T21:55:49.8063637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8064113Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8064507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8064914Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8065348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.8065852Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.8066045Z 2025-08-14T21:55:49.8066171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8066562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8066913Z return mod(**inputs) 2025-08-14T21:55:49.8067326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8067778Z outputs = self.model.decoder( 2025-08-14T21:55:49.8068194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8068639Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8069035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8069442Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8069873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.8070353Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.8070777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.8071146Z return self.act(input) 2025-08-14T21:55:49.8071277Z 2025-08-14T21:55:49.8071389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8071783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8072151Z return mod(**inputs) 2025-08-14T21:55:49.8072549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8072983Z outputs = self.model.decoder( 2025-08-14T21:55:49.8073410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8073848Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8074226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8074656Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8075092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.8075522Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.8075681Z 2025-08-14T21:55:49.8076056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8076523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8076902Z return mod(**inputs) 2025-08-14T21:55:49.8077310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8077753Z outputs = self.model.decoder( 2025-08-14T21:55:49.8078191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8078627Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8079047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8079446Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8079904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:55:49.8080340Z hidden_states = residual + hidden_states 2025-08-14T21:55:49.8080496Z 2025-08-14T21:55:49.8080610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8080997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8081348Z return mod(**inputs) 2025-08-14T21:55:49.8081748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8082165Z outputs = self.model.decoder( 2025-08-14T21:55:49.8082581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8082993Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8083366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8083754Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8084175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8084614Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8085063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:55:49.8085570Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:55:49.8085788Z 2025-08-14T21:55:49.8085907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8086281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8086626Z return mod(**inputs) 2025-08-14T21:55:49.8087020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8087443Z outputs = self.model.decoder( 2025-08-14T21:55:49.8087854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8088273Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8088637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8089021Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8089443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8089919Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8090356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:55:49.8090792Z key_states = self.k_proj(current_states) 2025-08-14T21:55:49.8090942Z 2025-08-14T21:55:49.8091050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8091453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8091795Z return mod(**inputs) 2025-08-14T21:55:49.8092192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8092620Z outputs = self.model.decoder( 2025-08-14T21:55:49.8093031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8093443Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8093834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8094223Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8094651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8095099Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8095538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:55:49.8095976Z value_states = self.v_proj(current_states) 2025-08-14T21:55:49.8096124Z 2025-08-14T21:55:49.8096215Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.8096448Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.8096672Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.8096884Z cudagraph partition due to non gpu ops 2025-08-14T21:55:49.8097140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8097523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8097866Z return mod(**inputs) 2025-08-14T21:55:49.8098256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8098678Z outputs = self.model.decoder( 2025-08-14T21:55:49.8099092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8099517Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8099898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8100289Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8100724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8101181Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8101655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.8102110Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.8102595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:55:49.8103125Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:49.8103325Z 2025-08-14T21:55:49.8103438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8103829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8104195Z return mod(**inputs) 2025-08-14T21:55:49.8104619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8105078Z outputs = self.model.decoder( 2025-08-14T21:55:49.8105518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8105950Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8106324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8106736Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8107171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8107627Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8108069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:55:49.8108531Z attn_output, attn_weights = attention_interface( 2025-08-14T21:55:49.8109392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:55:49.8109934Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:55:49.8110118Z 2025-08-14T21:55:49.8110235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8110626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8110980Z return mod(**inputs) 2025-08-14T21:55:49.8111398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8111835Z outputs = self.model.decoder( 2025-08-14T21:55:49.8112262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8112693Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8113065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8113457Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8113894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:55:49.8114355Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:55:49.8114801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:55:49.8115238Z attn_output = self.out_proj(attn_output) 2025-08-14T21:55:49.8115386Z 2025-08-14T21:55:49.8115508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8116074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8116465Z return mod(**inputs) 2025-08-14T21:55:49.8116892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8117327Z outputs = self.model.decoder( 2025-08-14T21:55:49.8117753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8118186Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8118567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8118962Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8119383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.8119862Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.8120051Z 2025-08-14T21:55:49.8120176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8120620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8120975Z return mod(**inputs) 2025-08-14T21:55:49.8121384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8121818Z outputs = self.model.decoder( 2025-08-14T21:55:49.8123206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8123640Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8124024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8124413Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8124857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:55:49.8125325Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:55:49.8125764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:49.8126125Z return self.act(input) 2025-08-14T21:55:49.8126250Z 2025-08-14T21:55:49.8126383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8126775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8127132Z return mod(**inputs) 2025-08-14T21:55:49.8127544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:55:49.8127985Z outputs = self.model.decoder( 2025-08-14T21:55:49.8128417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:55:49.8128841Z layer_outputs = decoder_layer( 2025-08-14T21:55:49.8129215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:49.8129603Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:49.8130034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:55:49.8130462Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:55:49.8130619Z 2025-08-14T21:55:49.8130731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8131116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8131463Z return mod(**inputs) 2025-08-14T21:55:49.8131884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1650, in forward 2025-08-14T21:55:49.8132316Z logits = self.lm_head(outputs[0]) 2025-08-14T21:55:49.8132457Z 2025-08-14T21:55:49.8132576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:49.8132959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:49.8133309Z return mod(**inputs) 2025-08-14T21:55:49.8133714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1656, in forward 2025-08-14T21:55:49.8134220Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:55:49.8134434Z 2025-08-14T21:55:58.9660812Z Compilation time (from dynamo_timed): 15.788451984 2025-08-14T21:55:58.9675051Z pass 2025-08-14T21:55:58.9675448Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:58.9676533Z TIMING: _recursive_pre_grad_passes:0.00792 _recursive_joint_graph_passes:0.67657 _recursive_post_grad_passes:0.09022 async_compile.wait:0.80256 code_gen:8.38068 inductor_compile:9.70113 backend_compile:13.10086 gc:0.00012 entire_frame_compile:15.78845 total_wall_time:15.78845 2025-08-14T21:55:58.9677769Z STATS: call_* op count: 369 | FakeTensorMode.__torch_dispatch__:13170 | FakeTensor.__torch_dispatch__:4856 | ProxyTorchDispatchMode.__torch_dispatch__:4803 2025-08-14T21:55:58.9678271Z Dynamo produced 1 graphs covering 369 ops with 0 graph breaks (0 unique) 2025-08-14T21:56:04.3901201Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:56:04.3902654Z from pkg_resources import resource_filename 2025-08-14T21:56:05.0064781Z 2025-08-14T21:56:10.9407622Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:56:10.9407998Z loading model: 0it [00:05, ?it/s] 2025-08-14T21:56:10.9436072Z cpu eval PegasusForConditionalGeneration 2025-08-14T21:56:11.6024274Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:11.8959014Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:12.1700947Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:29.3967961Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3969615Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3969867Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3970114Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3970382Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3970612Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3970846Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3971092Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3971324Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3971575Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3971797Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3972047Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3972349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.3972808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.3973259Z return mod(**inputs) 2025-08-14T21:56:29.3973701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.3974202Z outputs = self.model( 2025-08-14T21:56:29.3974653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.3975098Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.3975534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.3975969Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.3976364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.3976734Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.3977174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.3977658Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.3978116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.3978632Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.3978862Z 2025-08-14T21:56:29.3978978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.3979373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.3980012Z return mod(**inputs) 2025-08-14T21:56:29.3980415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.3980840Z outputs = self.model( 2025-08-14T21:56:29.3981243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.3981749Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.3982162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.3982591Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.3982963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.3983349Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.3983777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.3984323Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.3984785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.3985268Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.3985420Z 2025-08-14T21:56:29.3985547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.3985915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.3986249Z return mod(**inputs) 2025-08-14T21:56:29.3986627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.3987038Z outputs = self.model( 2025-08-14T21:56:29.3987440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.3987874Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.3988306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.3988724Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.3989108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.3989494Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.3989916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.3990355Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.3990815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.3991320Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.3991487Z 2025-08-14T21:56:29.3991581Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3991820Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3992044Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3992277Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.3992539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.3992931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.3993306Z return mod(**inputs) 2025-08-14T21:56:29.3993730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.3994174Z outputs = self.model( 2025-08-14T21:56:29.3994597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.3995037Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.3995484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.3996193Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.3996580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.3996978Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.3997411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.3997872Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.3998308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.3998755Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.3999236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.3999763Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.3999970Z 2025-08-14T21:56:29.4000081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4000488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4000834Z return mod(**inputs) 2025-08-14T21:56:29.4001236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4001647Z outputs = self.model( 2025-08-14T21:56:29.4002057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4002478Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4002888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4003300Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4003670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4004059Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4004491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4004923Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4005347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4005796Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4006262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4006745Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4006915Z 2025-08-14T21:56:29.4007029Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4007410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4007752Z return mod(**inputs) 2025-08-14T21:56:29.4008139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4008553Z outputs = self.model( 2025-08-14T21:56:29.4009143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4009565Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4009969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4010386Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4010761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4011191Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4011607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4012042Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4012478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4012924Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4013081Z 2025-08-14T21:56:29.4013193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4013578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4013921Z return mod(**inputs) 2025-08-14T21:56:29.4014307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4014719Z outputs = self.model( 2025-08-14T21:56:29.4015127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4015524Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4015932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4016326Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4016674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4017024Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4017426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4017893Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4018077Z 2025-08-14T21:56:29.4018197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4018573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4018916Z return mod(**inputs) 2025-08-14T21:56:29.4019287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4019671Z outputs = self.model( 2025-08-14T21:56:29.4020048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4020449Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4020836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4021218Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4021562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4022001Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4022393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4022823Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4023208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4023553Z return self.act(input) 2025-08-14T21:56:29.4023664Z 2025-08-14T21:56:29.4024066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4024428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4024750Z return mod(**inputs) 2025-08-14T21:56:29.4025125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4025545Z outputs = self.model( 2025-08-14T21:56:29.4025943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4026365Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4026769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4027206Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4027572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4027932Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4028319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4028725Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4028870Z 2025-08-14T21:56:29.4028975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4029379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4029699Z return mod(**inputs) 2025-08-14T21:56:29.4030088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4030483Z outputs = self.model( 2025-08-14T21:56:29.4030853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4031259Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4031649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4032060Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4032424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4032830Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4033263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4033719Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4034167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4034689Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4034904Z 2025-08-14T21:56:29.4035021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4035411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4035777Z return mod(**inputs) 2025-08-14T21:56:29.4036265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4036693Z outputs = self.model( 2025-08-14T21:56:29.4037092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4037529Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4037940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4038356Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4038713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4039101Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4039524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4039956Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4040394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4040839Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4040981Z 2025-08-14T21:56:29.4041098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4041473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4041834Z return mod(**inputs) 2025-08-14T21:56:29.4042231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4042639Z outputs = self.model( 2025-08-14T21:56:29.4043037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4043465Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4043883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4044309Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4044675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4045082Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4045513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4045944Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4046380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4046812Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4046962Z 2025-08-14T21:56:29.4047045Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4047273Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4047494Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4047703Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4047934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4048289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4048625Z return mod(**inputs) 2025-08-14T21:56:29.4049001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4049407Z outputs = self.model( 2025-08-14T21:56:29.4049802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4050219Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4050631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4051051Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4051417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4051796Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4052227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4052669Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4053110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4053551Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4054017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4054525Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4054726Z 2025-08-14T21:56:29.4054841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4055269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4055623Z return mod(**inputs) 2025-08-14T21:56:29.4055998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4056384Z outputs = self.model( 2025-08-14T21:56:29.4056784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4057180Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4057585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4057992Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4058361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4058754Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4059182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4059619Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4060067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4060483Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4060937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4061415Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4061590Z 2025-08-14T21:56:29.4061701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4062081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4062415Z return mod(**inputs) 2025-08-14T21:56:29.4062813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4063229Z outputs = self.model( 2025-08-14T21:56:29.4063619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4064038Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4064449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4064871Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4065245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4065634Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4066052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4066485Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4066906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4067333Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4067479Z 2025-08-14T21:56:29.4067592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4067962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4068326Z return mod(**inputs) 2025-08-14T21:56:29.4068720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4069129Z outputs = self.model( 2025-08-14T21:56:29.4069511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4069960Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4070384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4070815Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4071200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4071612Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4072042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4072512Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4072722Z 2025-08-14T21:56:29.4072833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4073240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4073610Z return mod(**inputs) 2025-08-14T21:56:29.4074021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4074456Z outputs = self.model( 2025-08-14T21:56:29.4074883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4075308Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4075733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4076270Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4076654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4077047Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4077489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4077960Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4078367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4078721Z return self.act(input) 2025-08-14T21:56:29.4078847Z 2025-08-14T21:56:29.4078958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4079342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4079674Z return mod(**inputs) 2025-08-14T21:56:29.4080072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4080482Z outputs = self.model( 2025-08-14T21:56:29.4080872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4081281Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4081690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4082101Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4082456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4082833Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4083246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4083677Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4083823Z 2025-08-14T21:56:29.4083932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4084306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4084642Z return mod(**inputs) 2025-08-14T21:56:29.4085055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4085453Z outputs = self.model( 2025-08-14T21:56:29.4085846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4086296Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4086698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4087115Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4087484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4087871Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4088289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4088745Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4089177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4089693Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4089908Z 2025-08-14T21:56:29.4090022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4090403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4090741Z return mod(**inputs) 2025-08-14T21:56:29.4091107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4091504Z outputs = self.model( 2025-08-14T21:56:29.4091879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4092285Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4092691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4093114Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4093483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4093862Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4094273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4094705Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4095137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4095554Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4095703Z 2025-08-14T21:56:29.4095817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4096196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4096539Z return mod(**inputs) 2025-08-14T21:56:29.4096929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4097346Z outputs = self.model( 2025-08-14T21:56:29.4097743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4098158Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4098572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4098995Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4099371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4099778Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4100207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4100657Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4101098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4101547Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4101707Z 2025-08-14T21:56:29.4101794Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4102025Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4102244Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4102469Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4102729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4103144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4103492Z return mod(**inputs) 2025-08-14T21:56:29.4103899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4104343Z outputs = self.model( 2025-08-14T21:56:29.4104743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4105179Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4105620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4106058Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4106427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4106825Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4107263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4107708Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4108154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4108606Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4109257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4109788Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4110011Z 2025-08-14T21:56:29.4110127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4110521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4110891Z return mod(**inputs) 2025-08-14T21:56:29.4111303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4111735Z outputs = self.model( 2025-08-14T21:56:29.4112149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4112584Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4113000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4113415Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4113787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4114207Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4114655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4115167Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4115630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4116156Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4116639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4117208Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4117385Z 2025-08-14T21:56:29.4117507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4117886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4118227Z return mod(**inputs) 2025-08-14T21:56:29.4118620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4119058Z outputs = self.model( 2025-08-14T21:56:29.4119471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4119903Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4120336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4120750Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4121116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4121504Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4121913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4122348Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4122779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4123202Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4123344Z 2025-08-14T21:56:29.4123453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4123833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4124181Z return mod(**inputs) 2025-08-14T21:56:29.4124577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4124957Z outputs = self.model( 2025-08-14T21:56:29.4125318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4125705Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4126086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4126479Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4126826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4127180Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4127579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4128019Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4128183Z 2025-08-14T21:56:29.4128291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4128631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4128949Z return mod(**inputs) 2025-08-14T21:56:29.4129310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4129713Z outputs = self.model( 2025-08-14T21:56:29.4130074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4130459Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4130840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4131234Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4131586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4131944Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4132341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4132770Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4133173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4133520Z return self.act(input) 2025-08-14T21:56:29.4133632Z 2025-08-14T21:56:29.4133744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4134145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4134491Z return mod(**inputs) 2025-08-14T21:56:29.4134889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4135294Z outputs = self.model( 2025-08-14T21:56:29.4135692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4136112Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4136524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4136945Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4137292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4137655Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4138048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4138454Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4138596Z 2025-08-14T21:56:29.4138700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4139061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4139378Z return mod(**inputs) 2025-08-14T21:56:29.4139750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4140138Z outputs = self.model( 2025-08-14T21:56:29.4140505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4140902Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4141290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4141685Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4142025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4142394Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4142794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:56:29.4143196Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4143329Z 2025-08-14T21:56:29.4143435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4143823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4144149Z return mod(**inputs) 2025-08-14T21:56:29.4144512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4144901Z outputs = self.model( 2025-08-14T21:56:29.4145288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4145685Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4146067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4146454Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4146797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4147149Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4147561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4147982Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4148417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4148880Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4149089Z 2025-08-14T21:56:29.4149193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4149548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4149868Z return mod(**inputs) 2025-08-14T21:56:29.4150232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4150622Z outputs = self.model( 2025-08-14T21:56:29.4151000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4151405Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4151815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4152228Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4152591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4152958Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4153374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4153818Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4154261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4154685Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4154835Z 2025-08-14T21:56:29.4154948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4155343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4155676Z return mod(**inputs) 2025-08-14T21:56:29.4156153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4156587Z outputs = self.model( 2025-08-14T21:56:29.4156992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4157417Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4157802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4158254Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4158627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4159020Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4159457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4159942Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4160397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4160857Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4161023Z 2025-08-14T21:56:29.4161116Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4161360Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4161590Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4161826Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4162107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4162508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4162890Z return mod(**inputs) 2025-08-14T21:56:29.4163313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4163751Z outputs = self.model( 2025-08-14T21:56:29.4164206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4164686Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4165070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4165449Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4165803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4166167Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4166568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4166975Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4167392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4167816Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4168258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4168746Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4168933Z 2025-08-14T21:56:29.4169036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4169401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4169718Z return mod(**inputs) 2025-08-14T21:56:29.4170094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4170488Z outputs = self.model( 2025-08-14T21:56:29.4170865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4171263Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4171649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4172040Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4172386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4172768Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4173160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4173568Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4173955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4174380Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4174815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4175272Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4175433Z 2025-08-14T21:56:29.4175537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4175891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4176217Z return mod(**inputs) 2025-08-14T21:56:29.4176605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4176985Z outputs = self.model( 2025-08-14T21:56:29.4177360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4177755Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4178137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4178532Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4178885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4179240Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4179639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4180056Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4180469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4180868Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4181014Z 2025-08-14T21:56:29.4181122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4181483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4181818Z return mod(**inputs) 2025-08-14T21:56:29.4182188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4182579Z outputs = self.model( 2025-08-14T21:56:29.4182957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4183366Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4183769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4184180Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4184539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4184904Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4185313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4185764Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4185944Z 2025-08-14T21:56:29.4186059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4186422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4186830Z return mod(**inputs) 2025-08-14T21:56:29.4187202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4187588Z outputs = self.model( 2025-08-14T21:56:29.4188021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4188439Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4188830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4189221Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4189569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4189932Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4190330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4190789Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4191178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4191545Z return self.act(input) 2025-08-14T21:56:29.4191663Z 2025-08-14T21:56:29.4191772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4192152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4192510Z return mod(**inputs) 2025-08-14T21:56:29.4192905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4193328Z outputs = self.model( 2025-08-14T21:56:29.4193728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4194169Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4194572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4194988Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4195386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4195773Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4196264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4196695Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4196845Z 2025-08-14T21:56:29.4196969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4197374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4197740Z return mod(**inputs) 2025-08-14T21:56:29.4198140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4198533Z outputs = self.model( 2025-08-14T21:56:29.4198904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4199305Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4199700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4200087Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4200434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4200797Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4201196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4201622Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4202035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4202509Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4202713Z 2025-08-14T21:56:29.4202840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4203187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4203511Z return mod(**inputs) 2025-08-14T21:56:29.4203873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4204256Z outputs = self.model( 2025-08-14T21:56:29.4204637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4205081Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4205491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4205905Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4206268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4206638Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4207023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4207414Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4207811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4208206Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4208340Z 2025-08-14T21:56:29.4208447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4208986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4209320Z return mod(**inputs) 2025-08-14T21:56:29.4209701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4210072Z outputs = self.model( 2025-08-14T21:56:29.4210446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4210847Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4211230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4211626Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4211975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4212343Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4212744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4213146Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4213546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4213946Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4214085Z 2025-08-14T21:56:29.4214167Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4214376Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4214585Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4214782Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4215018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4215376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4215741Z return mod(**inputs) 2025-08-14T21:56:29.4216103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4216484Z outputs = self.model( 2025-08-14T21:56:29.4216847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4217254Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4217639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4218017Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4218358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4218701Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4219132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4219536Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4220709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4221136Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4221569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4222031Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4222209Z 2025-08-14T21:56:29.4222311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4222662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4222981Z return mod(**inputs) 2025-08-14T21:56:29.4223350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4223738Z outputs = self.model( 2025-08-14T21:56:29.4224116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4224512Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4224894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4225281Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4225627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4225989Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4226376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4226790Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4227194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4227608Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4228039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4228493Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4228652Z 2025-08-14T21:56:29.4228763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4229112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4229435Z return mod(**inputs) 2025-08-14T21:56:29.4229807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4230224Z outputs = self.model( 2025-08-14T21:56:29.4230591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4230994Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4231387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4231786Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4232133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4232496Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4232894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4233302Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4233749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4234177Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4234320Z 2025-08-14T21:56:29.4234437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4234822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4235180Z return mod(**inputs) 2025-08-14T21:56:29.4235572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4236070Z outputs = self.model( 2025-08-14T21:56:29.4236471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4236917Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4237329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4237721Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4238074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4238444Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4238834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4239280Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4239462Z 2025-08-14T21:56:29.4239565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4239929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4240251Z return mod(**inputs) 2025-08-14T21:56:29.4240627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4241024Z outputs = self.model( 2025-08-14T21:56:29.4241401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4241790Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4242182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4242575Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4242919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4243280Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4243678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4244137Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4244556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4244918Z return self.act(input) 2025-08-14T21:56:29.4245027Z 2025-08-14T21:56:29.4245138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4245498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4245829Z return mod(**inputs) 2025-08-14T21:56:29.4246209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4246604Z outputs = self.model( 2025-08-14T21:56:29.4246975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4247376Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4247773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4248201Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4248542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4248906Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4249322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4249722Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4249868Z 2025-08-14T21:56:29.4249970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4250331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4250657Z return mod(**inputs) 2025-08-14T21:56:29.4251022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4251414Z outputs = self.model( 2025-08-14T21:56:29.4251792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4252185Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4252583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4252974Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4253348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4253734Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4254162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:56:29.4254596Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4254740Z 2025-08-14T21:56:29.4254857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4255227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4255542Z return mod(**inputs) 2025-08-14T21:56:29.4255905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4256281Z outputs = self.model( 2025-08-14T21:56:29.4256647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4257034Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4257413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4257790Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4258129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4258478Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4258886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4259287Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4259685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4260157Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4260352Z 2025-08-14T21:56:29.4260454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4260804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4261118Z return mod(**inputs) 2025-08-14T21:56:29.4261480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4261861Z outputs = self.model( 2025-08-14T21:56:29.4262241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4262626Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4263018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4263424Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4263761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4264113Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4264493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4264894Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4265296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4265702Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4265836Z 2025-08-14T21:56:29.4265941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4266300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4266626Z return mod(**inputs) 2025-08-14T21:56:29.4266993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4267384Z outputs = self.model( 2025-08-14T21:56:29.4267759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4268160Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4268542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4268937Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4269284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4269645Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4270064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4270502Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4270932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4271352Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4271508Z 2025-08-14T21:56:29.4271594Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4271819Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4272041Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4272281Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4272537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4272912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4273247Z return mod(**inputs) 2025-08-14T21:56:29.4273643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4274085Z outputs = self.model( 2025-08-14T21:56:29.4274490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4274917Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4275327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4275749Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4276223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4276621Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4277107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4277548Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4277953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4278372Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4278811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4279278Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4279468Z 2025-08-14T21:56:29.4279573Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4279934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4280251Z return mod(**inputs) 2025-08-14T21:56:29.4280619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4281009Z outputs = self.model( 2025-08-14T21:56:29.4281381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4281771Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4282154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4282542Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4282890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4283244Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4283657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4284090Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4284526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4284962Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4285439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4285896Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4286058Z 2025-08-14T21:56:29.4286172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4286526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4286894Z return mod(**inputs) 2025-08-14T21:56:29.4287289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4287693Z outputs = self.model( 2025-08-14T21:56:29.4288091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4288535Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4288950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4289360Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4289729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4290113Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4290528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4290985Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4291417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4291875Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4292020Z 2025-08-14T21:56:29.4292131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4292512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4292855Z return mod(**inputs) 2025-08-14T21:56:29.4293249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4293659Z outputs = self.model( 2025-08-14T21:56:29.4294074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4294498Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4294900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4295316Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4295683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4296064Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4296476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4296942Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4297120Z 2025-08-14T21:56:29.4297241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4297616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4297952Z return mod(**inputs) 2025-08-14T21:56:29.4298363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4298857Z outputs = self.model( 2025-08-14T21:56:29.4299255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4299689Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4300100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4300524Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4300863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4301221Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4301615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4302078Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4302468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4302811Z return self.act(input) 2025-08-14T21:56:29.4302922Z 2025-08-14T21:56:29.4303034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4303445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4303793Z return mod(**inputs) 2025-08-14T21:56:29.4304166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4304552Z outputs = self.model( 2025-08-14T21:56:29.4304929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4305331Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4305755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4306162Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4306561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4306955Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4307362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4307768Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4307915Z 2025-08-14T21:56:29.4308020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4308388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4308845Z return mod(**inputs) 2025-08-14T21:56:29.4309233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4309648Z outputs = self.model( 2025-08-14T21:56:29.4310061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4310480Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4310898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4311318Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4311682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4312077Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4312500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4312947Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4313375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4313876Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4314096Z 2025-08-14T21:56:29.4314209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4314589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4314923Z return mod(**inputs) 2025-08-14T21:56:29.4315320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4315732Z outputs = self.model( 2025-08-14T21:56:29.4316182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4316688Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4317113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4317528Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4317901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4318325Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4318761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4319212Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4319650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4320088Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4320233Z 2025-08-14T21:56:29.4320357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4320773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4321129Z return mod(**inputs) 2025-08-14T21:56:29.4321558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4321993Z outputs = self.model( 2025-08-14T21:56:29.4322398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4322832Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4323258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4323681Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4324057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4324456Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4324888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4325332Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4325783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4326231Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4326385Z 2025-08-14T21:56:29.4326484Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4326710Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4326939Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4327146Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4327373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4327733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4328063Z return mod(**inputs) 2025-08-14T21:56:29.4328432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4328829Z outputs = self.model( 2025-08-14T21:56:29.4329225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4329645Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4330054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4330460Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4330807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4331168Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4331560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4332000Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4332412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4332825Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4333289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4333773Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4333958Z 2025-08-14T21:56:29.4334071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4334425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4334755Z return mod(**inputs) 2025-08-14T21:56:29.4335148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4335539Z outputs = self.model( 2025-08-14T21:56:29.4335919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4336325Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4336720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4337107Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4337459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4337820Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4338219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4338627Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4339035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4339460Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4339896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4340352Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4340521Z 2025-08-14T21:56:29.4340627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4340986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4341308Z return mod(**inputs) 2025-08-14T21:56:29.4341683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4342081Z outputs = self.model( 2025-08-14T21:56:29.4342462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4342852Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4343245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4343648Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4343988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4344360Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4344781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4345219Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4345645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4346090Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4346243Z 2025-08-14T21:56:29.4346354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4346718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4347078Z return mod(**inputs) 2025-08-14T21:56:29.4347468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4347884Z outputs = self.model( 2025-08-14T21:56:29.4348269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4348688Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4349098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4349536Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4349894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4350287Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4350714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4351178Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4351369Z 2025-08-14T21:56:29.4351479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4351854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4352198Z return mod(**inputs) 2025-08-14T21:56:29.4352594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4353028Z outputs = self.model( 2025-08-14T21:56:29.4353424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4353842Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4354248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4354663Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4355028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4355400Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4355893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4356370Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4356783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4357136Z return self.act(input) 2025-08-14T21:56:29.4357262Z 2025-08-14T21:56:29.4357376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4357757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4358093Z return mod(**inputs) 2025-08-14T21:56:29.4358488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4358900Z outputs = self.model( 2025-08-14T21:56:29.4359295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4359706Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4360118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4360576Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4360925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4361346Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4361741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4362166Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4362305Z 2025-08-14T21:56:29.4362411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4362782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4363119Z return mod(**inputs) 2025-08-14T21:56:29.4363502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4363897Z outputs = self.model( 2025-08-14T21:56:29.4364304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4364702Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4365111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4365508Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4365857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4366238Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4366627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:56:29.4367025Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4367161Z 2025-08-14T21:56:29.4367273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4367478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4367552Z return mod(**inputs) 2025-08-14T21:56:29.4367823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4367891Z outputs = self.model( 2025-08-14T21:56:29.4368162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4368234Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4368493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4368574Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4368793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4368877Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4369138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4369230Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4369493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4369645Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4369649Z 2025-08-14T21:56:29.4369759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4369956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4370022Z return mod(**inputs) 2025-08-14T21:56:29.4370287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4370353Z outputs = self.model( 2025-08-14T21:56:29.4370639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4370720Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4370974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4371067Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4371282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4371358Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4371619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4371709Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4371969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4372067Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4372070Z 2025-08-14T21:56:29.4372174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4372394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4372461Z return mod(**inputs) 2025-08-14T21:56:29.4372722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4372798Z outputs = self.model( 2025-08-14T21:56:29.4373057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4373135Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4373389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4373461Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4373686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4373763Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4374029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4374123Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4374378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4374472Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4374476Z 2025-08-14T21:56:29.4374557Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4374636Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4374721Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4374797Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4374910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4375105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4375171Z return mod(**inputs) 2025-08-14T21:56:29.4375438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4375506Z outputs = self.model( 2025-08-14T21:56:29.4375767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4375846Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4376101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4376179Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4376397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4376491Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4376757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4376847Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4377119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4377224Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4377512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4377659Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4377663Z 2025-08-14T21:56:29.4377761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4377968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4378041Z return mod(**inputs) 2025-08-14T21:56:29.4378311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4378385Z outputs = self.model( 2025-08-14T21:56:29.4378641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4378712Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4378969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4379038Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4379254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4379335Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4379587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4379681Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4379929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4380024Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4380311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4380417Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4380421Z 2025-08-14T21:56:29.4380526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4380716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4380784Z return mod(**inputs) 2025-08-14T21:56:29.4381051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4381118Z outputs = self.model( 2025-08-14T21:56:29.4381376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4381461Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4381718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4381797Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4382013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4382091Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4382356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4382464Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4382736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4382817Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4382820Z 2025-08-14T21:56:29.4382938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4383134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4383198Z return mod(**inputs) 2025-08-14T21:56:29.4383448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4383521Z outputs = self.model( 2025-08-14T21:56:29.4383772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4383850Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4384111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4384184Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4384423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4384501Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4384759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4384874Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4384878Z 2025-08-14T21:56:29.4384977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4385172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4385235Z return mod(**inputs) 2025-08-14T21:56:29.4385487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4385562Z outputs = self.model( 2025-08-14T21:56:29.4385812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4385888Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4386135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4386204Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4386427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4386504Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4386768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4386897Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4387099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4387174Z return self.act(input) 2025-08-14T21:56:29.4387177Z 2025-08-14T21:56:29.4387275Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4387464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4387536Z return mod(**inputs) 2025-08-14T21:56:29.4387785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4387857Z outputs = self.model( 2025-08-14T21:56:29.4388108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4388197Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4388453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4388521Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4388731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4388835Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4389086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4389171Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4389175Z 2025-08-14T21:56:29.4389277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4389472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4389544Z return mod(**inputs) 2025-08-14T21:56:29.4389821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4389896Z outputs = self.model( 2025-08-14T21:56:29.4390171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4390244Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4390515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4390587Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4390802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4390888Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4391147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4391252Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4391525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4391684Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4391687Z 2025-08-14T21:56:29.4391803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4392011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4392086Z return mod(**inputs) 2025-08-14T21:56:29.4392369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4392446Z outputs = self.model( 2025-08-14T21:56:29.4392725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4392801Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4393072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4393157Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4393385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4393473Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4393743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4393839Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4394118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4394201Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4394204Z 2025-08-14T21:56:29.4394318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4394543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4394611Z return mod(**inputs) 2025-08-14T21:56:29.4394892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4394985Z outputs = self.model( 2025-08-14T21:56:29.4395261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4395344Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4395620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4395704Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4396027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4396144Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4396441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4396555Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4396837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4396939Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4396944Z 2025-08-14T21:56:29.4397038Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4397131Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4397213Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4397293Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4397412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4397620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4397692Z return mod(**inputs) 2025-08-14T21:56:29.4397977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4398049Z outputs = self.model( 2025-08-14T21:56:29.4398333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4398410Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4398683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4398763Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4398993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4399080Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4399353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4399446Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4399728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4399829Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4400134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4400279Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4400283Z 2025-08-14T21:56:29.4400388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4400600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4400669Z return mod(**inputs) 2025-08-14T21:56:29.4400967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4401043Z outputs = self.model( 2025-08-14T21:56:29.4401318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4401401Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4401690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4401765Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4402000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4402082Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4402355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4402457Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4402760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4402886Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4403189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4403306Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4403311Z 2025-08-14T21:56:29.4403426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4403633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4403709Z return mod(**inputs) 2025-08-14T21:56:29.4403985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4404057Z outputs = self.model( 2025-08-14T21:56:29.4404338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4404414Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4404686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4404768Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4404993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4405078Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4405336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4405425Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4405696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4405776Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4405780Z 2025-08-14T21:56:29.4405886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4406077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4406144Z return mod(**inputs) 2025-08-14T21:56:29.4406410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4406479Z outputs = self.model( 2025-08-14T21:56:29.4406737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4406816Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4407068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4407163Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4407392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4407469Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4407728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4407857Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4407861Z 2025-08-14T21:56:29.4407967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4408160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4408223Z return mod(**inputs) 2025-08-14T21:56:29.4408478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4408545Z outputs = self.model( 2025-08-14T21:56:29.4409020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4409107Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4409389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4409473Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4409692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4409769Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4410031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4410146Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4410355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4410434Z return self.act(input) 2025-08-14T21:56:29.4410438Z 2025-08-14T21:56:29.4410540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4410745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4410815Z return mod(**inputs) 2025-08-14T21:56:29.4411078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4411161Z outputs = self.model( 2025-08-14T21:56:29.4411417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4411497Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4411745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4411817Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4412035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4412110Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4412360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4412447Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4412451Z 2025-08-14T21:56:29.4412549Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4412747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4412811Z return mod(**inputs) 2025-08-14T21:56:29.4413062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4413133Z outputs = self.model( 2025-08-14T21:56:29.4413413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4413489Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4413740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4413808Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4414048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4414122Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4414375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:56:29.4414458Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4414461Z 2025-08-14T21:56:29.4414558Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4414769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4414834Z return mod(**inputs) 2025-08-14T21:56:29.4415107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4415179Z outputs = self.model( 2025-08-14T21:56:29.4415432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4415504Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4415760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4415831Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4416049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4416124Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4416378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4416473Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4416731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4416885Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4416888Z 2025-08-14T21:56:29.4416988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4417177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4417248Z return mod(**inputs) 2025-08-14T21:56:29.4417503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4417568Z outputs = self.model( 2025-08-14T21:56:29.4417831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4417901Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4418161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4418231Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4418441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4418526Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4418786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4418884Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4419134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4419235Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4419238Z 2025-08-14T21:56:29.4419344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4419538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4419601Z return mod(**inputs) 2025-08-14T21:56:29.4419878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4419946Z outputs = self.model( 2025-08-14T21:56:29.4420207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4420276Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4420529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4420607Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4420837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4420922Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4421187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4421277Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4421535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4421617Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4421621Z 2025-08-14T21:56:29.4421697Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4421780Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4421854Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4421934Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4422036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4422224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4422294Z return mod(**inputs) 2025-08-14T21:56:29.4422559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4422625Z outputs = self.model( 2025-08-14T21:56:29.4422876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4422944Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4423197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4423265Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4423477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4423564Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4423812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4423900Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4424158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4424253Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4424542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4424674Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4424678Z 2025-08-14T21:56:29.4424779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4425010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4425074Z return mod(**inputs) 2025-08-14T21:56:29.4425335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4425401Z outputs = self.model( 2025-08-14T21:56:29.4425651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4425745Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4426009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4426077Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4426311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4426389Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4426710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4426802Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4427074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4427184Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4427470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4427586Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4427590Z 2025-08-14T21:56:29.4427691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4427887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4427958Z return mod(**inputs) 2025-08-14T21:56:29.4428229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4428295Z outputs = self.model( 2025-08-14T21:56:29.4428557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4428628Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4428888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4428956Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4429166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4429249Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4429508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4429608Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4429867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4429949Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4429952Z 2025-08-14T21:56:29.4430059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4430257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4430322Z return mod(**inputs) 2025-08-14T21:56:29.4430590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4430657Z outputs = self.model( 2025-08-14T21:56:29.4430924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4431015Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4431279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4431357Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4431578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4431684Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4431939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4432058Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4432061Z 2025-08-14T21:56:29.4432170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4432362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4432427Z return mod(**inputs) 2025-08-14T21:56:29.4432711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4432780Z outputs = self.model( 2025-08-14T21:56:29.4433075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4433153Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4433424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4433506Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4433746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4433829Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4434111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4434238Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4434467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4434542Z return self.act(input) 2025-08-14T21:56:29.4434546Z 2025-08-14T21:56:29.4434654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4434878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4434947Z return mod(**inputs) 2025-08-14T21:56:29.4435228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4435300Z outputs = self.model( 2025-08-14T21:56:29.4435576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4435661Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4436023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4436107Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4436364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4436450Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4436742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4436830Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4436835Z 2025-08-14T21:56:29.4436943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4437190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4437257Z return mod(**inputs) 2025-08-14T21:56:29.4437528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4437615Z outputs = self.model( 2025-08-14T21:56:29.4437875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4437956Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4438230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4438301Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4438528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4438605Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4438870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4438963Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4439233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4439407Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4439411Z 2025-08-14T21:56:29.4439511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4439714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4439779Z return mod(**inputs) 2025-08-14T21:56:29.4440040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4440113Z outputs = self.model( 2025-08-14T21:56:29.4440407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4440479Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4440743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4440812Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4441038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4441118Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4441373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4441470Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4441727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4441815Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4441819Z 2025-08-14T21:56:29.4441920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4442147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4442291Z return mod(**inputs) 2025-08-14T21:56:29.4442564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4442630Z outputs = self.model( 2025-08-14T21:56:29.4442899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4442972Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4443232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4443303Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4443529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4443638Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4443916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4444022Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4444286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4444390Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4444394Z 2025-08-14T21:56:29.4444482Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4444561Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4444641Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4444727Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4444834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4445052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4445147Z return mod(**inputs) 2025-08-14T21:56:29.4445422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4445514Z outputs = self.model( 2025-08-14T21:56:29.4445789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4445866Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4446142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4446215Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4446449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4446529Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4446803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4446907Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4447181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4447283Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4447594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4447734Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4447738Z 2025-08-14T21:56:29.4447851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4448066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4448134Z return mod(**inputs) 2025-08-14T21:56:29.4448421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4448493Z outputs = self.model( 2025-08-14T21:56:29.4448779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4448855Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4449137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4449220Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4449454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4449537Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4449825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4449939Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4450230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4450336Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4450653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4450794Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4450797Z 2025-08-14T21:56:29.4450903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4451123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4451189Z return mod(**inputs) 2025-08-14T21:56:29.4451448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4451523Z outputs = self.model( 2025-08-14T21:56:29.4451799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4451873Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4452156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4452228Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4452464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4452547Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4452822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4452922Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4453197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4453300Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4453304Z 2025-08-14T21:56:29.4453411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4453619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4453696Z return mod(**inputs) 2025-08-14T21:56:29.4453975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4454047Z outputs = self.model( 2025-08-14T21:56:29.4454337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4454410Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4454694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4454771Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4455001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4455090Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4455364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4455489Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4455501Z 2025-08-14T21:56:29.4455609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4455812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4455889Z return mod(**inputs) 2025-08-14T21:56:29.4456166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4456261Z outputs = self.model( 2025-08-14T21:56:29.4456548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4456625Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4456910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4457000Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4457227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4457317Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4457592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4457713Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4457940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4458040Z return self.act(input) 2025-08-14T21:56:29.4458044Z 2025-08-14T21:56:29.4458160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4458387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4458458Z return mod(**inputs) 2025-08-14T21:56:29.4458743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4458813Z outputs = self.model( 2025-08-14T21:56:29.4459097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4459171Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4459446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4459530Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4459762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4459841Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4460125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4460209Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4460213Z 2025-08-14T21:56:29.4460324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4460531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4460599Z return mod(**inputs) 2025-08-14T21:56:29.4460883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4460953Z outputs = self.model( 2025-08-14T21:56:29.4461232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4461316Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4461592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4461673Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4461906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4461986Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4462265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:56:29.4462349Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4462352Z 2025-08-14T21:56:29.4462465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4462687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4462755Z return mod(**inputs) 2025-08-14T21:56:29.4463038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4463110Z outputs = self.model( 2025-08-14T21:56:29.4463387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4463484Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4463758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4463840Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4464071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4464152Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4464453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4464552Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4464850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4465012Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4465016Z 2025-08-14T21:56:29.4465123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4465339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4465408Z return mod(**inputs) 2025-08-14T21:56:29.4465685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4465765Z outputs = self.model( 2025-08-14T21:56:29.4466044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4466127Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4466403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4466480Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4466720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4466802Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4467084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4467179Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4467454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4467547Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4467550Z 2025-08-14T21:56:29.4467657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4467864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4467941Z return mod(**inputs) 2025-08-14T21:56:29.4468221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4468297Z outputs = self.model( 2025-08-14T21:56:29.4468577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4468653Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4468934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4469027Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4469256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4469345Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4469619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4469737Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4470006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4470095Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4470099Z 2025-08-14T21:56:29.4470190Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4470272Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4470359Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4470439Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4470564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4470778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4470846Z return mod(**inputs) 2025-08-14T21:56:29.4471155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4471239Z outputs = self.model( 2025-08-14T21:56:29.4471515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4471598Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4471870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4471944Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4472183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4472271Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4472545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4472648Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4472922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4473035Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4473337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4473483Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4473487Z 2025-08-14T21:56:29.4473603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4473816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4473896Z return mod(**inputs) 2025-08-14T21:56:29.4474187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4474257Z outputs = self.model( 2025-08-14T21:56:29.4474535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4474612Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4474883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4474967Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4475196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4475286Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4475586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4475685Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4476046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4476177Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4476490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4476611Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4476615Z 2025-08-14T21:56:29.4476724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4476945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4477016Z return mod(**inputs) 2025-08-14T21:56:29.4477321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4477399Z outputs = self.model( 2025-08-14T21:56:29.4477680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4477764Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4478024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4478095Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4478320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4478397Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4478660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:56:29.4478752Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:56:29.4479012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4479101Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4479105Z 2025-08-14T21:56:29.4479206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4479400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4479475Z return mod(**inputs) 2025-08-14T21:56:29.4479735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4479809Z outputs = self.model( 2025-08-14T21:56:29.4480069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4480143Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4480412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4480484Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4480701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4480788Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4481045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4481168Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4481172Z 2025-08-14T21:56:29.4481272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4481463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4481537Z return mod(**inputs) 2025-08-14T21:56:29.4481815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4481890Z outputs = self.model( 2025-08-14T21:56:29.4482150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4482223Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4482503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4482574Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4482788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4482874Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4483132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:56:29.4483272Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4483496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4483585Z return self.act(input) 2025-08-14T21:56:29.4483589Z 2025-08-14T21:56:29.4483708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4483915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4483989Z return mod(**inputs) 2025-08-14T21:56:29.4484263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4484334Z outputs = self.model( 2025-08-14T21:56:29.4484618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:56:29.4484695Z encoder_outputs = self.encoder( 2025-08-14T21:56:29.4484971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:56:29.4485051Z layer_outputs = encoder_layer( 2025-08-14T21:56:29.4485291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4485377Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4485638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:56:29.4485717Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4485721Z 2025-08-14T21:56:29.4485828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4486024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4486094Z return mod(**inputs) 2025-08-14T21:56:29.4486357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4486424Z outputs = self.model( 2025-08-14T21:56:29.4486694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4486765Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4487025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4487104Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4487320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4487405Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4487664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4487784Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4488049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4488200Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4488203Z 2025-08-14T21:56:29.4488309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4488518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4488583Z return mod(**inputs) 2025-08-14T21:56:29.4488848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4488914Z outputs = self.model( 2025-08-14T21:56:29.4489172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4489253Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4489522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4489602Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4489837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4489919Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4490180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4490280Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4490535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4490621Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4490624Z 2025-08-14T21:56:29.4490723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4490923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4490988Z return mod(**inputs) 2025-08-14T21:56:29.4491248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4491331Z outputs = self.model( 2025-08-14T21:56:29.4491584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4491660Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4491911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4491980Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4492205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4492284Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4492540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4492645Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4492900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4492994Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4492997Z 2025-08-14T21:56:29.4493074Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4493150Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4493232Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4493307Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4493407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4493607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4493691Z return mod(**inputs) 2025-08-14T21:56:29.4493957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4494037Z outputs = self.model( 2025-08-14T21:56:29.4494286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4494383Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4494642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4494719Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4494937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4495011Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4495295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4495394Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4495665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4495781Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4496061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4496195Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4496199Z 2025-08-14T21:56:29.4496297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4496489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4496562Z return mod(**inputs) 2025-08-14T21:56:29.4496820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4496894Z outputs = self.model( 2025-08-14T21:56:29.4497157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4497230Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4497499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4497581Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4497793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4497875Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4498128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4498234Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4498483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4498575Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4498862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4498971Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4498974Z 2025-08-14T21:56:29.4499080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4499274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4499337Z return mod(**inputs) 2025-08-14T21:56:29.4499602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4499684Z outputs = self.model( 2025-08-14T21:56:29.4499950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4500031Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4500296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4500390Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4500608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4500685Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4500952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4501049Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4501359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4501451Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4501454Z 2025-08-14T21:56:29.4501555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4501774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4501842Z return mod(**inputs) 2025-08-14T21:56:29.4502102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4502177Z outputs = self.model( 2025-08-14T21:56:29.4502437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4502517Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4502780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4502850Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4503069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4503147Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4503407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4503523Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4503782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4503941Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4503945Z 2025-08-14T21:56:29.4504046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4504245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4504324Z return mod(**inputs) 2025-08-14T21:56:29.4504584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4504658Z outputs = self.model( 2025-08-14T21:56:29.4504916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4504990Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4505264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4505331Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4505543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4505624Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4505896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4506007Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4506259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4506335Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4506352Z 2025-08-14T21:56:29.4506458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4506647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4506718Z return mod(**inputs) 2025-08-14T21:56:29.4506971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4507036Z outputs = self.model( 2025-08-14T21:56:29.4507310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4507382Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4507652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4507730Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4507942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4508022Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4508276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4508380Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4508767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4508863Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4508868Z 2025-08-14T21:56:29.4508953Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4509028Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4509104Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4509190Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4509297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4509505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4509584Z return mod(**inputs) 2025-08-14T21:56:29.4509863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4509941Z outputs = self.model( 2025-08-14T21:56:29.4510220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4510299Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4510588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4510663Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4510911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4511006Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4511283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4511402Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4511680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4511783Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4512100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4512294Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4512298Z 2025-08-14T21:56:29.4512415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4512621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4512716Z return mod(**inputs) 2025-08-14T21:56:29.4513011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4513080Z outputs = self.model( 2025-08-14T21:56:29.4513369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4513453Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4513772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4513860Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4514101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4514207Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4514500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4514615Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4514894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4514996Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4515299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4515418Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4515422Z 2025-08-14T21:56:29.4515529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4515740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4515863Z return mod(**inputs) 2025-08-14T21:56:29.4516147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4516229Z outputs = self.model( 2025-08-14T21:56:29.4516501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4516580Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4516867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4516943Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4517190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4517280Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4517553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4517674Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4517946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4518031Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4518035Z 2025-08-14T21:56:29.4518151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4518358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4518435Z return mod(**inputs) 2025-08-14T21:56:29.4518733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4518803Z outputs = self.model( 2025-08-14T21:56:29.4519084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4519160Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4519450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4519535Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4519761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4519852Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4520123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4520266Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4520270Z 2025-08-14T21:56:29.4520384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4520605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4520683Z return mod(**inputs) 2025-08-14T21:56:29.4520959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4521029Z outputs = self.model( 2025-08-14T21:56:29.4521310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4521387Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4521663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4521744Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4521980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4522069Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4522347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4522472Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4522704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4522776Z return self.act(input) 2025-08-14T21:56:29.4522780Z 2025-08-14T21:56:29.4522893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4523098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4523165Z return mod(**inputs) 2025-08-14T21:56:29.4523453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4523525Z outputs = self.model( 2025-08-14T21:56:29.4523804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4523888Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4524166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4524246Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4524478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4524561Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4524845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4524947Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4524953Z 2025-08-14T21:56:29.4525063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4525256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4525324Z return mod(**inputs) 2025-08-14T21:56:29.4525590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4525710Z outputs = self.model( 2025-08-14T21:56:29.4525968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4526048Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4526306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4526381Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4526615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4526693Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4527471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4527578Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4527860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4528019Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4528023Z 2025-08-14T21:56:29.4528129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4528351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4528420Z return mod(**inputs) 2025-08-14T21:56:29.4528716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4528794Z outputs = self.model( 2025-08-14T21:56:29.4529086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4529164Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4529430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4529500Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4529729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4529806Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4530076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4530178Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4530440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4530526Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4530529Z 2025-08-14T21:56:29.4530631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4530832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4530904Z return mod(**inputs) 2025-08-14T21:56:29.4531165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4531237Z outputs = self.model( 2025-08-14T21:56:29.4531505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4531576Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4531862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4531931Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4532149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4532250Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4532510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4532619Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4532876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4532960Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4532963Z 2025-08-14T21:56:29.4533049Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4533141Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4533224Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4533297Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4533397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4533612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4533679Z return mod(**inputs) 2025-08-14T21:56:29.4533941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4534016Z outputs = self.model( 2025-08-14T21:56:29.4534277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4534355Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4534619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4534692Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4534915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4534995Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4535255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4535367Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4535638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4535747Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4536052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4536191Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4536197Z 2025-08-14T21:56:29.4536311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4536525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4536596Z return mod(**inputs) 2025-08-14T21:56:29.4536857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4536926Z outputs = self.model( 2025-08-14T21:56:29.4537192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4537263Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4537521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4537600Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4537836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4537922Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4538182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4538281Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4538561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4538655Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4538944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4539049Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4539053Z 2025-08-14T21:56:29.4539152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4539367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4539434Z return mod(**inputs) 2025-08-14T21:56:29.4539707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4539782Z outputs = self.model( 2025-08-14T21:56:29.4540042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4540134Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4540388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4540459Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4540679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4540757Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4541017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4541115Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4541368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4541459Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4541463Z 2025-08-14T21:56:29.4541563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4541753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4541827Z return mod(**inputs) 2025-08-14T21:56:29.4542086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4542162Z outputs = self.model( 2025-08-14T21:56:29.4542424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4542500Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4542776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4542851Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4543070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4543147Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4543399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4543513Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4543767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4543931Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4543943Z 2025-08-14T21:56:29.4544044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4544238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4544326Z return mod(**inputs) 2025-08-14T21:56:29.4544589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4544655Z outputs = self.model( 2025-08-14T21:56:29.4544920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4544992Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4545257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4545358Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4545576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4545673Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4545932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4546041Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4546313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4546392Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4546395Z 2025-08-14T21:56:29.4546499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4546690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4546758Z return mod(**inputs) 2025-08-14T21:56:29.4547017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4547083Z outputs = self.model( 2025-08-14T21:56:29.4547335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4547415Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4547671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4547745Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4547960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4548035Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4548301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4548409Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4548676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4548759Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4548764Z 2025-08-14T21:56:29.4548843Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4548929Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4549006Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4549081Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4549189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4549383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4549455Z return mod(**inputs) 2025-08-14T21:56:29.4549738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4549803Z outputs = self.model( 2025-08-14T21:56:29.4550071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4550142Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4550417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4550494Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4550708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4550791Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4551048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4551168Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4551438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4551549Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4551839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4551971Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4551975Z 2025-08-14T21:56:29.4552075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4552279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4552347Z return mod(**inputs) 2025-08-14T21:56:29.4552620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4552699Z outputs = self.model( 2025-08-14T21:56:29.4552975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4553056Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4553331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4553405Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4553641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4553723Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4554003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4554116Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4554390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4554501Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4554804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4554916Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4554928Z 2025-08-14T21:56:29.4555035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4555242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4555317Z return mod(**inputs) 2025-08-14T21:56:29.4555593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4555664Z outputs = self.model( 2025-08-14T21:56:29.4556017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4556121Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4556406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4556481Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4556733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4556824Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4557104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4557215Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4557507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4557588Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4557609Z 2025-08-14T21:56:29.4557719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4557921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4557991Z return mod(**inputs) 2025-08-14T21:56:29.4558274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4558346Z outputs = self.model( 2025-08-14T21:56:29.4558629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4558704Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4558979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4559061Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4559295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4559380Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4559665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4559791Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4559796Z 2025-08-14T21:56:29.4559909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4560116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4560184Z return mod(**inputs) 2025-08-14T21:56:29.4560469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4560539Z outputs = self.model( 2025-08-14T21:56:29.4560813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4560899Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4561173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4561255Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4561488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4561569Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4561852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4561977Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4562207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4562297Z return self.act(input) 2025-08-14T21:56:29.4562301Z 2025-08-14T21:56:29.4562411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4562623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4562692Z return mod(**inputs) 2025-08-14T21:56:29.4562966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4563061Z outputs = self.model( 2025-08-14T21:56:29.4563340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4563423Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4563699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4563772Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4564028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4564112Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4564409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4564496Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4564501Z 2025-08-14T21:56:29.4564609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4564824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4564893Z return mod(**inputs) 2025-08-14T21:56:29.4565175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4565250Z outputs = self.model( 2025-08-14T21:56:29.4565525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4565612Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4565884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4565959Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4566196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4566278Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4566552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:56:29.4566644Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4566648Z 2025-08-14T21:56:29.4566752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4566963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4567034Z return mod(**inputs) 2025-08-14T21:56:29.4567308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4567385Z outputs = self.model( 2025-08-14T21:56:29.4567682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4567763Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4568039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4568112Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4568345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4568425Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4568700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4568830Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4569099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4569264Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4569284Z 2025-08-14T21:56:29.4569392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4569607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4569682Z return mod(**inputs) 2025-08-14T21:56:29.4569956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4570031Z outputs = self.model( 2025-08-14T21:56:29.4570328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4570407Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4570714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4570790Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4571019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4571110Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4571383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4571497Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4571770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4571856Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4571861Z 2025-08-14T21:56:29.4571976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4572192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4572269Z return mod(**inputs) 2025-08-14T21:56:29.4572560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4572633Z outputs = self.model( 2025-08-14T21:56:29.4572921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4572998Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4573287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4573369Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4573605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4573691Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4573971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4574073Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4574363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4574452Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4574455Z 2025-08-14T21:56:29.4574545Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4574630Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4574710Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4574797Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4574914Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4575128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4575211Z return mod(**inputs) 2025-08-14T21:56:29.4575472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4575546Z outputs = self.model( 2025-08-14T21:56:29.4575846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4575921Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4576206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4576279Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4576509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4576616Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4576888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4577012Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4577284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4577387Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4577702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4577832Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4577835Z 2025-08-14T21:56:29.4577941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4578132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4578202Z return mod(**inputs) 2025-08-14T21:56:29.4578471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4578539Z outputs = self.model( 2025-08-14T21:56:29.4578802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4578883Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4579143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4579222Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4579445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4579525Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4579806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4579911Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4580182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4580288Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4580588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4580714Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4580717Z 2025-08-14T21:56:29.4580816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4581009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4581081Z return mod(**inputs) 2025-08-14T21:56:29.4581340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4581432Z outputs = self.model( 2025-08-14T21:56:29.4581694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4581766Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4582048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4582117Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4582334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4582417Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4582675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4582777Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4583051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4583132Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4583150Z 2025-08-14T21:56:29.4583259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4583455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4583529Z return mod(**inputs) 2025-08-14T21:56:29.4583792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4583863Z outputs = self.model( 2025-08-14T21:56:29.4584135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4584208Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4584475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4584558Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4584782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4584868Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4585136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4585246Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4585519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4585672Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4585676Z 2025-08-14T21:56:29.4585787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4585989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4586058Z return mod(**inputs) 2025-08-14T21:56:29.4586349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4586424Z outputs = self.model( 2025-08-14T21:56:29.4586705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4586793Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4587074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4587162Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4587398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4587500Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4587781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4587895Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4588171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4588267Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4588271Z 2025-08-14T21:56:29.4588371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4588571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4588635Z return mod(**inputs) 2025-08-14T21:56:29.4588892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4588969Z outputs = self.model( 2025-08-14T21:56:29.4589250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4589330Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4589603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4589676Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4589913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4589994Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4590279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4590391Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4590666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4590763Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4590767Z 2025-08-14T21:56:29.4590850Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4590932Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4591021Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4591102Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4591216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4591433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4591500Z return mod(**inputs) 2025-08-14T21:56:29.4591789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4591858Z outputs = self.model( 2025-08-14T21:56:29.4592174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4592261Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4592547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4592628Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4592860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4592943Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4593223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4593334Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4593614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4593746Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4594061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4594211Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4594215Z 2025-08-14T21:56:29.4594323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4594555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4594632Z return mod(**inputs) 2025-08-14T21:56:29.4594927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4595008Z outputs = self.model( 2025-08-14T21:56:29.4595302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4595388Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4595706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4595784Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4596255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4596355Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4596634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4596760Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4597049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4597158Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4597489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4597611Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4597616Z 2025-08-14T21:56:29.4597739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4597958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4598035Z return mod(**inputs) 2025-08-14T21:56:29.4598329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4598406Z outputs = self.model( 2025-08-14T21:56:29.4598685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4598773Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4599060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4599152Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4599384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4599470Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4599755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4599871Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4600155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4600243Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4600247Z 2025-08-14T21:56:29.4600357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4600571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4600659Z return mod(**inputs) 2025-08-14T21:56:29.4600937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4601016Z outputs = self.model( 2025-08-14T21:56:29.4601290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4601391Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4601663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4601736Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4601974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4602054Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4602359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4602486Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4602490Z 2025-08-14T21:56:29.4602610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4602827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4602898Z return mod(**inputs) 2025-08-14T21:56:29.4603171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4603250Z outputs = self.model( 2025-08-14T21:56:29.4603527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4603611Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4603887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4603963Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4604206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4604294Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4604588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4604718Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4604952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4605035Z return self.act(input) 2025-08-14T21:56:29.4605039Z 2025-08-14T21:56:29.4605147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4605354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4605436Z return mod(**inputs) 2025-08-14T21:56:29.4605712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4605810Z outputs = self.model( 2025-08-14T21:56:29.4606088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4606166Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4606449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4606526Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4606755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4606845Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4607121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4607233Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4607237Z 2025-08-14T21:56:29.4607343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4607548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4607641Z return mod(**inputs) 2025-08-14T21:56:29.4607918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4607995Z outputs = self.model( 2025-08-14T21:56:29.4608290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4608365Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4608860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4608992Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4609225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4609342Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4609618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4609739Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4609994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4610142Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4610146Z 2025-08-14T21:56:29.4610257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4610453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4610527Z return mod(**inputs) 2025-08-14T21:56:29.4610785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4610854Z outputs = self.model( 2025-08-14T21:56:29.4611129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4611207Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4611479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4611561Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4611789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4611879Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4612152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4612257Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4612537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4612620Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4612625Z 2025-08-14T21:56:29.4612743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4612946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4613014Z return mod(**inputs) 2025-08-14T21:56:29.4613290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4613362Z outputs = self.model( 2025-08-14T21:56:29.4613642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4613748Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4614009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4614085Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4614308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4614416Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4614696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4614798Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4615075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4615166Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4615171Z 2025-08-14T21:56:29.4615264Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4615350Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4615425Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4615518Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4615629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4615826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4615890Z return mod(**inputs) 2025-08-14T21:56:29.4616158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4616225Z outputs = self.model( 2025-08-14T21:56:29.4616490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4616561Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4616822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4616901Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4617119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4617203Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4617463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4617558Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4617824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4617921Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4618210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4618351Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4618355Z 2025-08-14T21:56:29.4618458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4618662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4618728Z return mod(**inputs) 2025-08-14T21:56:29.4618989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4619064Z outputs = self.model( 2025-08-14T21:56:29.4619323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4619403Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4619660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4619751Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4619979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4620058Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4620323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4620451Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4620708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4620813Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4621098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4621205Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4621225Z 2025-08-14T21:56:29.4621335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4621530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4621616Z return mod(**inputs) 2025-08-14T21:56:29.4621888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4621962Z outputs = self.model( 2025-08-14T21:56:29.4622245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4622321Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4622592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4622669Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4622889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4622971Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4623229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4623325Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4623593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4623674Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4623677Z 2025-08-14T21:56:29.4623783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4623980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4624044Z return mod(**inputs) 2025-08-14T21:56:29.4624312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4624380Z outputs = self.model( 2025-08-14T21:56:29.4624644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4624724Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4624983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4625063Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4625278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4625356Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4625632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-08-14T21:56:29.4625717Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4625763Z 2025-08-14T21:56:29.4625880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4626088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4626158Z return mod(**inputs) 2025-08-14T21:56:29.4626442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4626529Z outputs = self.model( 2025-08-14T21:56:29.4626800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4626882Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4627156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4627237Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4627490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4627569Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4627851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4627958Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4628237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4628403Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4628407Z 2025-08-14T21:56:29.4628515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4628727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4628794Z return mod(**inputs) 2025-08-14T21:56:29.4629071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4629150Z outputs = self.model( 2025-08-14T21:56:29.4629425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4629506Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4629790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4629864Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4630105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4630186Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4630466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4630586Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4630867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4630958Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4630964Z 2025-08-14T21:56:29.4631070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4631280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4631357Z return mod(**inputs) 2025-08-14T21:56:29.4631658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4631738Z outputs = self.model( 2025-08-14T21:56:29.4632037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4632113Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4632427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4632514Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4632745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4632835Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4633125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4633243Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4633521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4633611Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4633615Z 2025-08-14T21:56:29.4633707Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4633806Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4633894Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4633973Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4634077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4634312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4634383Z return mod(**inputs) 2025-08-14T21:56:29.4634667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4634746Z outputs = self.model( 2025-08-14T21:56:29.4635056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4635142Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4635436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4635518Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4635761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4635912Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4636205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4636329Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4636610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4636723Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4637044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4637181Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4637188Z 2025-08-14T21:56:29.4637305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4637509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4637586Z return mod(**inputs) 2025-08-14T21:56:29.4637870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4637943Z outputs = self.model( 2025-08-14T21:56:29.4638243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4638320Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4638604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4638688Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4638942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4639029Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4639304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4639417Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4639719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4639820Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4640135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4640247Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4640251Z 2025-08-14T21:56:29.4640357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4640592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4640662Z return mod(**inputs) 2025-08-14T21:56:29.4640966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4641046Z outputs = self.model( 2025-08-14T21:56:29.4641329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4641415Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4641698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4641775Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4642010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4642095Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4642367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4642486Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4642757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4642851Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4642854Z 2025-08-14T21:56:29.4642960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4643165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4643241Z return mod(**inputs) 2025-08-14T21:56:29.4643514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4643592Z outputs = self.model( 2025-08-14T21:56:29.4643867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4643943Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4644225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4644300Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4644529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4644618Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4644893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4645025Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4645029Z 2025-08-14T21:56:29.4645161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4645367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4645442Z return mod(**inputs) 2025-08-14T21:56:29.4645718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4645812Z outputs = self.model( 2025-08-14T21:56:29.4646092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4646166Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4646452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4646520Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4646739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4646837Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4647094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4647232Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4647444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4647515Z return self.act(input) 2025-08-14T21:56:29.4647518Z 2025-08-14T21:56:29.4647629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4647821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4647893Z return mod(**inputs) 2025-08-14T21:56:29.4648150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4648215Z outputs = self.model( 2025-08-14T21:56:29.4648484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4648557Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4648815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4648896Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4649114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4649200Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4649462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4649544Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4649548Z 2025-08-14T21:56:29.4649654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4649851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4649916Z return mod(**inputs) 2025-08-14T21:56:29.4650186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4650254Z outputs = self.model( 2025-08-14T21:56:29.4650523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4650594Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4650852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4650933Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4651150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4651255Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4651512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4651610Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4651875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4652042Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4652046Z 2025-08-14T21:56:29.4652153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4652346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4652412Z return mod(**inputs) 2025-08-14T21:56:29.4652677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4652746Z outputs = self.model( 2025-08-14T21:56:29.4653020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4653101Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4653375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4653455Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4653672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4653749Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4654013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4654112Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4654370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4654461Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4654465Z 2025-08-14T21:56:29.4654565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4654767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4654835Z return mod(**inputs) 2025-08-14T21:56:29.4655091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4655165Z outputs = self.model( 2025-08-14T21:56:29.4655423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4655502Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4655774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4655853Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4656089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4656184Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4656441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4656549Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4656807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4656899Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4656902Z 2025-08-14T21:56:29.4656982Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4657060Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4657145Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4657237Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4657339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4657541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4657608Z return mod(**inputs) 2025-08-14T21:56:29.4657877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4657974Z outputs = self.model( 2025-08-14T21:56:29.4658233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4658314Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4658572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4658642Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4658889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4658967Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4659247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4659343Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4659599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4659702Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4659989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4660127Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4660131Z 2025-08-14T21:56:29.4660233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4660433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4660506Z return mod(**inputs) 2025-08-14T21:56:29.4660768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4660841Z outputs = self.model( 2025-08-14T21:56:29.4661100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4661174Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4661439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4661508Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4661724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4661809Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4662067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4662181Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4662431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4662524Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4662808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4662917Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4662921Z 2025-08-14T21:56:29.4663028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4663222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4663309Z return mod(**inputs) 2025-08-14T21:56:29.4663579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4663645Z outputs = self.model( 2025-08-14T21:56:29.4663898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4663992Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4664242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4664319Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4664528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4664602Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4664873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4664970Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4665245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4665334Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4665339Z 2025-08-14T21:56:29.4665443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4665641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4665704Z return mod(**inputs) 2025-08-14T21:56:29.4665964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4666039Z outputs = self.model( 2025-08-14T21:56:29.4666296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4666379Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4666640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4666716Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4666953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4667037Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4667314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4667428Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4667684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4667842Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4667848Z 2025-08-14T21:56:29.4667949Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4668140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4668214Z return mod(**inputs) 2025-08-14T21:56:29.4668479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4668551Z outputs = self.model( 2025-08-14T21:56:29.4668799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4668867Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4669130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4669200Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4669436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4669521Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4669780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4669890Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4670168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4670248Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4670251Z 2025-08-14T21:56:29.4670360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4670557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4670629Z return mod(**inputs) 2025-08-14T21:56:29.4670900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4670970Z outputs = self.model( 2025-08-14T21:56:29.4671262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4671338Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4671617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4671700Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4671931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4672019Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4672292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4672403Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4672692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4672782Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4672787Z 2025-08-14T21:56:29.4672877Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4672960Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4673040Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4673129Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4673233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4673440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4673514Z return mod(**inputs) 2025-08-14T21:56:29.4673810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4673881Z outputs = self.model( 2025-08-14T21:56:29.4674175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4674250Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4674535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4674609Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4674840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4674929Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4675203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4675318Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4675598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4675718Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4676153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4676301Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4676326Z 2025-08-14T21:56:29.4676446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4676660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4676742Z return mod(**inputs) 2025-08-14T21:56:29.4677033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4677105Z outputs = self.model( 2025-08-14T21:56:29.4677424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4677506Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4677773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4677853Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4678064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4678142Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4678399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4678505Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4678787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4678900Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4679222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4679347Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4679350Z 2025-08-14T21:56:29.4679460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4679673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4679751Z return mod(**inputs) 2025-08-14T21:56:29.4680032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4680111Z outputs = self.model( 2025-08-14T21:56:29.4680409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4680487Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4680783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4680859Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4681095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4681187Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4681471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4681596Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4681879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4681965Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4681969Z 2025-08-14T21:56:29.4682088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4682332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4682410Z return mod(**inputs) 2025-08-14T21:56:29.4682706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4682780Z outputs = self.model( 2025-08-14T21:56:29.4683090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4683167Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4683457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4683540Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4683776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4683867Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4684169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 433, in forward 2025-08-14T21:56:29.4684250Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4684269Z 2025-08-14T21:56:29.4684378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4684577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4684648Z return mod(**inputs) 2025-08-14T21:56:29.4684907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4684974Z outputs = self.model( 2025-08-14T21:56:29.4685246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4685315Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4685576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4685673Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4685892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4685976Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4686236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4686355Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4686358Z 2025-08-14T21:56:29.4686468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4686664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4686738Z return mod(**inputs) 2025-08-14T21:56:29.4687004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4687072Z outputs = self.model( 2025-08-14T21:56:29.4687345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4687416Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4687681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4687761Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4687977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4688060Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4688319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4688470Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4688691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4688761Z return self.act(input) 2025-08-14T21:56:29.4688766Z 2025-08-14T21:56:29.4688866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4689085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4689150Z return mod(**inputs) 2025-08-14T21:56:29.4689417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4689482Z outputs = self.model( 2025-08-14T21:56:29.4689744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4689826Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4690113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4690191Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4690427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4690506Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4690771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4690851Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4690854Z 2025-08-14T21:56:29.4690951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4691152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4691215Z return mod(**inputs) 2025-08-14T21:56:29.4691486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4691553Z outputs = self.model( 2025-08-14T21:56:29.4691815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4691894Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4692160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4692237Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4692456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4692532Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4692801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4692901Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4693165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4693321Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4693326Z 2025-08-14T21:56:29.4693425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4693630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4693696Z return mod(**inputs) 2025-08-14T21:56:29.4693960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4694035Z outputs = self.model( 2025-08-14T21:56:29.4694299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4694377Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4694653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4694724Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4694947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4695024Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4695298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4695404Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4695662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4695748Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4695751Z 2025-08-14T21:56:29.4695853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4696062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4696136Z return mod(**inputs) 2025-08-14T21:56:29.4696411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4696480Z outputs = self.model( 2025-08-14T21:56:29.4696746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4696826Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4697078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4697146Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4697354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4697437Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4697693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4697798Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4698056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4698141Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4698144Z 2025-08-14T21:56:29.4698232Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4698310Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4698386Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4698468Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4698567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4698763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4698838Z return mod(**inputs) 2025-08-14T21:56:29.4699095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4699169Z outputs = self.model( 2025-08-14T21:56:29.4699428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4699510Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4699768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4699836Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4700066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4700139Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4700383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4700501Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4700747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4700844Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4701130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4701255Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4701259Z 2025-08-14T21:56:29.4701361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4701545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4701606Z return mod(**inputs) 2025-08-14T21:56:29.4701883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4701950Z outputs = self.model( 2025-08-14T21:56:29.4702225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4702295Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4702541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4702615Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4702820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4702903Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4703152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4703247Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4703518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4703609Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4703885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4703996Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4703999Z 2025-08-14T21:56:29.4704095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4704284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4704346Z return mod(**inputs) 2025-08-14T21:56:29.4704589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4704661Z outputs = self.model( 2025-08-14T21:56:29.4704907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4704982Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4705226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4705295Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4705507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4705579Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4705822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4705919Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4706167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4706268Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4706271Z 2025-08-14T21:56:29.4706367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4706553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4706624Z return mod(**inputs) 2025-08-14T21:56:29.4706896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4706962Z outputs = self.model( 2025-08-14T21:56:29.4707227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4707295Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4707547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4707628Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4707834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4707914Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4708174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4708286Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4708543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4708836Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4708843Z 2025-08-14T21:56:29.4708956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4709151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4709228Z return mod(**inputs) 2025-08-14T21:56:29.4709494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4709562Z outputs = self.model( 2025-08-14T21:56:29.4709833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4709907Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4710313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4710398Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4710627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4710718Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4710994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4711110Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4711393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4711479Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4711485Z 2025-08-14T21:56:29.4711602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4711805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4711872Z return mod(**inputs) 2025-08-14T21:56:29.4712152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4712222Z outputs = self.model( 2025-08-14T21:56:29.4735844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4736295Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4736648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4736732Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4736980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4737136Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4737411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4737537Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4737807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4737895Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4737911Z 2025-08-14T21:56:29.4738034Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4738113Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4738195Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4738309Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4738422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4738646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4738718Z return mod(**inputs) 2025-08-14T21:56:29.4738986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4739066Z outputs = self.model( 2025-08-14T21:56:29.4739326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4739411Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4739666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4739737Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4739967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4740050Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4740316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4740423Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4740678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4740784Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4741069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4741203Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4741217Z 2025-08-14T21:56:29.4741324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4741528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4741604Z return mod(**inputs) 2025-08-14T21:56:29.4741865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4741935Z outputs = self.model( 2025-08-14T21:56:29.4742206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4742281Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4742548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4742647Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4742865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4742955Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4743215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4743338Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4743602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4743710Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4743985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4744087Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4744107Z 2025-08-14T21:56:29.4744209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4744405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4744512Z return mod(**inputs) 2025-08-14T21:56:29.4744765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4744832Z outputs = self.model( 2025-08-14T21:56:29.4745075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4745153Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4745401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4745472Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4745695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4745774Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4746031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4746133Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4746385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4746472Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4746475Z 2025-08-14T21:56:29.4746573Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4746772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4746847Z return mod(**inputs) 2025-08-14T21:56:29.4747093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4747166Z outputs = self.model( 2025-08-14T21:56:29.4747411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4747480Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4747737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4747805Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4748018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4748092Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4748335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4748456Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4748477Z 2025-08-14T21:56:29.4748579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4748779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4748845Z return mod(**inputs) 2025-08-14T21:56:29.4749098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4749187Z outputs = self.model( 2025-08-14T21:56:29.4749441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4749510Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4749772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4749842Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4750074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4750151Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4750415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4750542Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4750759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4750829Z return self.act(input) 2025-08-14T21:56:29.4750840Z 2025-08-14T21:56:29.4750941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4751140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4751213Z return mod(**inputs) 2025-08-14T21:56:29.4751478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4751547Z outputs = self.model( 2025-08-14T21:56:29.4751820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4751894Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4752164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4752237Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4752456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4752544Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4752802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4752883Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4752888Z 2025-08-14T21:56:29.4752998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4753205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4753281Z return mod(**inputs) 2025-08-14T21:56:29.4753559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4753634Z outputs = self.model( 2025-08-14T21:56:29.4753918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4753994Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4754279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4754352Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4754587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4754696Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4754977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:56:29.4755061Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4755081Z 2025-08-14T21:56:29.4755198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4755403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4755478Z return mod(**inputs) 2025-08-14T21:56:29.4755752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4755907Z outputs = self.model( 2025-08-14T21:56:29.4756201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4756303Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4756590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4756692Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4756936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4757032Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4757309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4757416Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4757699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4757861Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4757867Z 2025-08-14T21:56:29.4757984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4758193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4758260Z return mod(**inputs) 2025-08-14T21:56:29.4758521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4758590Z outputs = self.model( 2025-08-14T21:56:29.4758866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4758952Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4759226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4759310Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4759542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4759626Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4759909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4760014Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4760298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4760383Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4760387Z 2025-08-14T21:56:29.4760493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4760707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4760775Z return mod(**inputs) 2025-08-14T21:56:29.4761051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4761158Z outputs = self.model( 2025-08-14T21:56:29.4761438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4761523Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4761801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4761892Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4762130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4762214Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4762497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4762600Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4762893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4762994Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4762998Z 2025-08-14T21:56:29.4763099Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4763186Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4763278Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4763362Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4763481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4763691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4763762Z return mod(**inputs) 2025-08-14T21:56:29.4764048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4764121Z outputs = self.model( 2025-08-14T21:56:29.4764406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4764493Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4764774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4764861Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4765103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4765189Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4765471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4765578Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4765857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4765974Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4766290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4766443Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4766448Z 2025-08-14T21:56:29.4766560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4766772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4766852Z return mod(**inputs) 2025-08-14T21:56:29.4767132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4767211Z outputs = self.model( 2025-08-14T21:56:29.4767493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4767595Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4767852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4767922Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4768131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4768232Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4768485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4768586Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4768844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4768946Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4769260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4769371Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4769389Z 2025-08-14T21:56:29.4769502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4769698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4769764Z return mod(**inputs) 2025-08-14T21:56:29.4770029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4770098Z outputs = self.model( 2025-08-14T21:56:29.4770361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4770441Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4770694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4770772Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4770985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4771061Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4771323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4771417Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4771675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4771755Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4771758Z 2025-08-14T21:56:29.4771856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4772062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4772127Z return mod(**inputs) 2025-08-14T21:56:29.4772388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4772465Z outputs = self.model( 2025-08-14T21:56:29.4772728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4772808Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4773065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4773137Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4773364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4773445Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4773748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4773860Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4774135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4774322Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4774326Z 2025-08-14T21:56:29.4774434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4774645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4774717Z return mod(**inputs) 2025-08-14T21:56:29.4774975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4775049Z outputs = self.model( 2025-08-14T21:56:29.4775325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4775407Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4775682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4775762Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4775979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4776056Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4776321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4776426Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4776687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4776775Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4776778Z 2025-08-14T21:56:29.4776879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4777085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4777152Z return mod(**inputs) 2025-08-14T21:56:29.4777412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4777487Z outputs = self.model( 2025-08-14T21:56:29.4777746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4777826Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4778084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4778158Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4778379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4778461Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4778718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4778836Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4779091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4779180Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4779184Z 2025-08-14T21:56:29.4779261Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4779339Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4779420Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4779515Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4779618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4779817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4779882Z return mod(**inputs) 2025-08-14T21:56:29.4780146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4780239Z outputs = self.model( 2025-08-14T21:56:29.4780500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4780577Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4780839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4780918Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4781156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4781234Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4781514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4781622Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4781879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4781982Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4782273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4782406Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4782410Z 2025-08-14T21:56:29.4782508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4782700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4782769Z return mod(**inputs) 2025-08-14T21:56:29.4783021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4783092Z outputs = self.model( 2025-08-14T21:56:29.4783342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4783411Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4783666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4783735Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4783946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4784033Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4784284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4784395Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4784646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4784741Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4785027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4785130Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4785137Z 2025-08-14T21:56:29.4785245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4785438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4785521Z return mod(**inputs) 2025-08-14T21:56:29.4785789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4785855Z outputs = self.model( 2025-08-14T21:56:29.4786116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4786212Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4786476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4786558Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4786780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4786862Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4787149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4787256Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4787527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4787615Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4787620Z 2025-08-14T21:56:29.4787722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4787922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4787986Z return mod(**inputs) 2025-08-14T21:56:29.4788245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4788319Z outputs = self.model( 2025-08-14T21:56:29.4788576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4788658Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4788917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4788989Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4789211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4789291Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4789552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4789687Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4789691Z 2025-08-14T21:56:29.4789797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4790012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4790084Z return mod(**inputs) 2025-08-14T21:56:29.4790361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4790440Z outputs = self.model( 2025-08-14T21:56:29.4790717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4790801Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4791074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4791149Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4791387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4791469Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4791744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4791894Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4792117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4792197Z return self.act(input) 2025-08-14T21:56:29.4792216Z 2025-08-14T21:56:29.4792325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4792541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4792619Z return mod(**inputs) 2025-08-14T21:56:29.4792894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4792971Z outputs = self.model( 2025-08-14T21:56:29.4793243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4793336Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4793624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4793715Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4793954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4794047Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4794320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4794412Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4794417Z 2025-08-14T21:56:29.4794525Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4794743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4794821Z return mod(**inputs) 2025-08-14T21:56:29.4795098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4795169Z outputs = self.model( 2025-08-14T21:56:29.4795450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4795529Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4795907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4795994Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4796245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4796340Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4796629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4796749Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4797035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4797202Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4797207Z 2025-08-14T21:56:29.4797327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4797551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4797622Z return mod(**inputs) 2025-08-14T21:56:29.4797915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4797989Z outputs = self.model( 2025-08-14T21:56:29.4798285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4798380Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4798647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4798730Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4798951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4799054Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4799311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4799408Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4799672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4799751Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4799756Z 2025-08-14T21:56:29.4799876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4800082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4800164Z return mod(**inputs) 2025-08-14T21:56:29.4800433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4800503Z outputs = self.model( 2025-08-14T21:56:29.4800769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4800850Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4801115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4801191Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4801416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4801497Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4801766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4801864Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4802127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4802221Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4802224Z 2025-08-14T21:56:29.4802304Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4802390Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4802467Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4802542Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4802652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4802851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4802916Z return mod(**inputs) 2025-08-14T21:56:29.4803188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4803254Z outputs = self.model( 2025-08-14T21:56:29.4803527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4803598Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4803861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4803940Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4804163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4804244Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4804525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4804623Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4804893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4805016Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4805325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4805475Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4805479Z 2025-08-14T21:56:29.4805585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4805798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4805869Z return mod(**inputs) 2025-08-14T21:56:29.4806168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4806253Z outputs = self.model( 2025-08-14T21:56:29.4806525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4806600Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4806862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4806933Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4807156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4807235Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4807512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4807624Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4807897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4808008Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4808325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4808441Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4808445Z 2025-08-14T21:56:29.4808559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4808919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4808996Z return mod(**inputs) 2025-08-14T21:56:29.4809284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4809356Z outputs = self.model( 2025-08-14T21:56:29.4809638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4809718Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4809989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4810074Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4810311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4810400Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4810680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4810783Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4811108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4811194Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4811198Z 2025-08-14T21:56:29.4811306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4811520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4811986Z return mod(**inputs) 2025-08-14T21:56:29.4812272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4812342Z outputs = self.model( 2025-08-14T21:56:29.4812616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4812698Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4812997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4813082Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4813344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4813428Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4813708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-08-14T21:56:29.4813792Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4813796Z 2025-08-14T21:56:29.4813903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4814115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4814183Z return mod(**inputs) 2025-08-14T21:56:29.4814462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4814535Z outputs = self.model( 2025-08-14T21:56:29.4814807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4814888Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4815145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4815217Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4815440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4815518Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4815783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4815891Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4816156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4816309Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4816314Z 2025-08-14T21:56:29.4816412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4816610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4816673Z return mod(**inputs) 2025-08-14T21:56:29.4816923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4816998Z outputs = self.model( 2025-08-14T21:56:29.4817250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4817320Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4817598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4817668Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4817887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4817962Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4818228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4818338Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4818586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4818669Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4818673Z 2025-08-14T21:56:29.4818772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4818975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4819047Z return mod(**inputs) 2025-08-14T21:56:29.4819316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4819382Z outputs = self.model( 2025-08-14T21:56:29.4819645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4819715Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4819976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4820043Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4820255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4820338Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4820595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4820705Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4820960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4821043Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4821046Z 2025-08-14T21:56:29.4821130Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4821206Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4821280Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4821360Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4821461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4821656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4821721Z return mod(**inputs) 2025-08-14T21:56:29.4821980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4822052Z outputs = self.model( 2025-08-14T21:56:29.4822308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4822379Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4822645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4822714Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4822934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4823009Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4823263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4823387Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4823643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4823744Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4824038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4824166Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4824170Z 2025-08-14T21:56:29.4824273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4824460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4824524Z return mod(**inputs) 2025-08-14T21:56:29.4824807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4824874Z outputs = self.model( 2025-08-14T21:56:29.4825146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4825217Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4825473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4825550Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4825761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4825843Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4826095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4826196Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4826461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4826554Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4826835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4826946Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4826949Z 2025-08-14T21:56:29.4827046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4827242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4827304Z return mod(**inputs) 2025-08-14T21:56:29.4827557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4827632Z outputs = self.model( 2025-08-14T21:56:29.4827890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4827967Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4828219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4828288Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4828509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4828584Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4828837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4828946Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4829202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4829302Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4829306Z 2025-08-14T21:56:29.4829403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4829595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4829687Z return mod(**inputs) 2025-08-14T21:56:29.4829940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4830010Z outputs = self.model( 2025-08-14T21:56:29.4830260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4830329Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4830585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4830669Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4830887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4830974Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4831248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4831378Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4831382Z 2025-08-14T21:56:29.4831483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4831679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4831751Z return mod(**inputs) 2025-08-14T21:56:29.4832018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4832088Z outputs = self.model( 2025-08-14T21:56:29.4832348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4832418Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4832682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4832754Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4832963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4833049Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4833304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4833428Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4833636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4833708Z return self.act(input) 2025-08-14T21:56:29.4833712Z 2025-08-14T21:56:29.4833818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4834014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4834077Z return mod(**inputs) 2025-08-14T21:56:29.4834346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4834417Z outputs = self.model( 2025-08-14T21:56:29.4834697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4834772Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4835042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4835142Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4835371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4835460Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4835734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4835913Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4835920Z 2025-08-14T21:56:29.4836046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4836256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4836325Z return mod(**inputs) 2025-08-14T21:56:29.4836609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4836681Z outputs = self.model( 2025-08-14T21:56:29.4836988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4837066Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4837367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4837462Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4837681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4837768Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4838029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4838127Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4838392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4838546Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4838550Z 2025-08-14T21:56:29.4838659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4838858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4838924Z return mod(**inputs) 2025-08-14T21:56:29.4839190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4839256Z outputs = self.model( 2025-08-14T21:56:29.4839517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4839595Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4839855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4839933Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4840149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4840224Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4840489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4840591Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4840846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4840932Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4840936Z 2025-08-14T21:56:29.4841037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4841237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4841324Z return mod(**inputs) 2025-08-14T21:56:29.4841585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4841661Z outputs = self.model( 2025-08-14T21:56:29.4841920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4842013Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4842269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4842340Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4842565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4842642Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4842896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4843042Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4843300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4843405Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4843408Z 2025-08-14T21:56:29.4843490Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4843568Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4843652Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4843726Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4843829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4844030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4844093Z return mod(**inputs) 2025-08-14T21:56:29.4844364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4844432Z outputs = self.model( 2025-08-14T21:56:29.4844689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4844770Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4845027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4845098Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4845324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4845403Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4845666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4845763Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4846023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4846126Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4846413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4846554Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4846558Z 2025-08-14T21:56:29.4846659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4846854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4846923Z return mod(**inputs) 2025-08-14T21:56:29.4847180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4847253Z outputs = self.model( 2025-08-14T21:56:29.4847528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4847601Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4847865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4847960Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4848169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4848254Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4848504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4848603Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4848850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4848959Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4849262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4849369Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4849375Z 2025-08-14T21:56:29.4849483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4849673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4849736Z return mod(**inputs) 2025-08-14T21:56:29.4849995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4850060Z outputs = self.model( 2025-08-14T21:56:29.4850310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4850393Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4850644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4850721Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4850931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4851011Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4851271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4851366Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4851619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4851705Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4851710Z 2025-08-14T21:56:29.4851810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4852008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4852072Z return mod(**inputs) 2025-08-14T21:56:29.4852322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4852398Z outputs = self.model( 2025-08-14T21:56:29.4852647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4852725Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4852975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4853044Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4853260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4853359Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4853610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4853720Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4853985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4854136Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4854139Z 2025-08-14T21:56:29.4854236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4854425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4854493Z return mod(**inputs) 2025-08-14T21:56:29.4854756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4854827Z outputs = self.model( 2025-08-14T21:56:29.4855095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4855166Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4855427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4855499Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4855712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4855795Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4856050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4856158Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4856417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4856498Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4856501Z 2025-08-14T21:56:29.4856612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4856805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4856879Z return mod(**inputs) 2025-08-14T21:56:29.4857148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4857210Z outputs = self.model( 2025-08-14T21:56:29.4857474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4857541Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4857800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4857868Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4858078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4858160Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4858412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4858513Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4858769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4858852Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4858855Z 2025-08-14T21:56:29.4858936Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4859030Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4859105Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4859187Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4859285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4859474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4859558Z return mod(**inputs) 2025-08-14T21:56:29.4859815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4859887Z outputs = self.model( 2025-08-14T21:56:29.4860140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4860211Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4860468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4860554Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4860773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4860849Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4861117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4861230Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4861482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4861576Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4861861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4861988Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4861993Z 2025-08-14T21:56:29.4862099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4862288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4862351Z return mod(**inputs) 2025-08-14T21:56:29.4862610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4862676Z outputs = self.model( 2025-08-14T21:56:29.4862938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4863009Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4863280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4863362Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4863593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4863685Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4863954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4864059Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4864331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4864432Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4864734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4864855Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4864859Z 2025-08-14T21:56:29.4864964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4865197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4865266Z return mod(**inputs) 2025-08-14T21:56:29.4865551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4865628Z outputs = self.model( 2025-08-14T21:56:29.4865921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4865999Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4866283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4866358Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4866594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4866678Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4866967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4867090Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4867378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4867470Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4867474Z 2025-08-14T21:56:29.4867580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4867786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4867861Z return mod(**inputs) 2025-08-14T21:56:29.4868134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4868203Z outputs = self.model( 2025-08-14T21:56:29.4868487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4868563Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4868844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4868922Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4869149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4869240Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4869513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 433, in forward 2025-08-14T21:56:29.4869596Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4869607Z 2025-08-14T21:56:29.4869713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4869923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4869999Z return mod(**inputs) 2025-08-14T21:56:29.4870274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4870345Z outputs = self.model( 2025-08-14T21:56:29.4870618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4870688Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4870943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4871013Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4871227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4871313Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4871590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4871708Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4871721Z 2025-08-14T21:56:29.4871823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4872031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4872102Z return mod(**inputs) 2025-08-14T21:56:29.4872368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4872438Z outputs = self.model( 2025-08-14T21:56:29.4872719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4872794Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4873094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4873170Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4873425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4873518Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4873790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4873912Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4874141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4874213Z return self.act(input) 2025-08-14T21:56:29.4874217Z 2025-08-14T21:56:29.4874329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4874540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4874608Z return mod(**inputs) 2025-08-14T21:56:29.4874889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4874959Z outputs = self.model( 2025-08-14T21:56:29.4875231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4875316Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4875596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4875679Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4875995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4876085Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4876383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4876474Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4876478Z 2025-08-14T21:56:29.4876599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4876814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4876886Z return mod(**inputs) 2025-08-14T21:56:29.4877175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4877256Z outputs = self.model( 2025-08-14T21:56:29.4877516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4877595Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4877857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4877963Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4878184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4878263Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4878548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4878648Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4878918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4879062Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4879066Z 2025-08-14T21:56:29.4879164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4879379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4879443Z return mod(**inputs) 2025-08-14T21:56:29.4879707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4879779Z outputs = self.model( 2025-08-14T21:56:29.4880033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4880109Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4880359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4880427Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4880649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4880725Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4880992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4881098Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4881347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4881433Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4881436Z 2025-08-14T21:56:29.4881533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4881721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4881790Z return mod(**inputs) 2025-08-14T21:56:29.4882041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4882112Z outputs = self.model( 2025-08-14T21:56:29.4882371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4882443Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4882711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4882781Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4883014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4883090Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4883341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4883442Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4883698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4883807Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4883812Z 2025-08-14T21:56:29.4883900Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4883977Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4884062Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4884139Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4884257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4884463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4884527Z return mod(**inputs) 2025-08-14T21:56:29.4884800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4884877Z outputs = self.model( 2025-08-14T21:56:29.4885126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4885219Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4885475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4885564Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4885788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4885868Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4886125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4886229Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4886484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4886601Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4886880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4887007Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4887011Z 2025-08-14T21:56:29.4887120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4887309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4887381Z return mod(**inputs) 2025-08-14T21:56:29.4887634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4887698Z outputs = self.model( 2025-08-14T21:56:29.4887962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4888034Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4888294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4888371Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4888587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4888672Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4888928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4889026Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4889294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4889392Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4889683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4889812Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4889815Z 2025-08-14T21:56:29.4889916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4890121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4890187Z return mod(**inputs) 2025-08-14T21:56:29.4890465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4890539Z outputs = self.model( 2025-08-14T21:56:29.4890795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4890874Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4891131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4891205Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4891442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4891523Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4891821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4891920Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4892180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4892268Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4892271Z 2025-08-14T21:56:29.4892374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4892570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4892644Z return mod(**inputs) 2025-08-14T21:56:29.4892902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4892975Z outputs = self.model( 2025-08-14T21:56:29.4893240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4893313Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4893582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4893651Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4893876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4893952Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4894208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4894323Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4894584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4894738Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4894751Z 2025-08-14T21:56:29.4894854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4895049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4895121Z return mod(**inputs) 2025-08-14T21:56:29.4895381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4895447Z outputs = self.model( 2025-08-14T21:56:29.4895714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4895803Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4896075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4896146Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4896367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4896469Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4896728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4896834Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4897100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4897179Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4897184Z 2025-08-14T21:56:29.4897318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4897517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4897581Z return mod(**inputs) 2025-08-14T21:56:29.4897864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4897933Z outputs = self.model( 2025-08-14T21:56:29.4898202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4898273Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4898528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4898604Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4898820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4898898Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4899164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4899271Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4899538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4899623Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4899627Z 2025-08-14T21:56:29.4899704Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4899788Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4899862Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4899937Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4900046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4900246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4900317Z return mod(**inputs) 2025-08-14T21:56:29.4900579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4900645Z outputs = self.model( 2025-08-14T21:56:29.4900912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4900984Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4901240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4901319Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4901533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4901618Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4901908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4902018Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4902297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4902418Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4902733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4902873Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4902876Z 2025-08-14T21:56:29.4902988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4903192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4903258Z return mod(**inputs) 2025-08-14T21:56:29.4903532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4903610Z outputs = self.model( 2025-08-14T21:56:29.4903882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4903961Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4904223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4904293Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4904516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4904591Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4904863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4904968Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4905231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4905334Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4905626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4905730Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4905741Z 2025-08-14T21:56:29.4905843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4906052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4906128Z return mod(**inputs) 2025-08-14T21:56:29.4906410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4906482Z outputs = self.model( 2025-08-14T21:56:29.4906771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4906847Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4907131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4907206Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4907452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4907542Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4907821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4907941Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4908225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4908306Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4908310Z 2025-08-14T21:56:29.4908421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4908627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4908958Z return mod(**inputs) 2025-08-14T21:56:29.4909248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4909319Z outputs = self.model( 2025-08-14T21:56:29.4909599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4909673Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4909973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4910058Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4910321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4910405Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4910692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4910818Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4910822Z 2025-08-14T21:56:29.4910938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4911148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4911218Z return mod(**inputs) 2025-08-14T21:56:29.4911500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4911572Z outputs = self.model( 2025-08-14T21:56:29.4911853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4911928Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4912199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4912281Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4912518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4912601Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4912882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4913006Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4913237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4913309Z return self.act(input) 2025-08-14T21:56:29.4913312Z 2025-08-14T21:56:29.4913421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4913635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4913704Z return mod(**inputs) 2025-08-14T21:56:29.4913977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4914054Z outputs = self.model( 2025-08-14T21:56:29.4914326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4914409Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4914685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4914787Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4915035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4915117Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4915415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4915502Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4915506Z 2025-08-14T21:56:29.4915613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4915876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4915952Z return mod(**inputs) 2025-08-14T21:56:29.4916231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4916330Z outputs = self.model( 2025-08-14T21:56:29.4916604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4916706Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4916980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4917056Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4917295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4917376Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4917659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:56:29.4917745Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4917750Z 2025-08-14T21:56:29.4917861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4918084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4918154Z return mod(**inputs) 2025-08-14T21:56:29.4918428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4918507Z outputs = self.model( 2025-08-14T21:56:29.4918780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4918863Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4919138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4919212Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4919449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4919534Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4919819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4919924Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4920172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4920327Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4920330Z 2025-08-14T21:56:29.4920429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4920619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4920689Z return mod(**inputs) 2025-08-14T21:56:29.4920940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4921031Z outputs = self.model( 2025-08-14T21:56:29.4921282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4921353Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4921613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4921706Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4921921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4922004Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4922261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4922364Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4922663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4922741Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4922745Z 2025-08-14T21:56:29.4922868Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4923066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4923139Z return mod(**inputs) 2025-08-14T21:56:29.4923397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4923463Z outputs = self.model( 2025-08-14T21:56:29.4923729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4923801Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4924059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4924137Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4924355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4924439Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4924708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4924803Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4925063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4925144Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4925147Z 2025-08-14T21:56:29.4925257Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4925334Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4925410Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4925488Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4925584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4925774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4925845Z return mod(**inputs) 2025-08-14T21:56:29.4926101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4926172Z outputs = self.model( 2025-08-14T21:56:29.4926421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4926490Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4926748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4926835Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4927048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4927132Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4927384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4927503Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4927760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4927857Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4928151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4928280Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4928286Z 2025-08-14T21:56:29.4928409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4928606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4928670Z return mod(**inputs) 2025-08-14T21:56:29.4928949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4929020Z outputs = self.model( 2025-08-14T21:56:29.4929276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4929356Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4929611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4929690Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4929912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4929992Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4930257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4930355Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4930609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4930715Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4931001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4931123Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4931126Z 2025-08-14T21:56:29.4931233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4931444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4931524Z return mod(**inputs) 2025-08-14T21:56:29.4931794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4931868Z outputs = self.model( 2025-08-14T21:56:29.4932123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4932196Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4932459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4932529Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4932744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4932833Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4933130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4933238Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4933512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4933614Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4933617Z 2025-08-14T21:56:29.4933732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4933941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4934017Z return mod(**inputs) 2025-08-14T21:56:29.4934294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4934365Z outputs = self.model( 2025-08-14T21:56:29.4934674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4934760Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4935041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4935120Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4935338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4935422Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4935681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4935787Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4936060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4936207Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4936210Z 2025-08-14T21:56:29.4936313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4936506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4936569Z return mod(**inputs) 2025-08-14T21:56:29.4936835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4936902Z outputs = self.model( 2025-08-14T21:56:29.4937158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4937236Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4937495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4937574Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4937792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4937870Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4938132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4938240Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4938505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4938583Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4938586Z 2025-08-14T21:56:29.4938684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4938885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4938969Z return mod(**inputs) 2025-08-14T21:56:29.4939229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4939301Z outputs = self.model( 2025-08-14T21:56:29.4939564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4939679Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4939937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4940005Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4940231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4940306Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4940572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4940695Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4940954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4941061Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4941065Z 2025-08-14T21:56:29.4941146Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4941223Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4941305Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4941379Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4941486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4941682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4941747Z return mod(**inputs) 2025-08-14T21:56:29.4942018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4942086Z outputs = self.model( 2025-08-14T21:56:29.4942342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4942421Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4942679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4942757Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4942975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4943052Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4943317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4943423Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4943684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4943791Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4944079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4944220Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4944223Z 2025-08-14T21:56:29.4944326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4944524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4944598Z return mod(**inputs) 2025-08-14T21:56:29.4944858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4944932Z outputs = self.model( 2025-08-14T21:56:29.4945219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4945292Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4945562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4945651Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4945913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4946002Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4946278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4946397Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4946672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4946793Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4947130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4947236Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4947241Z 2025-08-14T21:56:29.4947349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4947545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4947610Z return mod(**inputs) 2025-08-14T21:56:29.4947875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4947941Z outputs = self.model( 2025-08-14T21:56:29.4948201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4948282Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4948541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4948617Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4948832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4948911Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4949176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4949280Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4949547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4949629Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4949634Z 2025-08-14T21:56:29.4949736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4949937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4950003Z return mod(**inputs) 2025-08-14T21:56:29.4950262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4950341Z outputs = self.model( 2025-08-14T21:56:29.4950611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4950695Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4950968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4951042Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4951281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4951382Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4951664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4951791Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4951811Z 2025-08-14T21:56:29.4951921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4952135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4952203Z return mod(**inputs) 2025-08-14T21:56:29.4952475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4952552Z outputs = self.model( 2025-08-14T21:56:29.4952823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4952921Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4953196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4953286Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4953523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4953607Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4953887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4954008Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4954228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4954309Z return self.act(input) 2025-08-14T21:56:29.4954314Z 2025-08-14T21:56:29.4954420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4954627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4954702Z return mod(**inputs) 2025-08-14T21:56:29.4954974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4955051Z outputs = self.model( 2025-08-14T21:56:29.4955327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4955403Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4955682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4955754Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4956066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4956170Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4956453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4956551Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4956555Z 2025-08-14T21:56:29.4956667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4956880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4956960Z return mod(**inputs) 2025-08-14T21:56:29.4957241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4957333Z outputs = self.model( 2025-08-14T21:56:29.4957604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4957709Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4957993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4958068Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4958297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4958408Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4958681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4958792Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4959062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4959220Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4959225Z 2025-08-14T21:56:29.4959353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4959563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4959637Z return mod(**inputs) 2025-08-14T21:56:29.4959930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4960002Z outputs = self.model( 2025-08-14T21:56:29.4960280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4960355Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4960627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4960709Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4960936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4961030Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4961302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4961407Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4961687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4961770Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4961773Z 2025-08-14T21:56:29.4961888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4962096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4962165Z return mod(**inputs) 2025-08-14T21:56:29.4962444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4962519Z outputs = self.model( 2025-08-14T21:56:29.4962786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4962870Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4963142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4963224Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4963453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4963534Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4963810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4963913Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4964213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4964302Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4964306Z 2025-08-14T21:56:29.4964391Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4964481Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4964587Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4964660Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4964767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4964957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4965021Z return mod(**inputs) 2025-08-14T21:56:29.4965277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4965343Z outputs = self.model( 2025-08-14T21:56:29.4965616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4965687Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4965957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4966033Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4966251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4966336Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4966597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4966697Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4966968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4967070Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4967361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4967504Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4967508Z 2025-08-14T21:56:29.4967610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4967824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4967887Z return mod(**inputs) 2025-08-14T21:56:29.4968148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4968222Z outputs = self.model( 2025-08-14T21:56:29.4968478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4968559Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4968815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4968885Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4969108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4969186Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4969439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4969541Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4969797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4969895Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4970188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4970292Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4970296Z 2025-08-14T21:56:29.4970401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4970592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4970676Z return mod(**inputs) 2025-08-14T21:56:29.4970928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4970994Z outputs = self.model( 2025-08-14T21:56:29.4971253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4971323Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4971593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4971671Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4971898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4971982Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4972235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:56:29.4972330Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:29.4972589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4972668Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4972672Z 2025-08-14T21:56:29.4972776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4972967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4973031Z return mod(**inputs) 2025-08-14T21:56:29.4973292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4973357Z outputs = self.model( 2025-08-14T21:56:29.4973609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4973685Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4973940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4974014Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4974227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4974302Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4974565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-08-14T21:56:29.4974643Z hidden_states = residual + hidden_states 2025-08-14T21:56:29.4974646Z 2025-08-14T21:56:29.4974753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4974942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4975005Z return mod(**inputs) 2025-08-14T21:56:29.4975269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4975335Z outputs = self.model( 2025-08-14T21:56:29.4975587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4975665Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4975919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4976010Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4976226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4976302Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4976566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4976695Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4976954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:56:29.4977111Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:56:29.4977114Z 2025-08-14T21:56:29.4977215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4977434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4977500Z return mod(**inputs) 2025-08-14T21:56:29.4977773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4977850Z outputs = self.model( 2025-08-14T21:56:29.4978111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4978191Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4978458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4978529Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4978761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4978835Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4979090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4979202Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4979455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:56:29.4979541Z key_states = self.k_proj(current_states) 2025-08-14T21:56:29.4979544Z 2025-08-14T21:56:29.4979642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4979830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4979900Z return mod(**inputs) 2025-08-14T21:56:29.4980151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4980221Z outputs = self.model( 2025-08-14T21:56:29.4980477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4980547Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4980808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4980876Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4981093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4981174Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4981428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4981538Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4981788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:56:29.4981892Z value_states = self.v_proj(current_states) 2025-08-14T21:56:29.4981895Z 2025-08-14T21:56:29.4981978Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4982053Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4982134Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4982206Z cudagraph partition due to non gpu ops 2025-08-14T21:56:29.4982320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4982518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4982582Z return mod(**inputs) 2025-08-14T21:56:29.4982835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4982908Z outputs = self.model( 2025-08-14T21:56:29.4983160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4983255Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4983505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4983589Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4983807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4983886Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4984144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4984255Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4984500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4984600Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4984874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:56:29.4985000Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:29.4985003Z 2025-08-14T21:56:29.4985108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4985295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4985368Z return mod(**inputs) 2025-08-14T21:56:29.4985614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4985680Z outputs = self.model( 2025-08-14T21:56:29.4985934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4986005Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4986254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4986333Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4986544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4986628Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4986882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4986987Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4987248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:56:29.4987345Z attn_output, attn_weights = attention_interface( 2025-08-14T21:56:29.4987636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:56:29.4987766Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:29.4987770Z 2025-08-14T21:56:29.4987871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4988076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4988140Z return mod(**inputs) 2025-08-14T21:56:29.4988417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4988491Z outputs = self.model( 2025-08-14T21:56:29.4988748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4988827Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4989087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4989156Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4989390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4989467Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4989725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:56:29.4989833Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:56:29.4990080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:56:29.4990165Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:29.4990168Z 2025-08-14T21:56:29.4990268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4990461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4990533Z return mod(**inputs) 2025-08-14T21:56:29.4990788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4990863Z outputs = self.model( 2025-08-14T21:56:29.4991118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4991189Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4991451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4991519Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4991733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4991817Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4992076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4992204Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4992207Z 2025-08-14T21:56:29.4992308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4992505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4992577Z return mod(**inputs) 2025-08-14T21:56:29.4992839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4992911Z outputs = self.model( 2025-08-14T21:56:29.4993169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4993240Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4993506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4993595Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4993810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4993895Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4994154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:56:29.4994295Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:29.4994509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:29.4994576Z return self.act(input) 2025-08-14T21:56:29.4994580Z 2025-08-14T21:56:29.4994689Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4994889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4994959Z return mod(**inputs) 2025-08-14T21:56:29.4995241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:56:29.4995308Z outputs = self.model( 2025-08-14T21:56:29.4995592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:56:29.4995671Z decoder_outputs = self.decoder( 2025-08-14T21:56:29.4996014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:56:29.4996103Z layer_outputs = decoder_layer( 2025-08-14T21:56:29.4996333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:29.4996424Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:29.4996694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:56:29.4996784Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:29.4996789Z 2025-08-14T21:56:29.4996905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4997112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4997188Z return mod(**inputs) 2025-08-14T21:56:29.4997467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1489, in forward 2025-08-14T21:56:29.4997596Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-08-14T21:56:29.4997600Z 2025-08-14T21:56:29.4997716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:29.4997923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:29.4997993Z return mod(**inputs) 2025-08-14T21:56:29.4998281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1494, in forward 2025-08-14T21:56:29.4998444Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:56:29.4998448Z 2025-08-14T21:56:41.2473698Z Compilation time (from dynamo_timed): 27.675170774 2025-08-14T21:56:41.2478940Z pass 2025-08-14T21:56:41.2481966Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:41.2482786Z TIMING: _recursive_pre_grad_passes:0.01501 _recursive_joint_graph_passes:1.16825 _recursive_post_grad_passes:0.1717 async_compile.wait:0.79519 code_gen:11.31749 inductor_compile:14.37419 backend_compile:21.74577 gc:0.00138 entire_frame_compile:27.67517 total_wall_time:27.67517 2025-08-14T21:56:41.2483821Z STATS: call_* op count: 965 | FakeTensorMode.__torch_dispatch__:33299 | FakeTensor.__torch_dispatch__:11840 | ProxyTorchDispatchMode.__torch_dispatch__:12299 2025-08-14T21:56:41.2484325Z Dynamo produced 1 graphs covering 965 ops with 0 graph breaks (0 unique) 2025-08-14T21:56:47.1331527Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:56:47.1333639Z from pkg_resources import resource_filename 2025-08-14T21:56:47.7271464Z 2025-08-14T21:56:47.7395004Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-08-14T21:56:47.7395679Z WARNING:transformers.models.roberta.modeling_roberta:If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-08-14T21:56:49.1841088Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-14T21:56:49.1842410Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-14T21:56:49.1843430Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-14T21:56:49.1844423Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-14T21:56:49.3663768Z 2025-08-14T21:56:49.3664362Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:56:49.3681684Z cpu eval RobertaForCausalLM 2025-08-14T21:56:49.9595584Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:50.2487751Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:50.5273906Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:58.3415957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3416829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3417295Z return mod(**inputs) 2025-08-14T21:56:58.3417738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3418158Z outputs = self.roberta( 2025-08-14T21:56:58.3418562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:56:58.3419016Z embedding_output = self.embeddings( 2025-08-14T21:56:58.3419517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:56:58.3420070Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:56:58.3420663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1576, in create_position_ids_from_input_ids 2025-08-14T21:56:58.3421146Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:56:58.3421299Z 2025-08-14T21:56:58.3421383Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3421601Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3421801Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3422006Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3422212Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3422419Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3422633Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3422905Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3423471Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3423690Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3423903Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3424122Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3424372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3424777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3425213Z return mod(**inputs) 2025-08-14T21:56:58.3425665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3426062Z outputs = self.roberta( 2025-08-14T21:56:58.3426443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:56:58.3426885Z embedding_output = self.embeddings( 2025-08-14T21:56:58.3427333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:56:58.3427855Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:56:58.3428488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-14T21:56:58.3429085Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:56:58.3429336Z 2025-08-14T21:56:58.3429457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3429865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3430230Z return mod(**inputs) 2025-08-14T21:56:58.3430640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3431066Z outputs = self.roberta( 2025-08-14T21:56:58.3431480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:56:58.3431919Z embedding_output = self.embeddings( 2025-08-14T21:56:58.3432378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:56:58.3432936Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:56:58.3433574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-14T21:56:58.3434196Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:56:58.3434457Z 2025-08-14T21:56:58.3434569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3434972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3435320Z return mod(**inputs) 2025-08-14T21:56:58.3435733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3436379Z outputs = self.roberta( 2025-08-14T21:56:58.3436795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3437234Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3437650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3438069Z layer_outputs = layer_module( 2025-08-14T21:56:58.3438442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3438835Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3439291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3439718Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3440122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3440641Z return func(*args, **kwargs) 2025-08-14T21:56:58.3441103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3441524Z self_outputs = self.self( 2025-08-14T21:56:58.3441904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3442304Z return func(*args, **kwargs) 2025-08-14T21:56:58.3442710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3443309Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3443618Z 2025-08-14T21:56:58.3443737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3444159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3444508Z return mod(**inputs) 2025-08-14T21:56:58.3445006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3445422Z outputs = self.roberta( 2025-08-14T21:56:58.3445824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3446243Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3446706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3447142Z layer_outputs = layer_module( 2025-08-14T21:56:58.3447525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3447923Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3448363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3448796Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3449201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3449586Z return func(*args, **kwargs) 2025-08-14T21:56:58.3449990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3450403Z self_outputs = self.self( 2025-08-14T21:56:58.3450775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3451168Z return func(*args, **kwargs) 2025-08-14T21:56:58.3451568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3451988Z self.key(current_states) 2025-08-14T21:56:58.3452113Z 2025-08-14T21:56:58.3452243Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3452648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3452994Z return mod(**inputs) 2025-08-14T21:56:58.3453394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3453801Z outputs = self.roberta( 2025-08-14T21:56:58.3454203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3454647Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3455050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3455463Z layer_outputs = layer_module( 2025-08-14T21:56:58.3455830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3456235Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3456646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3457070Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3457478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3457894Z return func(*args, **kwargs) 2025-08-14T21:56:58.3458306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3458719Z self_outputs = self.self( 2025-08-14T21:56:58.3459119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3459501Z return func(*args, **kwargs) 2025-08-14T21:56:58.3459903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3460316Z self.value(current_states) 2025-08-14T21:56:58.3460442Z 2025-08-14T21:56:58.3460534Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3460783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3461178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3461553Z return mod(**inputs) 2025-08-14T21:56:58.3461948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3462373Z outputs = self.roberta( 2025-08-14T21:56:58.3462784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3463244Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3463664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3464094Z layer_outputs = layer_module( 2025-08-14T21:56:58.3464471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3464867Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3465302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3465740Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3466158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3466559Z return func(*args, **kwargs) 2025-08-14T21:56:58.3466973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3467400Z self_outputs = self.self( 2025-08-14T21:56:58.3467789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3468193Z return func(*args, **kwargs) 2025-08-14T21:56:58.3468608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3469102Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3469305Z 2025-08-14T21:56:58.3469419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3469852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3470205Z return mod(**inputs) 2025-08-14T21:56:58.3470608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3471027Z outputs = self.roberta( 2025-08-14T21:56:58.3471457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3471882Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3472295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3472715Z layer_outputs = layer_module( 2025-08-14T21:56:58.3473093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3473490Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3473947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3474387Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3474823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3475232Z return func(*args, **kwargs) 2025-08-14T21:56:58.3475648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3476222Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3476711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3477143Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3477308Z 2025-08-14T21:56:58.3477437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3477828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3478173Z return mod(**inputs) 2025-08-14T21:56:58.3478557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3478971Z outputs = self.roberta( 2025-08-14T21:56:58.3479365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3479782Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3480184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3480595Z layer_outputs = layer_module( 2025-08-14T21:56:58.3480965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3481352Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3481769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3482279Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3482717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3483136Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3483590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3484096Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3484566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3485008Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3485194Z 2025-08-14T21:56:58.3485309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3485691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3486030Z return mod(**inputs) 2025-08-14T21:56:58.3486425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3486889Z outputs = self.roberta( 2025-08-14T21:56:58.3487294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3487698Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3488107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3488514Z layer_outputs = layer_module( 2025-08-14T21:56:58.3488899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3489284Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3489724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3490149Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3490562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3490976Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3491424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3491919Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3492372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3492829Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3493231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3493603Z return self.act(input) 2025-08-14T21:56:58.3493730Z 2025-08-14T21:56:58.3493839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3494221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3494563Z return mod(**inputs) 2025-08-14T21:56:58.3494922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3495320Z outputs = self.roberta( 2025-08-14T21:56:58.3495709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3496113Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3496526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3496936Z layer_outputs = layer_module( 2025-08-14T21:56:58.3497296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3497650Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3498046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3498460Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3498897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3499312Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3499755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3500291Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3500750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3501161Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3501326Z 2025-08-14T21:56:58.3501431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3501787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3502104Z return mod(**inputs) 2025-08-14T21:56:58.3502471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3502856Z outputs = self.roberta( 2025-08-14T21:56:58.3503241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3503641Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3504034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3504441Z layer_outputs = layer_module( 2025-08-14T21:56:58.3504781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3505142Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3505541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3505945Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3506319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3506692Z return func(*args, **kwargs) 2025-08-14T21:56:58.3507076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3507458Z self_outputs = self.self( 2025-08-14T21:56:58.3507824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3508196Z return func(*args, **kwargs) 2025-08-14T21:56:58.3508570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3509441Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3509717Z 2025-08-14T21:56:58.3509827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3510215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3510565Z return mod(**inputs) 2025-08-14T21:56:58.3510935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3511329Z outputs = self.roberta( 2025-08-14T21:56:58.3511718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3512134Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3512556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3512973Z layer_outputs = layer_module( 2025-08-14T21:56:58.3513345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3513732Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3514152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3514644Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3515448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3515902Z return func(*args, **kwargs) 2025-08-14T21:56:58.3516316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3516761Z self_outputs = self.self( 2025-08-14T21:56:58.3517143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3517517Z return func(*args, **kwargs) 2025-08-14T21:56:58.3517885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3518265Z self.key(current_states) 2025-08-14T21:56:58.3518378Z 2025-08-14T21:56:58.3518482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3518863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3519181Z return mod(**inputs) 2025-08-14T21:56:58.3519558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3519937Z outputs = self.roberta( 2025-08-14T21:56:58.3520303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3520683Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3521052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3521433Z layer_outputs = layer_module( 2025-08-14T21:56:58.3521771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3522125Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3522505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3522894Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3523265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3523622Z return func(*args, **kwargs) 2025-08-14T21:56:58.3524000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3524387Z self_outputs = self.self( 2025-08-14T21:56:58.3524745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3525105Z return func(*args, **kwargs) 2025-08-14T21:56:58.3525481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3525873Z self.value(current_states) 2025-08-14T21:56:58.3525988Z 2025-08-14T21:56:58.3526071Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3526312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3526671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3527006Z return mod(**inputs) 2025-08-14T21:56:58.3527354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3527727Z outputs = self.roberta( 2025-08-14T21:56:58.3528087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3528460Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3528836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3530278Z layer_outputs = layer_module( 2025-08-14T21:56:58.3530628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3530989Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3531404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3531831Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3532202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3532559Z return func(*args, **kwargs) 2025-08-14T21:56:58.3532950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3533365Z self_outputs = self.self( 2025-08-14T21:56:58.3533729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3534101Z return func(*args, **kwargs) 2025-08-14T21:56:58.3534499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3534951Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3535134Z 2025-08-14T21:56:58.3535236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3535588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3535905Z return mod(**inputs) 2025-08-14T21:56:58.3536264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3536658Z outputs = self.roberta( 2025-08-14T21:56:58.3537042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3537422Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3537792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3538172Z layer_outputs = layer_module( 2025-08-14T21:56:58.3538519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3538881Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3539271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3539661Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3540027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3540382Z return func(*args, **kwargs) 2025-08-14T21:56:58.3540755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3541185Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3541618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3542014Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3542159Z 2025-08-14T21:56:58.3542265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3542619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3542945Z return mod(**inputs) 2025-08-14T21:56:58.3543305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3543693Z outputs = self.roberta( 2025-08-14T21:56:58.3544085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3544477Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3544845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3545215Z layer_outputs = layer_module( 2025-08-14T21:56:58.3545563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3545900Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3546287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3546683Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3547083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3547493Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3547917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3548402Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3548833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3549237Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3549374Z 2025-08-14T21:56:58.3549487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3549840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3550157Z return mod(**inputs) 2025-08-14T21:56:58.3550524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3550914Z outputs = self.roberta( 2025-08-14T21:56:58.3551282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3551677Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3552062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3552455Z layer_outputs = layer_module( 2025-08-14T21:56:58.3552811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3553189Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3553608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3554027Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3554448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3554863Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3555311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3555892Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3556370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3556826Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3557229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3557586Z return self.act(input) 2025-08-14T21:56:58.3557723Z 2025-08-14T21:56:58.3557828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3558194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3558539Z return mod(**inputs) 2025-08-14T21:56:58.3558912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3559309Z outputs = self.roberta( 2025-08-14T21:56:58.3559697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3560094Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3560482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3560881Z layer_outputs = layer_module( 2025-08-14T21:56:58.3561211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3561575Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3561990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3562392Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3562803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3563197Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3563642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3564139Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3564577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3564979Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3565117Z 2025-08-14T21:56:58.3565229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3565591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3565906Z return mod(**inputs) 2025-08-14T21:56:58.3566271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3566654Z outputs = self.roberta( 2025-08-14T21:56:58.3567022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3567418Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3567809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3568197Z layer_outputs = layer_module( 2025-08-14T21:56:58.3568532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3568892Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3569286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3569682Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3570064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3570437Z return func(*args, **kwargs) 2025-08-14T21:56:58.3570814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3571199Z self_outputs = self.self( 2025-08-14T21:56:58.3571560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3571934Z return func(*args, **kwargs) 2025-08-14T21:56:58.3572305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3572867Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3573140Z 2025-08-14T21:56:58.3573251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3573628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3573981Z return mod(**inputs) 2025-08-14T21:56:58.3574367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3574754Z outputs = self.roberta( 2025-08-14T21:56:58.3575129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3575516Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3575943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3576338Z layer_outputs = layer_module( 2025-08-14T21:56:58.3576677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3577051Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3577449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3577852Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3578227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3578606Z return func(*args, **kwargs) 2025-08-14T21:56:58.3579007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3579422Z self_outputs = self.self( 2025-08-14T21:56:58.3579803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3580194Z return func(*args, **kwargs) 2025-08-14T21:56:58.3580592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3580981Z self.key(current_states) 2025-08-14T21:56:58.3581113Z 2025-08-14T21:56:58.3581223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3581600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3581940Z return mod(**inputs) 2025-08-14T21:56:58.3582320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3582732Z outputs = self.roberta( 2025-08-14T21:56:58.3583127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3583532Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3583939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3584348Z layer_outputs = layer_module( 2025-08-14T21:56:58.3584711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3585090Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3585508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3585932Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3586335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3586729Z return func(*args, **kwargs) 2025-08-14T21:56:58.3587166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3587576Z self_outputs = self.self( 2025-08-14T21:56:58.3587947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3588335Z return func(*args, **kwargs) 2025-08-14T21:56:58.3588782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3589200Z self.value(current_states) 2025-08-14T21:56:58.3589327Z 2025-08-14T21:56:58.3589413Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3589671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3590052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3590387Z return mod(**inputs) 2025-08-14T21:56:58.3590799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3591210Z outputs = self.roberta( 2025-08-14T21:56:58.3591621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3592042Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3592478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3592906Z layer_outputs = layer_module( 2025-08-14T21:56:58.3593270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3593668Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3594096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3594534Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3594940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3595354Z return func(*args, **kwargs) 2025-08-14T21:56:58.3595770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3596287Z self_outputs = self.self( 2025-08-14T21:56:58.3596680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3597089Z return func(*args, **kwargs) 2025-08-14T21:56:58.3597509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3597980Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3598190Z 2025-08-14T21:56:58.3598309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3598709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3599059Z return mod(**inputs) 2025-08-14T21:56:58.3599460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3599891Z outputs = self.roberta( 2025-08-14T21:56:58.3600299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3600719Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3601145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3601568Z layer_outputs = layer_module( 2025-08-14T21:56:58.3601946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3602374Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3602805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3603243Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3603664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3604090Z return func(*args, **kwargs) 2025-08-14T21:56:58.3604502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3604988Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3605463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3605901Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3606059Z 2025-08-14T21:56:58.3606191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3606582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3606944Z return mod(**inputs) 2025-08-14T21:56:58.3607346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3607770Z outputs = self.roberta( 2025-08-14T21:56:58.3608148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3608543Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3609096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3609517Z layer_outputs = layer_module( 2025-08-14T21:56:58.3609884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3610276Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3610705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3611114Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3611515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3611915Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3612349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3612832Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3613275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3613684Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3613822Z 2025-08-14T21:56:58.3613933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3614284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3614611Z return mod(**inputs) 2025-08-14T21:56:58.3614986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3615373Z outputs = self.roberta( 2025-08-14T21:56:58.3615743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3616139Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3616528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3616912Z layer_outputs = layer_module( 2025-08-14T21:56:58.3617331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3617693Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3618088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3618515Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3618916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3619311Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3619731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3620209Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3620670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3621101Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3621500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3621842Z return self.act(input) 2025-08-14T21:56:58.3621960Z 2025-08-14T21:56:58.3622067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3622426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3622743Z return mod(**inputs) 2025-08-14T21:56:58.3623111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3623498Z outputs = self.roberta( 2025-08-14T21:56:58.3623860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3624251Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3624643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3625034Z layer_outputs = layer_module( 2025-08-14T21:56:58.3625373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3625734Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3626137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3626531Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3626920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3627307Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3627729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3628220Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3628700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3629108Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3629245Z 2025-08-14T21:56:58.3629359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3629710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3630035Z return mod(**inputs) 2025-08-14T21:56:58.3630408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3630791Z outputs = self.roberta( 2025-08-14T21:56:58.3631181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3631625Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3632035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3632435Z layer_outputs = layer_module( 2025-08-14T21:56:58.3632825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3633210Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3633624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3634044Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3634445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3634843Z return func(*args, **kwargs) 2025-08-14T21:56:58.3635270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3635699Z self_outputs = self.self( 2025-08-14T21:56:58.3636199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3636610Z return func(*args, **kwargs) 2025-08-14T21:56:58.3637026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3637593Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3637854Z 2025-08-14T21:56:58.3637968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3638330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3638655Z return mod(**inputs) 2025-08-14T21:56:58.3639035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3639433Z outputs = self.roberta( 2025-08-14T21:56:58.3639804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3640218Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3640633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3641037Z layer_outputs = layer_module( 2025-08-14T21:56:58.3641378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3641743Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3642142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3642540Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3642924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3643304Z return func(*args, **kwargs) 2025-08-14T21:56:58.3643687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3644071Z self_outputs = self.self( 2025-08-14T21:56:58.3644433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3644806Z return func(*args, **kwargs) 2025-08-14T21:56:58.3645181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3645571Z self.key(current_states) 2025-08-14T21:56:58.3645699Z 2025-08-14T21:56:58.3645834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3646212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3646559Z return mod(**inputs) 2025-08-14T21:56:58.3646949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3647385Z outputs = self.roberta( 2025-08-14T21:56:58.3647778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3648170Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3648563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3648959Z layer_outputs = layer_module( 2025-08-14T21:56:58.3649303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3649732Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3650147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3650588Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3650986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3651376Z return func(*args, **kwargs) 2025-08-14T21:56:58.3651756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3652149Z self_outputs = self.self( 2025-08-14T21:56:58.3652498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3652887Z return func(*args, **kwargs) 2025-08-14T21:56:58.3653265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3653648Z self.value(current_states) 2025-08-14T21:56:58.3653772Z 2025-08-14T21:56:58.3653855Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3654100Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3654460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3654780Z return mod(**inputs) 2025-08-14T21:56:58.3655150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3655542Z outputs = self.roberta( 2025-08-14T21:56:58.3655908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3656299Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3656685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3657073Z layer_outputs = layer_module( 2025-08-14T21:56:58.3657422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3657808Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3658221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3658614Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3658993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3659365Z return func(*args, **kwargs) 2025-08-14T21:56:58.3659741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3660208Z self_outputs = self.self( 2025-08-14T21:56:58.3660571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3660939Z return func(*args, **kwargs) 2025-08-14T21:56:58.3661310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3661776Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3661968Z 2025-08-14T21:56:58.3662072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3662422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3662740Z return mod(**inputs) 2025-08-14T21:56:58.3663111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3663500Z outputs = self.roberta( 2025-08-14T21:56:58.3663892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3664281Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3664730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3665127Z layer_outputs = layer_module( 2025-08-14T21:56:58.3665473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3665838Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3666238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3666644Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3667023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3667402Z return func(*args, **kwargs) 2025-08-14T21:56:58.3667782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3668236Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3668675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3669084Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3669224Z 2025-08-14T21:56:58.3669337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3669685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3670014Z return mod(**inputs) 2025-08-14T21:56:58.3670408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3670822Z outputs = self.roberta( 2025-08-14T21:56:58.3671213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3671634Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3672045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3672458Z layer_outputs = layer_module( 2025-08-14T21:56:58.3672828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3673211Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3673630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3674055Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3674485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3674922Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3675371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3675942Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3676440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3676868Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3677013Z 2025-08-14T21:56:58.3677126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3677506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3677851Z return mod(**inputs) 2025-08-14T21:56:58.3678239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3678670Z outputs = self.roberta( 2025-08-14T21:56:58.3679068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3679502Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3679916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3680325Z layer_outputs = layer_module( 2025-08-14T21:56:58.3680689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3681077Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3681486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3681912Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3682355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3682826Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3683276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3683781Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3684239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3684695Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3685088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3685461Z return self.act(input) 2025-08-14T21:56:58.3685581Z 2025-08-14T21:56:58.3685705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3686094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3686453Z return mod(**inputs) 2025-08-14T21:56:58.3686845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3687253Z outputs = self.roberta( 2025-08-14T21:56:58.3687641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3688055Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3688464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3688867Z layer_outputs = layer_module( 2025-08-14T21:56:58.3689235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3689700Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3690117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3690538Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3690965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3691396Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3691834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3692347Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3692821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3693246Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3693400Z 2025-08-14T21:56:58.3693535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3693897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3694219Z return mod(**inputs) 2025-08-14T21:56:58.3694614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3695018Z outputs = self.roberta( 2025-08-14T21:56:58.3695404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3695795Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3696175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3696577Z layer_outputs = layer_module( 2025-08-14T21:56:58.3696940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3697324Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3697733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3698168Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3698552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3698927Z return func(*args, **kwargs) 2025-08-14T21:56:58.3699302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3699716Z self_outputs = self.self( 2025-08-14T21:56:58.3700096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3700492Z return func(*args, **kwargs) 2025-08-14T21:56:58.3700894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3701450Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3701727Z 2025-08-14T21:56:58.3701854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3702208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3702534Z return mod(**inputs) 2025-08-14T21:56:58.3702906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3703300Z outputs = self.roberta( 2025-08-14T21:56:58.3703672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3704064Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3704481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3704860Z layer_outputs = layer_module( 2025-08-14T21:56:58.3705203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3705564Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3705976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3706373Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3706756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3707127Z return func(*args, **kwargs) 2025-08-14T21:56:58.3707494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3707884Z self_outputs = self.self( 2025-08-14T21:56:58.3708258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3708632Z return func(*args, **kwargs) 2025-08-14T21:56:58.3709254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3709655Z self.key(current_states) 2025-08-14T21:56:58.3709777Z 2025-08-14T21:56:58.3709891Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3710258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3710596Z return mod(**inputs) 2025-08-14T21:56:58.3710991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3711408Z outputs = self.roberta( 2025-08-14T21:56:58.3711801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3712221Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3712633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3713051Z layer_outputs = layer_module( 2025-08-14T21:56:58.3713414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3713814Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3714246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3714684Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3715100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3715500Z return func(*args, **kwargs) 2025-08-14T21:56:58.3715963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3716378Z self_outputs = self.self( 2025-08-14T21:56:58.3716763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3717169Z return func(*args, **kwargs) 2025-08-14T21:56:58.3717578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3717994Z self.value(current_states) 2025-08-14T21:56:58.3718128Z 2025-08-14T21:56:58.3718216Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3718473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3718850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3719239Z return mod(**inputs) 2025-08-14T21:56:58.3719632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3720035Z outputs = self.roberta( 2025-08-14T21:56:58.3720433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3720880Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3721297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3721717Z layer_outputs = layer_module( 2025-08-14T21:56:58.3722093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3722499Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3722939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3723407Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3723822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3724239Z return func(*args, **kwargs) 2025-08-14T21:56:58.3724639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3725056Z self_outputs = self.self( 2025-08-14T21:56:58.3725445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3725847Z return func(*args, **kwargs) 2025-08-14T21:56:58.3726240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3726716Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3726926Z 2025-08-14T21:56:58.3727049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3727432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3727783Z return mod(**inputs) 2025-08-14T21:56:58.3728170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3728582Z outputs = self.roberta( 2025-08-14T21:56:58.3728951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3729324Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3729691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3730061Z layer_outputs = layer_module( 2025-08-14T21:56:58.3730386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3730733Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3731115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3731494Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3731881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3732252Z return func(*args, **kwargs) 2025-08-14T21:56:58.3732628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3733065Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3733513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3733936Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3734073Z 2025-08-14T21:56:58.3734183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3734529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3734845Z return mod(**inputs) 2025-08-14T21:56:58.3735213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3735620Z outputs = self.roberta( 2025-08-14T21:56:58.3736001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3736402Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3736800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3737196Z layer_outputs = layer_module( 2025-08-14T21:56:58.3737579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3737937Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3738336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3738742Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3739143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3739533Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3739945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3740418Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3740863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3741278Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3741421Z 2025-08-14T21:56:58.3741525Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3741891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3742221Z return mod(**inputs) 2025-08-14T21:56:58.3742590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3742987Z outputs = self.roberta( 2025-08-14T21:56:58.3743363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3743763Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3744158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3744548Z layer_outputs = layer_module( 2025-08-14T21:56:58.3744897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3745268Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3745662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3746070Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3746474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3746864Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3747297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3747774Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3748228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3748650Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3749029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3749405Z return self.act(input) 2025-08-14T21:56:58.3749521Z 2025-08-14T21:56:58.3749637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3749999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3750335Z return mod(**inputs) 2025-08-14T21:56:58.3750708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3751091Z outputs = self.roberta( 2025-08-14T21:56:58.3751487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3751882Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3752285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3752671Z layer_outputs = layer_module( 2025-08-14T21:56:58.3753014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3753379Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3753766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3754170Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3754566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3754959Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3755375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3755975Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3756483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3756926Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3757084Z 2025-08-14T21:56:58.3757189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3757553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3757886Z return mod(**inputs) 2025-08-14T21:56:58.3758255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3758663Z outputs = self.roberta( 2025-08-14T21:56:58.3759066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3759486Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3759902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3760323Z layer_outputs = layer_module( 2025-08-14T21:56:58.3760694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3761049Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3761449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3761871Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3762276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3762695Z return func(*args, **kwargs) 2025-08-14T21:56:58.3763104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3763518Z self_outputs = self.self( 2025-08-14T21:56:58.3763911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3764325Z return func(*args, **kwargs) 2025-08-14T21:56:58.3764723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3765279Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3765551Z 2025-08-14T21:56:58.3765662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3766044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3766417Z return mod(**inputs) 2025-08-14T21:56:58.3766809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3767229Z outputs = self.roberta( 2025-08-14T21:56:58.3767627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3768048Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3768463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3768873Z layer_outputs = layer_module( 2025-08-14T21:56:58.3769244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3769642Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3770067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3770461Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3770839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3771210Z return func(*args, **kwargs) 2025-08-14T21:56:58.3771577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3771959Z self_outputs = self.self( 2025-08-14T21:56:58.3772318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3772677Z return func(*args, **kwargs) 2025-08-14T21:56:58.3773062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3773465Z self.key(current_states) 2025-08-14T21:56:58.3773592Z 2025-08-14T21:56:58.3773715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3774093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3774448Z return mod(**inputs) 2025-08-14T21:56:58.3774853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3775271Z outputs = self.roberta( 2025-08-14T21:56:58.3775638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3776026Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3776409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3776787Z layer_outputs = layer_module( 2025-08-14T21:56:58.3777131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3777508Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3777910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3778306Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3778713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3779091Z return func(*args, **kwargs) 2025-08-14T21:56:58.3779452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3779836Z self_outputs = self.self( 2025-08-14T21:56:58.3780188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3780552Z return func(*args, **kwargs) 2025-08-14T21:56:58.3780934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3781323Z self.value(current_states) 2025-08-14T21:56:58.3781437Z 2025-08-14T21:56:58.3781544Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3781778Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3782142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3782467Z return mod(**inputs) 2025-08-14T21:56:58.3782834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3783213Z outputs = self.roberta( 2025-08-14T21:56:58.3783583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3783973Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3784356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3784750Z layer_outputs = layer_module( 2025-08-14T21:56:58.3785094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3785454Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3785839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3786239Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3786666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3787067Z return func(*args, **kwargs) 2025-08-14T21:56:58.3787461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3787877Z self_outputs = self.self( 2025-08-14T21:56:58.3788257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3788663Z return func(*args, **kwargs) 2025-08-14T21:56:58.3789041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3789493Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3789678Z 2025-08-14T21:56:58.3789789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3790141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3790470Z return mod(**inputs) 2025-08-14T21:56:58.3790839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3791247Z outputs = self.roberta( 2025-08-14T21:56:58.3791643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3792056Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3792473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3792902Z layer_outputs = layer_module( 2025-08-14T21:56:58.3793273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3793658Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3794077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3794496Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3794897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3795324Z return func(*args, **kwargs) 2025-08-14T21:56:58.3795730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3796348Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3796848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3797296Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3797447Z 2025-08-14T21:56:58.3797562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3797950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3798299Z return mod(**inputs) 2025-08-14T21:56:58.3798696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3799105Z outputs = self.roberta( 2025-08-14T21:56:58.3799503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3799925Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3800334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3800753Z layer_outputs = layer_module( 2025-08-14T21:56:58.3801123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3801508Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3801921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3802355Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3802785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3803197Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3803656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3804156Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3804621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3805041Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3805198Z 2025-08-14T21:56:58.3805307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3805697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3806053Z return mod(**inputs) 2025-08-14T21:56:58.3806437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3806887Z outputs = self.roberta( 2025-08-14T21:56:58.3807285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3807692Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3808122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3808536Z layer_outputs = layer_module( 2025-08-14T21:56:58.3809086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3809478Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3809894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3810328Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3810821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3811240Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3811712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3812215Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3812665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3813121Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3813527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3813908Z return self.act(input) 2025-08-14T21:56:58.3814025Z 2025-08-14T21:56:58.3814143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3814529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3814890Z return mod(**inputs) 2025-08-14T21:56:58.3815275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3815686Z outputs = self.roberta( 2025-08-14T21:56:58.3816078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3816489Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3816890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3817302Z layer_outputs = layer_module( 2025-08-14T21:56:58.3817650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3818014Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3818401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3818805Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3819204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3819589Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3820009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3820490Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3820935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3821377Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3821524Z 2025-08-14T21:56:58.3821631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3821990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3822060Z return mod(**inputs) 2025-08-14T21:56:58.3822323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3822423Z outputs = self.roberta( 2025-08-14T21:56:58.3822690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3822771Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3823038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3823110Z layer_outputs = layer_module( 2025-08-14T21:56:58.3823359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3823442Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3823723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3823808Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3824060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3824143Z return func(*args, **kwargs) 2025-08-14T21:56:58.3824401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3824479Z self_outputs = self.self( 2025-08-14T21:56:58.3824721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3824794Z return func(*args, **kwargs) 2025-08-14T21:56:58.3825058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3825272Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3825276Z 2025-08-14T21:56:58.3825387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3825587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3825654Z return mod(**inputs) 2025-08-14T21:56:58.3825920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3825987Z outputs = self.roberta( 2025-08-14T21:56:58.3826243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3826323Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3826582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3826659Z layer_outputs = layer_module( 2025-08-14T21:56:58.3826880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3826960Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3827227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3827309Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3827551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3827626Z return func(*args, **kwargs) 2025-08-14T21:56:58.3827882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3827978Z self_outputs = self.self( 2025-08-14T21:56:58.3828217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3828287Z return func(*args, **kwargs) 2025-08-14T21:56:58.3828554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3828643Z self.key(current_states) 2025-08-14T21:56:58.3828646Z 2025-08-14T21:56:58.3828768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3828961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3829024Z return mod(**inputs) 2025-08-14T21:56:58.3829283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3829351Z outputs = self.roberta( 2025-08-14T21:56:58.3829621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3829703Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3829970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3830051Z layer_outputs = layer_module( 2025-08-14T21:56:58.3830268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3830344Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3830601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3830680Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3830909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3830986Z return func(*args, **kwargs) 2025-08-14T21:56:58.3831240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3831316Z self_outputs = self.self( 2025-08-14T21:56:58.3831547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3831616Z return func(*args, **kwargs) 2025-08-14T21:56:58.3831878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3831950Z self.value(current_states) 2025-08-14T21:56:58.3831953Z 2025-08-14T21:56:58.3832041Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3832144Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3832340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3832415Z return mod(**inputs) 2025-08-14T21:56:58.3832672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3832740Z outputs = self.roberta( 2025-08-14T21:56:58.3833011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3833089Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3833366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3833440Z layer_outputs = layer_module( 2025-08-14T21:56:58.3833670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3833760Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3834052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3834138Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3834397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3834469Z return func(*args, **kwargs) 2025-08-14T21:56:58.3834770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3834844Z self_outputs = self.self( 2025-08-14T21:56:58.3835096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3835178Z return func(*args, **kwargs) 2025-08-14T21:56:58.3835453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3835620Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3835625Z 2025-08-14T21:56:58.3835737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3836034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3836117Z return mod(**inputs) 2025-08-14T21:56:58.3836392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3836466Z outputs = self.roberta( 2025-08-14T21:56:58.3836747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3836823Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3837102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3837177Z layer_outputs = layer_module( 2025-08-14T21:56:58.3837411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3837502Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3837776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3837866Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3838135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3838209Z return func(*args, **kwargs) 2025-08-14T21:56:58.3838490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3838629Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3838902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3839004Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3839008Z 2025-08-14T21:56:58.3839118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3839337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3839408Z return mod(**inputs) 2025-08-14T21:56:58.3839684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3839775Z outputs = self.roberta( 2025-08-14T21:56:58.3840027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3840097Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3840354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3840443Z layer_outputs = layer_module( 2025-08-14T21:56:58.3840664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3840738Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3840994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3841099Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3841347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3841429Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3841711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3841826Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3842104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3842189Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3842193Z 2025-08-14T21:56:58.3842314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3842518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3842586Z return mod(**inputs) 2025-08-14T21:56:58.3842850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3842918Z outputs = self.roberta( 2025-08-14T21:56:58.3843171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3843257Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3843528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3843612Z layer_outputs = layer_module( 2025-08-14T21:56:58.3843842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3843926Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3844206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3844295Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3844564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3844651Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3844961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3845097Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3845375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3845495Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3845724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3845800Z return self.act(input) 2025-08-14T21:56:58.3845804Z 2025-08-14T21:56:58.3845918Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3846129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3846198Z return mod(**inputs) 2025-08-14T21:56:58.3846476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3846549Z outputs = self.roberta( 2025-08-14T21:56:58.3846849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3846936Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3847215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3847299Z layer_outputs = layer_module( 2025-08-14T21:56:58.3847551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3847636Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3847920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3848007Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3848275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3848381Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3848693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3848860Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3849130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3849218Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3849222Z 2025-08-14T21:56:58.3849337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3849543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3849619Z return mod(**inputs) 2025-08-14T21:56:58.3849889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3849965Z outputs = self.roberta( 2025-08-14T21:56:58.3850248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3850325Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3850596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3850681Z layer_outputs = layer_module( 2025-08-14T21:56:58.3850910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3850999Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3851274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3851361Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3851626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3851702Z return func(*args, **kwargs) 2025-08-14T21:56:58.3851981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3852056Z self_outputs = self.self( 2025-08-14T21:56:58.3852306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3852390Z return func(*args, **kwargs) 2025-08-14T21:56:58.3852658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3852875Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3852886Z 2025-08-14T21:56:58.3852993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3853226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3853302Z return mod(**inputs) 2025-08-14T21:56:58.3853576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3853651Z outputs = self.roberta( 2025-08-14T21:56:58.3853928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3854026Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3854302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3854384Z layer_outputs = layer_module( 2025-08-14T21:56:58.3854600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3854685Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3854989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3855074Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3855339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3855411Z return func(*args, **kwargs) 2025-08-14T21:56:58.3855675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3855747Z self_outputs = self.self( 2025-08-14T21:56:58.3855986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3856060Z return func(*args, **kwargs) 2025-08-14T21:56:58.3856318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3856398Z self.key(current_states) 2025-08-14T21:56:58.3856402Z 2025-08-14T21:56:58.3856503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3856701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3856772Z return mod(**inputs) 2025-08-14T21:56:58.3857030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3857100Z outputs = self.roberta( 2025-08-14T21:56:58.3857364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3857435Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3857714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3857782Z layer_outputs = layer_module( 2025-08-14T21:56:58.3857999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3858086Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3858340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3858423Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3858665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3858733Z return func(*args, **kwargs) 2025-08-14T21:56:58.3858992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3859066Z self_outputs = self.self( 2025-08-14T21:56:58.3859304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3859402Z return func(*args, **kwargs) 2025-08-14T21:56:58.3859662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3859742Z self.value(current_states) 2025-08-14T21:56:58.3859747Z 2025-08-14T21:56:58.3859830Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3859953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3860157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3860224Z return mod(**inputs) 2025-08-14T21:56:58.3860481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3860557Z outputs = self.roberta( 2025-08-14T21:56:58.3860825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3860932Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3861183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3861254Z layer_outputs = layer_module( 2025-08-14T21:56:58.3861497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3861578Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3861839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3861927Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3862170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3862246Z return func(*args, **kwargs) 2025-08-14T21:56:58.3862520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3862589Z self_outputs = self.self( 2025-08-14T21:56:58.3862848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3862914Z return func(*args, **kwargs) 2025-08-14T21:56:58.3863178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3863311Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3863315Z 2025-08-14T21:56:58.3863413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3863611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3863674Z return mod(**inputs) 2025-08-14T21:56:58.3863937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3864017Z outputs = self.roberta( 2025-08-14T21:56:58.3864277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3864356Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3864619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3864691Z layer_outputs = layer_module( 2025-08-14T21:56:58.3864920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3864998Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3865260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3865347Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3865610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3865686Z return func(*args, **kwargs) 2025-08-14T21:56:58.3865944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3866074Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3866358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3866445Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3866448Z 2025-08-14T21:56:58.3866557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3866752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3866817Z return mod(**inputs) 2025-08-14T21:56:58.3867104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3867175Z outputs = self.roberta( 2025-08-14T21:56:58.3867440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3867521Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3867770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3867846Z layer_outputs = layer_module( 2025-08-14T21:56:58.3868055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3868131Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3868387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3868467Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3868724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3868800Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3869084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3869210Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3869466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3869547Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3869558Z 2025-08-14T21:56:58.3869659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3869855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3869932Z return mod(**inputs) 2025-08-14T21:56:58.3870207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3870279Z outputs = self.roberta( 2025-08-14T21:56:58.3870560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3870637Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3870914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3870989Z layer_outputs = layer_module( 2025-08-14T21:56:58.3871217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3871307Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3871577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3871688Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3871984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3872065Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3872382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3872528Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3872798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3872925Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3873146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3873228Z return self.act(input) 2025-08-14T21:56:58.3873233Z 2025-08-14T21:56:58.3873358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3873564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3873658Z return mod(**inputs) 2025-08-14T21:56:58.3873933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3874008Z outputs = self.roberta( 2025-08-14T21:56:58.3874289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3874366Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3874649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3874725Z layer_outputs = layer_module( 2025-08-14T21:56:58.3874960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3875053Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3875330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3875428Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3875728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3875896Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3876237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3876377Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3876651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3876755Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3876759Z 2025-08-14T21:56:58.3876872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3877101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3877174Z return mod(**inputs) 2025-08-14T21:56:58.3877458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3877545Z outputs = self.roberta( 2025-08-14T21:56:58.3877824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3877912Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3878187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3878283Z layer_outputs = layer_module( 2025-08-14T21:56:58.3878515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3878605Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3878861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3878969Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3879203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3879284Z return func(*args, **kwargs) 2025-08-14T21:56:58.3879540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3879612Z self_outputs = self.self( 2025-08-14T21:56:58.3879873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3879967Z return func(*args, **kwargs) 2025-08-14T21:56:58.3880239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3880483Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3880490Z 2025-08-14T21:56:58.3880599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3880811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3880880Z return mod(**inputs) 2025-08-14T21:56:58.3881150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3881229Z outputs = self.roberta( 2025-08-14T21:56:58.3881498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3881586Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3881857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3881933Z layer_outputs = layer_module( 2025-08-14T21:56:58.3882172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3882257Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3882534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3882630Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3882882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3882964Z return func(*args, **kwargs) 2025-08-14T21:56:58.3883242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3883316Z self_outputs = self.self( 2025-08-14T21:56:58.3883576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3883649Z return func(*args, **kwargs) 2025-08-14T21:56:58.3883929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3884004Z self.key(current_states) 2025-08-14T21:56:58.3884007Z 2025-08-14T21:56:58.3884115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3884329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3884398Z return mod(**inputs) 2025-08-14T21:56:58.3884670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3884776Z outputs = self.roberta( 2025-08-14T21:56:58.3885049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3885138Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3885408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3885514Z layer_outputs = layer_module( 2025-08-14T21:56:58.3885748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3885829Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3886099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3886191Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3886461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3886543Z return func(*args, **kwargs) 2025-08-14T21:56:58.3886835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3886910Z self_outputs = self.self( 2025-08-14T21:56:58.3887169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3887240Z return func(*args, **kwargs) 2025-08-14T21:56:58.3887520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3887597Z self.value(current_states) 2025-08-14T21:56:58.3887600Z 2025-08-14T21:56:58.3887686Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3887800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3888013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3888081Z return mod(**inputs) 2025-08-14T21:56:58.3888368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3888439Z outputs = self.roberta( 2025-08-14T21:56:58.3888730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3888807Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3889083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3889166Z layer_outputs = layer_module( 2025-08-14T21:56:58.3889402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3889483Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3889769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3889853Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3890121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3890194Z return func(*args, **kwargs) 2025-08-14T21:56:58.3890473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3890553Z self_outputs = self.self( 2025-08-14T21:56:58.3890806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3890888Z return func(*args, **kwargs) 2025-08-14T21:56:58.3891165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3891329Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3891333Z 2025-08-14T21:56:58.3891447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3891652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3891721Z return mod(**inputs) 2025-08-14T21:56:58.3892021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3892095Z outputs = self.roberta( 2025-08-14T21:56:58.3892373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3892449Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3892723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3892826Z layer_outputs = layer_module( 2025-08-14T21:56:58.3893060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3893152Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3893442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3893532Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3893792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3893866Z return func(*args, **kwargs) 2025-08-14T21:56:58.3894138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3894285Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3894557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3894654Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3894658Z 2025-08-14T21:56:58.3894767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3894973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3895051Z return mod(**inputs) 2025-08-14T21:56:58.3895321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3895392Z outputs = self.roberta( 2025-08-14T21:56:58.3895673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3895748Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3896024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3896102Z layer_outputs = layer_module( 2025-08-14T21:56:58.3896332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3896424Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3896696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3896795Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3897062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3897143Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3897456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3897583Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3897879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3897974Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3897980Z 2025-08-14T21:56:58.3898087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3898319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3898389Z return mod(**inputs) 2025-08-14T21:56:58.3898661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3898742Z outputs = self.roberta( 2025-08-14T21:56:58.3899015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3899099Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3899386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3899463Z layer_outputs = layer_module( 2025-08-14T21:56:58.3899717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3899800Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3900071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3900163Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3900429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3900515Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3900820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3900948Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3901230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3901351Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3901578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3901653Z return self.act(input) 2025-08-14T21:56:58.3901657Z 2025-08-14T21:56:58.3901764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3901981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3902050Z return mod(**inputs) 2025-08-14T21:56:58.3902317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3902399Z outputs = self.roberta( 2025-08-14T21:56:58.3902670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3902751Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3903022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3903100Z layer_outputs = layer_module( 2025-08-14T21:56:58.3903335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3903417Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3903692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3903777Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3904045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3904156Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3904468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3904605Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3904905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3904991Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3904995Z 2025-08-14T21:56:58.3905109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3905315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3905383Z return mod(**inputs) 2025-08-14T21:56:58.3905676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3905751Z outputs = self.roberta( 2025-08-14T21:56:58.3906033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3906130Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3906405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3906490Z layer_outputs = layer_module( 2025-08-14T21:56:58.3906720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3906802Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3907083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3907171Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3907438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3907513Z return func(*args, **kwargs) 2025-08-14T21:56:58.3907787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3907867Z self_outputs = self.self( 2025-08-14T21:56:58.3908124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3908197Z return func(*args, **kwargs) 2025-08-14T21:56:58.3908478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3908826Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3908833Z 2025-08-14T21:56:58.3908956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3909166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3909237Z return mod(**inputs) 2025-08-14T21:56:58.3909518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3909594Z outputs = self.roberta( 2025-08-14T21:56:58.3909878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3909956Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3910233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3910320Z layer_outputs = layer_module( 2025-08-14T21:56:58.3910556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3910709Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3910998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3911088Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3911354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3911459Z return func(*args, **kwargs) 2025-08-14T21:56:58.3911741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3911824Z self_outputs = self.self( 2025-08-14T21:56:58.3912097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3912181Z return func(*args, **kwargs) 2025-08-14T21:56:58.3912485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3912565Z self.key(current_states) 2025-08-14T21:56:58.3912569Z 2025-08-14T21:56:58.3912691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3912929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3913004Z return mod(**inputs) 2025-08-14T21:56:58.3913295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3913370Z outputs = self.roberta( 2025-08-14T21:56:58.3913664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3913743Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3914037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3914123Z layer_outputs = layer_module( 2025-08-14T21:56:58.3914359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3914441Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3914728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3914816Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3915084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3915156Z return func(*args, **kwargs) 2025-08-14T21:56:58.3915437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3915519Z self_outputs = self.self( 2025-08-14T21:56:58.3915826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3915924Z return func(*args, **kwargs) 2025-08-14T21:56:58.3916212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3916290Z self.value(current_states) 2025-08-14T21:56:58.3916294Z 2025-08-14T21:56:58.3916392Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3916502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3916715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3916794Z return mod(**inputs) 2025-08-14T21:56:58.3917075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3917157Z outputs = self.roberta( 2025-08-14T21:56:58.3917431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3917525Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3917792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3917871Z layer_outputs = layer_module( 2025-08-14T21:56:58.3918110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3918228Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3918513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3918610Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3918887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3918965Z return func(*args, **kwargs) 2025-08-14T21:56:58.3919276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3919355Z self_outputs = self.self( 2025-08-14T21:56:58.3919648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3919724Z return func(*args, **kwargs) 2025-08-14T21:56:58.3920005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3920171Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3920174Z 2025-08-14T21:56:58.3920289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3920510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3920588Z return mod(**inputs) 2025-08-14T21:56:58.3920872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3920955Z outputs = self.roberta( 2025-08-14T21:56:58.3921238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3921317Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3921608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3921684Z layer_outputs = layer_module( 2025-08-14T21:56:58.3921930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3922013Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3922297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3922391Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3922666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3922741Z return func(*args, **kwargs) 2025-08-14T21:56:58.3923034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3923174Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3923466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3923559Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3923563Z 2025-08-14T21:56:58.3923674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3923901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3923966Z return mod(**inputs) 2025-08-14T21:56:58.3924256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3924325Z outputs = self.roberta( 2025-08-14T21:56:58.3924582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3924678Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3924933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3925004Z layer_outputs = layer_module( 2025-08-14T21:56:58.3925233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3925310Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3925577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3925678Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3925937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3926036Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3926330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3926454Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3926718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3926799Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3926803Z 2025-08-14T21:56:58.3926914Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3927128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3927195Z return mod(**inputs) 2025-08-14T21:56:58.3927453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3927520Z outputs = self.roberta( 2025-08-14T21:56:58.3927778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3927851Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3928099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3928175Z layer_outputs = layer_module( 2025-08-14T21:56:58.3928385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3928459Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3928719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3928799Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3929054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3929128Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3929413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3929537Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3929787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3929904Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3930109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3930200Z return self.act(input) 2025-08-14T21:56:58.3930204Z 2025-08-14T21:56:58.3930313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3930511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3930578Z return mod(**inputs) 2025-08-14T21:56:58.3930841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3930927Z outputs = self.roberta( 2025-08-14T21:56:58.3931192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3931264Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3931527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3931606Z layer_outputs = layer_module( 2025-08-14T21:56:58.3931846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3931932Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3932211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3932297Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3932555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3932629Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3932918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3933054Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3933311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3933401Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3933405Z 2025-08-14T21:56:58.3933506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3933703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3933779Z return mod(**inputs) 2025-08-14T21:56:58.3934033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3934108Z outputs = self.roberta( 2025-08-14T21:56:58.3934365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3934437Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3934698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3934771Z layer_outputs = layer_module( 2025-08-14T21:56:58.3934989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3935076Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3935332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3935421Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3935665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3935734Z return func(*args, **kwargs) 2025-08-14T21:56:58.3935995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3936065Z self_outputs = self.self( 2025-08-14T21:56:58.3936304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3936399Z return func(*args, **kwargs) 2025-08-14T21:56:58.3936659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3936873Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3936894Z 2025-08-14T21:56:58.3936998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3937191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3937264Z return mod(**inputs) 2025-08-14T21:56:58.3937521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3937596Z outputs = self.roberta( 2025-08-14T21:56:58.3937864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3937938Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3939303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3939401Z layer_outputs = layer_module( 2025-08-14T21:56:58.3939616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3939701Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3939950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3940037Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3940270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3940339Z return func(*args, **kwargs) 2025-08-14T21:56:58.3940600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3940670Z self_outputs = self.self( 2025-08-14T21:56:58.3940910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3940978Z return func(*args, **kwargs) 2025-08-14T21:56:58.3941238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3941319Z self.key(current_states) 2025-08-14T21:56:58.3941322Z 2025-08-14T21:56:58.3941423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3941619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3941694Z return mod(**inputs) 2025-08-14T21:56:58.3941949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3942025Z outputs = self.roberta( 2025-08-14T21:56:58.3942279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3942352Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3942616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3942689Z layer_outputs = layer_module( 2025-08-14T21:56:58.3942905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3942989Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3943244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3943334Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3943604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3943671Z return func(*args, **kwargs) 2025-08-14T21:56:58.3943934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3944020Z self_outputs = self.self( 2025-08-14T21:56:58.3944266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3944336Z return func(*args, **kwargs) 2025-08-14T21:56:58.3944598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3944678Z self.value(current_states) 2025-08-14T21:56:58.3944683Z 2025-08-14T21:56:58.3944762Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3944867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3945090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3945157Z return mod(**inputs) 2025-08-14T21:56:58.3945445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3945519Z outputs = self.roberta( 2025-08-14T21:56:58.3945793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3945878Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3946152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3946226Z layer_outputs = layer_module( 2025-08-14T21:56:58.3946469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3946556Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3946842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3946924Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3947160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3947238Z return func(*args, **kwargs) 2025-08-14T21:56:58.3947496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3947572Z self_outputs = self.self( 2025-08-14T21:56:58.3947810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3947877Z return func(*args, **kwargs) 2025-08-14T21:56:58.3948146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3948282Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3948285Z 2025-08-14T21:56:58.3948386Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3948590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3948658Z return mod(**inputs) 2025-08-14T21:56:58.3948921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3948988Z outputs = self.roberta( 2025-08-14T21:56:58.3949245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3949325Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3949583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3949684Z layer_outputs = layer_module( 2025-08-14T21:56:58.3949900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3949978Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3950242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3950343Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3950580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3950661Z return func(*args, **kwargs) 2025-08-14T21:56:58.3950917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3951056Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3951332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3951417Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3951420Z 2025-08-14T21:56:58.3951548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3951743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3951818Z return mod(**inputs) 2025-08-14T21:56:58.3952070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3952138Z outputs = self.roberta( 2025-08-14T21:56:58.3952400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3952470Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3952728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3952807Z layer_outputs = layer_module( 2025-08-14T21:56:58.3953041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3953128Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3953410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3953498Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3953777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3953856Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3954183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3954319Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3954596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3954689Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3954693Z 2025-08-14T21:56:58.3954800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3955016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3955094Z return mod(**inputs) 2025-08-14T21:56:58.3955369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3955446Z outputs = self.roberta( 2025-08-14T21:56:58.3955720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3955890Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3956197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3956277Z layer_outputs = layer_module( 2025-08-14T21:56:58.3956517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3956678Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3956962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3957061Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3957344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3957427Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3957787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3957929Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3958209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3958323Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3958537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3958617Z return self.act(input) 2025-08-14T21:56:58.3958621Z 2025-08-14T21:56:58.3958721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3958923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3958998Z return mod(**inputs) 2025-08-14T21:56:58.3959254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3959332Z outputs = self.roberta( 2025-08-14T21:56:58.3959587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3959660Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3959924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3959996Z layer_outputs = layer_module( 2025-08-14T21:56:58.3960218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3960297Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3960552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3960641Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3960898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3960983Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3961273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3961401Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3961659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3961738Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3961741Z 2025-08-14T21:56:58.3961840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3962042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3962105Z return mod(**inputs) 2025-08-14T21:56:58.3962383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3962452Z outputs = self.roberta( 2025-08-14T21:56:58.3962705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3962783Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3963053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3963121Z layer_outputs = layer_module( 2025-08-14T21:56:58.3963344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3963421Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3963679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3963761Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3964014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3964094Z return func(*args, **kwargs) 2025-08-14T21:56:58.3964358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3964429Z self_outputs = self.self( 2025-08-14T21:56:58.3964665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3964735Z return func(*args, **kwargs) 2025-08-14T21:56:58.3964996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:56:58.3965201Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:56:58.3965206Z 2025-08-14T21:56:58.3965311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3965515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3965579Z return mod(**inputs) 2025-08-14T21:56:58.3965836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3965903Z outputs = self.roberta( 2025-08-14T21:56:58.3966153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3966232Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3966479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3966548Z layer_outputs = layer_module( 2025-08-14T21:56:58.3966765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3966843Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3967100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3967180Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3967410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3967486Z return func(*args, **kwargs) 2025-08-14T21:56:58.3967737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3967811Z self_outputs = self.self( 2025-08-14T21:56:58.3968042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3968109Z return func(*args, **kwargs) 2025-08-14T21:56:58.3968384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:56:58.3968453Z self.key(current_states) 2025-08-14T21:56:58.3968456Z 2025-08-14T21:56:58.3968557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3968755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3968835Z return mod(**inputs) 2025-08-14T21:56:58.3969099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3969163Z outputs = self.roberta( 2025-08-14T21:56:58.3969418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3969496Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3969780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3969860Z layer_outputs = layer_module( 2025-08-14T21:56:58.3970070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3970163Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3970421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3970502Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3970733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3970809Z return func(*args, **kwargs) 2025-08-14T21:56:58.3971059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3971134Z self_outputs = self.self( 2025-08-14T21:56:58.3971368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3971433Z return func(*args, **kwargs) 2025-08-14T21:56:58.3971692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:56:58.3971761Z self.value(current_states) 2025-08-14T21:56:58.3971766Z 2025-08-14T21:56:58.3971845Z cudagraph partition due to non gpu ops 2025-08-14T21:56:58.3971951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3972139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3972210Z return mod(**inputs) 2025-08-14T21:56:58.3972459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3972525Z outputs = self.roberta( 2025-08-14T21:56:58.3972791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3972862Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3973120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3973191Z layer_outputs = layer_module( 2025-08-14T21:56:58.3973408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3973496Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3973753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3973835Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3974078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3974167Z return func(*args, **kwargs) 2025-08-14T21:56:58.3974439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:56:58.3974506Z self_outputs = self.self( 2025-08-14T21:56:58.3974735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3974825Z return func(*args, **kwargs) 2025-08-14T21:56:58.3975077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:56:58.3975206Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:56:58.3975217Z 2025-08-14T21:56:58.3975315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3975504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3975577Z return mod(**inputs) 2025-08-14T21:56:58.3975848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3975915Z outputs = self.roberta( 2025-08-14T21:56:58.3976189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3976264Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3976531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3976602Z layer_outputs = layer_module( 2025-08-14T21:56:58.3976829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3976914Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3977164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:56:58.3977246Z self_attention_outputs = self.attention( 2025-08-14T21:56:58.3977486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:56:58.3977552Z return func(*args, **kwargs) 2025-08-14T21:56:58.3977811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:56:58.3977940Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:56:58.3978189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:56:58.3978281Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3978284Z 2025-08-14T21:56:58.3978381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3978580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3978648Z return mod(**inputs) 2025-08-14T21:56:58.3978898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3978971Z outputs = self.roberta( 2025-08-14T21:56:58.3979223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3979294Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3979554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3979623Z layer_outputs = layer_module( 2025-08-14T21:56:58.3979839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3979914Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3980165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3980269Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3980517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3980592Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3980907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3981021Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3981276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:56:58.3981357Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3981360Z 2025-08-14T21:56:58.3981459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3981674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3981741Z return mod(**inputs) 2025-08-14T21:56:58.3982015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3982083Z outputs = self.roberta( 2025-08-14T21:56:58.3982338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3982422Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3982679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3982753Z layer_outputs = layer_module( 2025-08-14T21:56:58.3982983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3983060Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3983326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3983409Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3983661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3983746Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3984050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:56:58.3984172Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:56:58.3984421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:56:58.3984529Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:56:58.3984747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:58.3984819Z return self.act(input) 2025-08-14T21:56:58.3984823Z 2025-08-14T21:56:58.3984933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3985130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3985196Z return mod(**inputs) 2025-08-14T21:56:58.3985461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:56:58.3985531Z outputs = self.roberta( 2025-08-14T21:56:58.3985787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:56:58.3985868Z encoder_outputs = self.encoder( 2025-08-14T21:56:58.3986126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:56:58.3986225Z layer_outputs = layer_module( 2025-08-14T21:56:58.3986443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:58.3986521Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:58.3986784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:56:58.3986886Z layer_output = apply_chunking_to_forward( 2025-08-14T21:56:58.3987139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:56:58.3987223Z return forward_fn(*input_tensors) 2025-08-14T21:56:58.3987511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:56:58.3987651Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:56:58.3987924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:56:58.3988011Z hidden_states = self.dense(hidden_states) 2025-08-14T21:56:58.3988015Z 2025-08-14T21:56:58.3988140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3988337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3988412Z return mod(**inputs) 2025-08-14T21:56:58.3988679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1016, in forward 2025-08-14T21:56:58.3988783Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:56:58.3989056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1149, in forward 2025-08-14T21:56:58.3989126Z x = self.dense(features) 2025-08-14T21:56:58.3989129Z 2025-08-14T21:56:58.3989233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3989436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3989499Z return mod(**inputs) 2025-08-14T21:56:58.3989772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1016, in forward 2025-08-14T21:56:58.3989870Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:56:58.3990130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1154, in forward 2025-08-14T21:56:58.3990205Z x = self.decoder(x) 2025-08-14T21:56:58.3990209Z 2025-08-14T21:56:58.3990311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:58.3990514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:58.3990579Z return mod(**inputs) 2025-08-14T21:56:58.3990843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1022, in forward 2025-08-14T21:56:58.3990924Z lm_loss = self.loss_function( 2025-08-14T21:56:58.3991166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:56:58.3991339Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:56:58.3991603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:56:58.3991798Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:56:58.3991802Z 2025-08-14T21:57:07.3995224Z Compilation time (from dynamo_timed): 15.442315164 2025-08-14T21:57:07.4115979Z pass 2025-08-14T21:57:07.4116432Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:07.4117590Z TIMING: _recursive_pre_grad_passes:0.00744 _recursive_joint_graph_passes:0.67517 _recursive_post_grad_passes:0.08619 async_compile.wait:0.83603 code_gen:7.78941 inductor_compile:9.01853 backend_compile:12.29996 gc:0.00107 entire_frame_compile:15.44232 total_wall_time:15.44232 2025-08-14T21:57:07.4118725Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:12464 | FakeTensor.__torch_dispatch__:4759 | ProxyTorchDispatchMode.__torch_dispatch__:4539 2025-08-14T21:57:07.4119394Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-08-14T21:57:12.7042748Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:57:12.7043655Z from pkg_resources import resource_filename 2025-08-14T21:57:13.2747265Z 2025-08-14T21:57:14.4351059Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-14T21:57:14.4352178Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-14T21:57:14.4353199Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-14T21:57:14.4354150Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-14T21:57:14.5725077Z 2025-08-14T21:57:14.5725996Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:57:14.5739551Z cpu eval RobertaForQuestionAnswering 2025-08-14T21:57:15.0084749Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:15.2141409Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:15.4238436Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:23.0951037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.0954053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.0954458Z return mod(**inputs) 2025-08-14T21:57:23.0954929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.0955393Z outputs = self.roberta( 2025-08-14T21:57:23.0956056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:57:23.0956555Z embedding_output = self.embeddings( 2025-08-14T21:57:23.0957015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:57:23.0957609Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:57:23.0958551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1576, in create_position_ids_from_input_ids 2025-08-14T21:57:23.0959071Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:57:23.0959225Z 2025-08-14T21:57:23.0959314Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0959545Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0959771Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0959991Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0960532Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0960758Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0960978Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0961189Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0961411Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0961628Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0961900Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0962119Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0962373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.0962768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.0963124Z return mod(**inputs) 2025-08-14T21:57:23.0963526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.0963949Z outputs = self.roberta( 2025-08-14T21:57:23.0964412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:57:23.0964856Z embedding_output = self.embeddings( 2025-08-14T21:57:23.0965336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:57:23.0965899Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:57:23.0966548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-14T21:57:23.0967189Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:57:23.0967462Z 2025-08-14T21:57:23.0967588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.0967975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.0968334Z return mod(**inputs) 2025-08-14T21:57:23.0968748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.0969209Z outputs = self.roberta( 2025-08-14T21:57:23.0969618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:57:23.0970054Z embedding_output = self.embeddings( 2025-08-14T21:57:23.0970483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:57:23.0971046Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:57:23.0971778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-14T21:57:23.0972419Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:57:23.0972686Z 2025-08-14T21:57:23.0972803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.0973222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.0973576Z return mod(**inputs) 2025-08-14T21:57:23.0973980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.0974414Z outputs = self.roberta( 2025-08-14T21:57:23.0974828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.0975255Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.0975683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.0976142Z layer_outputs = layer_module( 2025-08-14T21:57:23.0976529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.0976930Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.0977370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.0977837Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.0978262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.0978684Z return func(*args, **kwargs) 2025-08-14T21:57:23.0979116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.0979553Z self_outputs = self.self( 2025-08-14T21:57:23.0979970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.0980378Z return func(*args, **kwargs) 2025-08-14T21:57:23.0980821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.0981407Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.0981701Z 2025-08-14T21:57:23.0981814Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.0982208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.0982580Z return mod(**inputs) 2025-08-14T21:57:23.0982997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.0983439Z outputs = self.roberta( 2025-08-14T21:57:23.0983858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.0984276Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.0984688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.0985117Z layer_outputs = layer_module( 2025-08-14T21:57:23.0985502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.0985886Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.0986300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.0986729Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.0987133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.0987520Z return func(*args, **kwargs) 2025-08-14T21:57:23.0987923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.0988341Z self_outputs = self.self( 2025-08-14T21:57:23.0988720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.0989105Z return func(*args, **kwargs) 2025-08-14T21:57:23.0989507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.0989930Z self.key(current_states) 2025-08-14T21:57:23.0990056Z 2025-08-14T21:57:23.0990170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.0990566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.0990921Z return mod(**inputs) 2025-08-14T21:57:23.0991327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.0991768Z outputs = self.roberta( 2025-08-14T21:57:23.0992161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.0992576Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.0993009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.0993434Z layer_outputs = layer_module( 2025-08-14T21:57:23.0993814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.0994213Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.0994635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.0995074Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.0995509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.0995985Z return func(*args, **kwargs) 2025-08-14T21:57:23.0996407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.0996835Z self_outputs = self.self( 2025-08-14T21:57:23.0997235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.0997620Z return func(*args, **kwargs) 2025-08-14T21:57:23.0998046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.0998465Z self.value(current_states) 2025-08-14T21:57:23.0998591Z 2025-08-14T21:57:23.0998687Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.0998935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.0999319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.0999663Z return mod(**inputs) 2025-08-14T21:57:23.1000064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1000476Z outputs = self.roberta( 2025-08-14T21:57:23.1000873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1001289Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1001691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1002107Z layer_outputs = layer_module( 2025-08-14T21:57:23.1002475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1002864Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1003280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1003709Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1004114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1004511Z return func(*args, **kwargs) 2025-08-14T21:57:23.1004905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1005315Z self_outputs = self.self( 2025-08-14T21:57:23.1005692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1006073Z return func(*args, **kwargs) 2025-08-14T21:57:23.1006475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1006980Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1007177Z 2025-08-14T21:57:23.1007295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1007669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1008031Z return mod(**inputs) 2025-08-14T21:57:23.1008423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1009017Z outputs = self.roberta( 2025-08-14T21:57:23.1009416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1009829Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1010234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1010756Z layer_outputs = layer_module( 2025-08-14T21:57:23.1011127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1011543Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1011958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1012389Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1012794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1013190Z return func(*args, **kwargs) 2025-08-14T21:57:23.1013585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1014064Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1014542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1014966Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1015105Z 2025-08-14T21:57:23.1015212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1015584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1015934Z return mod(**inputs) 2025-08-14T21:57:23.1016341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1016755Z outputs = self.roberta( 2025-08-14T21:57:23.1017150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1017561Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1017962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1018378Z layer_outputs = layer_module( 2025-08-14T21:57:23.1018746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1019122Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1019532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1019959Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1020360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1020746Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1021168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1021671Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1022109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1022506Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1022651Z 2025-08-14T21:57:23.1022754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1023147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1023471Z return mod(**inputs) 2025-08-14T21:57:23.1023836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1024227Z outputs = self.roberta( 2025-08-14T21:57:23.1024604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1024989Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1025396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1025785Z layer_outputs = layer_module( 2025-08-14T21:57:23.1026141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1026503Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1026925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1027359Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1027785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1028206Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1028663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1029170Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1029617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1030085Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1030497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1030873Z return self.act(input) 2025-08-14T21:57:23.1030987Z 2025-08-14T21:57:23.1031093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1031473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1031840Z return mod(**inputs) 2025-08-14T21:57:23.1032240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1032665Z outputs = self.roberta( 2025-08-14T21:57:23.1033080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1033622Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1034043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1034469Z layer_outputs = layer_module( 2025-08-14T21:57:23.1034837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1035224Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1035658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1036198Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1036667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1037087Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1037551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1038059Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1038573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1039013Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1039174Z 2025-08-14T21:57:23.1039289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1039677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1040027Z return mod(**inputs) 2025-08-14T21:57:23.1040451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1040868Z outputs = self.roberta( 2025-08-14T21:57:23.1041280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1041688Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1042107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1042540Z layer_outputs = layer_module( 2025-08-14T21:57:23.1042915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1043300Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1043731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1044183Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1044586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1044971Z return func(*args, **kwargs) 2025-08-14T21:57:23.1045366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1045769Z self_outputs = self.self( 2025-08-14T21:57:23.1046134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1046544Z return func(*args, **kwargs) 2025-08-14T21:57:23.1046963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1047500Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1047775Z 2025-08-14T21:57:23.1047885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1048274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1048628Z return mod(**inputs) 2025-08-14T21:57:23.1049031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1049465Z outputs = self.roberta( 2025-08-14T21:57:23.1049870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1050298Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1050718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1051125Z layer_outputs = layer_module( 2025-08-14T21:57:23.1051480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1051865Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1052359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1052799Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1053198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1053607Z return func(*args, **kwargs) 2025-08-14T21:57:23.1054025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1054464Z self_outputs = self.self( 2025-08-14T21:57:23.1054847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1055240Z return func(*args, **kwargs) 2025-08-14T21:57:23.1055677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1056082Z self.key(current_states) 2025-08-14T21:57:23.1056195Z 2025-08-14T21:57:23.1056315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1056673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1057004Z return mod(**inputs) 2025-08-14T21:57:23.1057392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1057801Z outputs = self.roberta( 2025-08-14T21:57:23.1058197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1058606Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1059016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1059407Z layer_outputs = layer_module( 2025-08-14T21:57:23.1059751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1060109Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1060491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1060891Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1061266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1061626Z return func(*args, **kwargs) 2025-08-14T21:57:23.1062003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1062389Z self_outputs = self.self( 2025-08-14T21:57:23.1062745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1063100Z return func(*args, **kwargs) 2025-08-14T21:57:23.1063477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1063864Z self.value(current_states) 2025-08-14T21:57:23.1063981Z 2025-08-14T21:57:23.1064068Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1064300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1064672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1065018Z return mod(**inputs) 2025-08-14T21:57:23.1065399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1065806Z outputs = self.roberta( 2025-08-14T21:57:23.1066232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1066649Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1067052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1067465Z layer_outputs = layer_module( 2025-08-14T21:57:23.1067853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1068210Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1068607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1069032Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1069434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1069819Z return func(*args, **kwargs) 2025-08-14T21:57:23.1070236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1070651Z self_outputs = self.self( 2025-08-14T21:57:23.1071038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1071434Z return func(*args, **kwargs) 2025-08-14T21:57:23.1071838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1072315Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1072514Z 2025-08-14T21:57:23.1072623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1073016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1073358Z return mod(**inputs) 2025-08-14T21:57:23.1073761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1074172Z outputs = self.roberta( 2025-08-14T21:57:23.1074573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1074987Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1075388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1075858Z layer_outputs = layer_module( 2025-08-14T21:57:23.1076238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1076618Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1077026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1077459Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1077860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1078264Z return func(*args, **kwargs) 2025-08-14T21:57:23.1078658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1079135Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1079608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1080032Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1080178Z 2025-08-14T21:57:23.1080283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1080643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1080998Z return mod(**inputs) 2025-08-14T21:57:23.1081361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1081752Z outputs = self.roberta( 2025-08-14T21:57:23.1082128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1082529Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1082912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1083301Z layer_outputs = layer_module( 2025-08-14T21:57:23.1083645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1083998Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1084427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1084867Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1085317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1085727Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1086155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1086628Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1087059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1087471Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1087623Z 2025-08-14T21:57:23.1087733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1088111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1088462Z return mod(**inputs) 2025-08-14T21:57:23.1088856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1089267Z outputs = self.roberta( 2025-08-14T21:57:23.1089651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1090034Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1090418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1090806Z layer_outputs = layer_module( 2025-08-14T21:57:23.1091143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1091503Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1091904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1092306Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1092696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1093089Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1093507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1093973Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1094410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1094846Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1095247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1095622Z return self.act(input) 2025-08-14T21:57:23.1095740Z 2025-08-14T21:57:23.1095844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1096204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1096551Z return mod(**inputs) 2025-08-14T21:57:23.1096919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1097309Z outputs = self.roberta( 2025-08-14T21:57:23.1097677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1098058Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1098444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1098849Z layer_outputs = layer_module( 2025-08-14T21:57:23.1099195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1099546Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1099959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1100367Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1100758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1101146Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1101562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1102039Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1102498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1102922Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1103072Z 2025-08-14T21:57:23.1103183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1103560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1103898Z return mod(**inputs) 2025-08-14T21:57:23.1104291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1104702Z outputs = self.roberta( 2025-08-14T21:57:23.1105089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1105514Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1105925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1106338Z layer_outputs = layer_module( 2025-08-14T21:57:23.1106696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1107075Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1107496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1107930Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1108329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1108871Z return func(*args, **kwargs) 2025-08-14T21:57:23.1109279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1109687Z self_outputs = self.self( 2025-08-14T21:57:23.1110150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1110553Z return func(*args, **kwargs) 2025-08-14T21:57:23.1110949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1111535Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1111823Z 2025-08-14T21:57:23.1111934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1112317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1112675Z return mod(**inputs) 2025-08-14T21:57:23.1113063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1113491Z outputs = self.roberta( 2025-08-14T21:57:23.1113914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1114326Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1114759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1115177Z layer_outputs = layer_module( 2025-08-14T21:57:23.1115550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1116017Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1116456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1116898Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1117324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1117737Z return func(*args, **kwargs) 2025-08-14T21:57:23.1118142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1118560Z self_outputs = self.self( 2025-08-14T21:57:23.1118946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1119343Z return func(*args, **kwargs) 2025-08-14T21:57:23.1119755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1120148Z self.key(current_states) 2025-08-14T21:57:23.1120262Z 2025-08-14T21:57:23.1120366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1120728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1121069Z return mod(**inputs) 2025-08-14T21:57:23.1121477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1121903Z outputs = self.roberta( 2025-08-14T21:57:23.1122302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1122721Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1123124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1123541Z layer_outputs = layer_module( 2025-08-14T21:57:23.1123910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1124290Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1124721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1125178Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1125557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1125926Z return func(*args, **kwargs) 2025-08-14T21:57:23.1126303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1126712Z self_outputs = self.self( 2025-08-14T21:57:23.1127064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1127437Z return func(*args, **kwargs) 2025-08-14T21:57:23.1127818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1128209Z self.value(current_states) 2025-08-14T21:57:23.1128328Z 2025-08-14T21:57:23.1128412Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1128678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1129041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1129385Z return mod(**inputs) 2025-08-14T21:57:23.1129751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1130140Z outputs = self.roberta( 2025-08-14T21:57:23.1130508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1130888Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1131275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1131665Z layer_outputs = layer_module( 2025-08-14T21:57:23.1132014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1132368Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1132764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1133168Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1133550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1133944Z return func(*args, **kwargs) 2025-08-14T21:57:23.1134348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1134755Z self_outputs = self.self( 2025-08-14T21:57:23.1135125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1135518Z return func(*args, **kwargs) 2025-08-14T21:57:23.1135923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1136362Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1136552Z 2025-08-14T21:57:23.1136654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1137012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1137331Z return mod(**inputs) 2025-08-14T21:57:23.1137694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1138082Z outputs = self.roberta( 2025-08-14T21:57:23.1138449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1138835Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1139240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1139627Z layer_outputs = layer_module( 2025-08-14T21:57:23.1139971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1140324Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1140738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1141159Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1141559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1141952Z return func(*args, **kwargs) 2025-08-14T21:57:23.1142352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1142847Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1143292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1143706Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1143853Z 2025-08-14T21:57:23.1143957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1144317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1144632Z return mod(**inputs) 2025-08-14T21:57:23.1145004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1145396Z outputs = self.roberta( 2025-08-14T21:57:23.1145768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1146164Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1146552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1146943Z layer_outputs = layer_module( 2025-08-14T21:57:23.1147285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1147650Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1148071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1148501Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1148916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1149334Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1149787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1150291Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1150750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1151181Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1151328Z 2025-08-14T21:57:23.1151444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1151818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1152156Z return mod(**inputs) 2025-08-14T21:57:23.1152552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1152965Z outputs = self.roberta( 2025-08-14T21:57:23.1153351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1153797Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1154206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1154620Z layer_outputs = layer_module( 2025-08-14T21:57:23.1154976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1155377Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1155868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1156306Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1156746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1157186Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1157656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1158155Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1158661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1159142Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1159550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1159920Z return self.act(input) 2025-08-14T21:57:23.1160046Z 2025-08-14T21:57:23.1160156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1160540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1160894Z return mod(**inputs) 2025-08-14T21:57:23.1161296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1161716Z outputs = self.roberta( 2025-08-14T21:57:23.1162130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1162547Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1162960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1163382Z layer_outputs = layer_module( 2025-08-14T21:57:23.1163744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1164127Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1164548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1164973Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1165380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1165798Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1166248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1166765Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1167246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1167658Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1167797Z 2025-08-14T21:57:23.1167909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1168263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1168629Z return mod(**inputs) 2025-08-14T21:57:23.1169021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1169433Z outputs = self.roberta( 2025-08-14T21:57:23.1169830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1170240Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1170633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1171019Z layer_outputs = layer_module( 2025-08-14T21:57:23.1171365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1171733Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1172134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1172521Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1172894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1173292Z return func(*args, **kwargs) 2025-08-14T21:57:23.1173687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1174085Z self_outputs = self.self( 2025-08-14T21:57:23.1174461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1174850Z return func(*args, **kwargs) 2025-08-14T21:57:23.1175239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1175781Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1176055Z 2025-08-14T21:57:23.1176165Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1176544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1176870Z return mod(**inputs) 2025-08-14T21:57:23.1177241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1177633Z outputs = self.roberta( 2025-08-14T21:57:23.1178014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1178418Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1178819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1179227Z layer_outputs = layer_module( 2025-08-14T21:57:23.1179583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1179961Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1180359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1180781Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1181177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1181565Z return func(*args, **kwargs) 2025-08-14T21:57:23.1181963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1182367Z self_outputs = self.self( 2025-08-14T21:57:23.1182746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1183161Z return func(*args, **kwargs) 2025-08-14T21:57:23.1183560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1183965Z self.key(current_states) 2025-08-14T21:57:23.1184094Z 2025-08-14T21:57:23.1184205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1184606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1184941Z return mod(**inputs) 2025-08-14T21:57:23.1185335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1185745Z outputs = self.roberta( 2025-08-14T21:57:23.1186142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1186545Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1187010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1187426Z layer_outputs = layer_module( 2025-08-14T21:57:23.1187805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1188185Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1188604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1189027Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1189417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1189808Z return func(*args, **kwargs) 2025-08-14T21:57:23.1190205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1190623Z self_outputs = self.self( 2025-08-14T21:57:23.1190995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1191384Z return func(*args, **kwargs) 2025-08-14T21:57:23.1191786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1192191Z self.value(current_states) 2025-08-14T21:57:23.1192322Z 2025-08-14T21:57:23.1192407Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1192660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1193037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1193384Z return mod(**inputs) 2025-08-14T21:57:23.1193786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1194210Z outputs = self.roberta( 2025-08-14T21:57:23.1194598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1195010Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1195417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1195900Z layer_outputs = layer_module( 2025-08-14T21:57:23.1196284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1196682Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1197116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1197567Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1197975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1198412Z return func(*args, **kwargs) 2025-08-14T21:57:23.1198809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1199212Z self_outputs = self.self( 2025-08-14T21:57:23.1199599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1200024Z return func(*args, **kwargs) 2025-08-14T21:57:23.1200430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1200875Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1201066Z 2025-08-14T21:57:23.1201170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1201535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1201889Z return mod(**inputs) 2025-08-14T21:57:23.1202255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1202667Z outputs = self.roberta( 2025-08-14T21:57:23.1203037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1203422Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1203808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1204194Z layer_outputs = layer_module( 2025-08-14T21:57:23.1204542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1204895Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1205293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1205715Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1206106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1206507Z return func(*args, **kwargs) 2025-08-14T21:57:23.1206910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1207356Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1207787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1208190Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1208327Z 2025-08-14T21:57:23.1208438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1208911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1209242Z return mod(**inputs) 2025-08-14T21:57:23.1209659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1210070Z outputs = self.roberta( 2025-08-14T21:57:23.1210436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1210827Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1211212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1211605Z layer_outputs = layer_module( 2025-08-14T21:57:23.1211940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1212302Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1212758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1213157Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1213558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1213976Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1214401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1214864Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1215304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1215708Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1215845Z 2025-08-14T21:57:23.1215958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1216341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1216667Z return mod(**inputs) 2025-08-14T21:57:23.1217071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1217456Z outputs = self.roberta( 2025-08-14T21:57:23.1217833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1218223Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1218609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1218991Z layer_outputs = layer_module( 2025-08-14T21:57:23.1219335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1219700Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1220107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1220505Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1220900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1221293Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1221706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1222171Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1222607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1223036Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1223411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1223756Z return self.act(input) 2025-08-14T21:57:23.1223873Z 2025-08-14T21:57:23.1223994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1224380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1224701Z return mod(**inputs) 2025-08-14T21:57:23.1225074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1225459Z outputs = self.roberta( 2025-08-14T21:57:23.1225821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1226217Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1226591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1226999Z layer_outputs = layer_module( 2025-08-14T21:57:23.1227334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1227696Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1228140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1228552Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1228947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1229343Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1229785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1230321Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1230797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1231258Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1231407Z 2025-08-14T21:57:23.1231526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1231898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1232242Z return mod(**inputs) 2025-08-14T21:57:23.1232632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1233048Z outputs = self.roberta( 2025-08-14T21:57:23.1233470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1233903Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1234336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1234757Z layer_outputs = layer_module( 2025-08-14T21:57:23.1235142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1235542Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1236041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1236495Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1236914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1237310Z return func(*args, **kwargs) 2025-08-14T21:57:23.1237679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1238069Z self_outputs = self.self( 2025-08-14T21:57:23.1238435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1238813Z return func(*args, **kwargs) 2025-08-14T21:57:23.1239202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1239762Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1240035Z 2025-08-14T21:57:23.1240155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1240537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1240898Z return mod(**inputs) 2025-08-14T21:57:23.1241305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1241813Z outputs = self.roberta( 2025-08-14T21:57:23.1242199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1242616Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1243024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1243456Z layer_outputs = layer_module( 2025-08-14T21:57:23.1243810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1244189Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1244608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1245025Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1245452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1245849Z return func(*args, **kwargs) 2025-08-14T21:57:23.1246268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1246675Z self_outputs = self.self( 2025-08-14T21:57:23.1247058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1247450Z return func(*args, **kwargs) 2025-08-14T21:57:23.1247840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1248252Z self.key(current_states) 2025-08-14T21:57:23.1248377Z 2025-08-14T21:57:23.1248485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1248864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1249205Z return mod(**inputs) 2025-08-14T21:57:23.1249596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1250011Z outputs = self.roberta( 2025-08-14T21:57:23.1250399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1250814Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1251225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1251640Z layer_outputs = layer_module( 2025-08-14T21:57:23.1251999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1252380Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1252805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1253235Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1253633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1254031Z return func(*args, **kwargs) 2025-08-14T21:57:23.1254435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1254839Z self_outputs = self.self( 2025-08-14T21:57:23.1255224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1255612Z return func(*args, **kwargs) 2025-08-14T21:57:23.1256017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1256448Z self.value(current_states) 2025-08-14T21:57:23.1256577Z 2025-08-14T21:57:23.1256665Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1256920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1257291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1257635Z return mod(**inputs) 2025-08-14T21:57:23.1258079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1258497Z outputs = self.roberta( 2025-08-14T21:57:23.1258886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1259301Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1259713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1260118Z layer_outputs = layer_module( 2025-08-14T21:57:23.1260503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1260887Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1261335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1261736Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1262139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1262529Z return func(*args, **kwargs) 2025-08-14T21:57:23.1262926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1263329Z self_outputs = self.self( 2025-08-14T21:57:23.1263703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1264098Z return func(*args, **kwargs) 2025-08-14T21:57:23.1264491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1264942Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1265132Z 2025-08-14T21:57:23.1265237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1265607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1265943Z return mod(**inputs) 2025-08-14T21:57:23.1266353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1266822Z outputs = self.roberta( 2025-08-14T21:57:23.1267220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1267654Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1268068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1268479Z layer_outputs = layer_module( 2025-08-14T21:57:23.1268837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1269200Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1269595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1270016Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1270415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1270813Z return func(*args, **kwargs) 2025-08-14T21:57:23.1271218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1271712Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1272181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1272606Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1272773Z 2025-08-14T21:57:23.1272888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1273259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1273600Z return mod(**inputs) 2025-08-14T21:57:23.1274011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1274435Z outputs = self.roberta( 2025-08-14T21:57:23.1274817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1275250Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1275660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1276174Z layer_outputs = layer_module( 2025-08-14T21:57:23.1276569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1276968Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1277402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1277841Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1278289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1278710Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1279157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1279664Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1280130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1280563Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1280710Z 2025-08-14T21:57:23.1280820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1281199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1281548Z return mod(**inputs) 2025-08-14T21:57:23.1281941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1282346Z outputs = self.roberta( 2025-08-14T21:57:23.1282744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1283162Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1283561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1283976Z layer_outputs = layer_module( 2025-08-14T21:57:23.1284348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1284731Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1285141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1285568Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1285989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1286420Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1286861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1287356Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1287814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1288298Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1288700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1289060Z return self.act(input) 2025-08-14T21:57:23.1289180Z 2025-08-14T21:57:23.1289301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1289683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1290033Z return mod(**inputs) 2025-08-14T21:57:23.1290451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1290861Z outputs = self.roberta( 2025-08-14T21:57:23.1291275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1291697Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1292107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1292521Z layer_outputs = layer_module( 2025-08-14T21:57:23.1292867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1293241Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1293660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1294087Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1294514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1294933Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1295369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1295878Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1296358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1296786Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1296932Z 2025-08-14T21:57:23.1297045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1297424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1297765Z return mod(**inputs) 2025-08-14T21:57:23.1298159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1298569Z outputs = self.roberta( 2025-08-14T21:57:23.1298963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1299383Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1299780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1300191Z layer_outputs = layer_module( 2025-08-14T21:57:23.1300558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1300939Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1301371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1301798Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1302209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1302600Z return func(*args, **kwargs) 2025-08-14T21:57:23.1303024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1303431Z self_outputs = self.self( 2025-08-14T21:57:23.1303811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1304196Z return func(*args, **kwargs) 2025-08-14T21:57:23.1304597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1305179Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1305458Z 2025-08-14T21:57:23.1305576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1305966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1306312Z return mod(**inputs) 2025-08-14T21:57:23.1306710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1307117Z outputs = self.roberta( 2025-08-14T21:57:23.1307515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1307934Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1308344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1309067Z layer_outputs = layer_module( 2025-08-14T21:57:23.1309447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1309835Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1310269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1310703Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1311121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1311531Z return func(*args, **kwargs) 2025-08-14T21:57:23.1311939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1312366Z self_outputs = self.self( 2025-08-14T21:57:23.1312762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1313166Z return func(*args, **kwargs) 2025-08-14T21:57:23.1313569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1313992Z self.key(current_states) 2025-08-14T21:57:23.1314118Z 2025-08-14T21:57:23.1314241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1314623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1314975Z return mod(**inputs) 2025-08-14T21:57:23.1315377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1315851Z outputs = self.roberta( 2025-08-14T21:57:23.1316269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1316762Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1317189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1317597Z layer_outputs = layer_module( 2025-08-14T21:57:23.1317965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1318383Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1318804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1319225Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1319640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1320041Z return func(*args, **kwargs) 2025-08-14T21:57:23.1320463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1320872Z self_outputs = self.self( 2025-08-14T21:57:23.1321256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1321693Z return func(*args, **kwargs) 2025-08-14T21:57:23.1322085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1322494Z self.value(current_states) 2025-08-14T21:57:23.1322623Z 2025-08-14T21:57:23.1322709Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1322961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1323330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1323678Z return mod(**inputs) 2025-08-14T21:57:23.1324071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1324482Z outputs = self.roberta( 2025-08-14T21:57:23.1324871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1325272Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1325654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1326034Z layer_outputs = layer_module( 2025-08-14T21:57:23.1326377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1326733Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1327117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1327520Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1327921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1328309Z return func(*args, **kwargs) 2025-08-14T21:57:23.1328680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1329071Z self_outputs = self.self( 2025-08-14T21:57:23.1329429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1329795Z return func(*args, **kwargs) 2025-08-14T21:57:23.1330165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1330612Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1330797Z 2025-08-14T21:57:23.1330908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1331296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1331617Z return mod(**inputs) 2025-08-14T21:57:23.1331993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1332379Z outputs = self.roberta( 2025-08-14T21:57:23.1332767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1333159Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1333546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1333934Z layer_outputs = layer_module( 2025-08-14T21:57:23.1334270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1334630Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1335048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1335442Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1335837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1336211Z return func(*args, **kwargs) 2025-08-14T21:57:23.1336599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1337038Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1337480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1337888Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1338026Z 2025-08-14T21:57:23.1338130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1338493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1338819Z return mod(**inputs) 2025-08-14T21:57:23.1339201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1339607Z outputs = self.roberta( 2025-08-14T21:57:23.1340002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1340414Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1340824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1341220Z layer_outputs = layer_module( 2025-08-14T21:57:23.1341562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1341925Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1342314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1342722Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1343120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1343515Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1343960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1344460Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1344925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1345353Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1345535Z 2025-08-14T21:57:23.1345645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1346027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1346372Z return mod(**inputs) 2025-08-14T21:57:23.1346761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1347211Z outputs = self.roberta( 2025-08-14T21:57:23.1347612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1348029Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1348439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1348850Z layer_outputs = layer_module( 2025-08-14T21:57:23.1349238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1349615Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1350051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1350487Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1350909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1351312Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1351758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1352257Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1352720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1353177Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1353581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1353946Z return self.act(input) 2025-08-14T21:57:23.1354067Z 2025-08-14T21:57:23.1354179Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1354564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1354907Z return mod(**inputs) 2025-08-14T21:57:23.1355298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1355703Z outputs = self.roberta( 2025-08-14T21:57:23.1356177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1356596Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1357030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1357457Z layer_outputs = layer_module( 2025-08-14T21:57:23.1357833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1358235Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1358644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1359074Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1359504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1359927Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1360375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1360931Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1361418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1361847Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1362025Z 2025-08-14T21:57:23.1362390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1362784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1362854Z return mod(**inputs) 2025-08-14T21:57:23.1363148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1363223Z outputs = self.roberta( 2025-08-14T21:57:23.1363504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1363617Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1363902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1364005Z layer_outputs = layer_module( 2025-08-14T21:57:23.1364250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1364340Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1364634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1364724Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1364995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1365071Z return func(*args, **kwargs) 2025-08-14T21:57:23.1365357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1365445Z self_outputs = self.self( 2025-08-14T21:57:23.1365710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1365795Z return func(*args, **kwargs) 2025-08-14T21:57:23.1366058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1366261Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1366265Z 2025-08-14T21:57:23.1366374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1366574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1366639Z return mod(**inputs) 2025-08-14T21:57:23.1366913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1366982Z outputs = self.roberta( 2025-08-14T21:57:23.1367251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1367325Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1367586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1367666Z layer_outputs = layer_module( 2025-08-14T21:57:23.1367887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1367968Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1368236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1368340Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1368601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1368669Z return func(*args, **kwargs) 2025-08-14T21:57:23.1368919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1369015Z self_outputs = self.self( 2025-08-14T21:57:23.1369245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1369318Z return func(*args, **kwargs) 2025-08-14T21:57:23.1369568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1369636Z self.key(current_states) 2025-08-14T21:57:23.1369639Z 2025-08-14T21:57:23.1369748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1369961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1370025Z return mod(**inputs) 2025-08-14T21:57:23.1370305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1370376Z outputs = self.roberta( 2025-08-14T21:57:23.1370642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1370714Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1370970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1371050Z layer_outputs = layer_module( 2025-08-14T21:57:23.1371267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1371345Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1371614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1371696Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1371941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1372012Z return func(*args, **kwargs) 2025-08-14T21:57:23.1372275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1372351Z self_outputs = self.self( 2025-08-14T21:57:23.1372582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1372655Z return func(*args, **kwargs) 2025-08-14T21:57:23.1372906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1372978Z self.value(current_states) 2025-08-14T21:57:23.1372981Z 2025-08-14T21:57:23.1373067Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1373166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1373362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1373436Z return mod(**inputs) 2025-08-14T21:57:23.1373696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1373768Z outputs = self.roberta( 2025-08-14T21:57:23.1374026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1374097Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1374360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1374450Z layer_outputs = layer_module( 2025-08-14T21:57:23.1374667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1374752Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1375009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1375116Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1375355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1375428Z return func(*args, **kwargs) 2025-08-14T21:57:23.1375695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1375766Z self_outputs = self.self( 2025-08-14T21:57:23.1376033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1376103Z return func(*args, **kwargs) 2025-08-14T21:57:23.1376390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1376529Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1376535Z 2025-08-14T21:57:23.1376637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1376830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1376902Z return mod(**inputs) 2025-08-14T21:57:23.1377177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1377255Z outputs = self.roberta( 2025-08-14T21:57:23.1377529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1377606Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1377883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1377959Z layer_outputs = layer_module( 2025-08-14T21:57:23.1378189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1378281Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1378555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1378648Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1378900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1378968Z return func(*args, **kwargs) 2025-08-14T21:57:23.1379234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1379362Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1379633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1379716Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1379719Z 2025-08-14T21:57:23.1379818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1380016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1380081Z return mod(**inputs) 2025-08-14T21:57:23.1380335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1380410Z outputs = self.roberta( 2025-08-14T21:57:23.1380684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1380761Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1381018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1381089Z layer_outputs = layer_module( 2025-08-14T21:57:23.1381331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1381409Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1381671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1381755Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1382009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1382112Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1382408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1382545Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1382811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1382896Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1382899Z 2025-08-14T21:57:23.1383007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1383201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1383265Z return mod(**inputs) 2025-08-14T21:57:23.1383530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1383601Z outputs = self.roberta( 2025-08-14T21:57:23.1383878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1383955Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1384227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1384311Z layer_outputs = layer_module( 2025-08-14T21:57:23.1384540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1384621Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1384905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1384986Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1385250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1385327Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1385616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1385742Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1385998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1386133Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1386343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1386414Z return self.act(input) 2025-08-14T21:57:23.1386418Z 2025-08-14T21:57:23.1386528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1386734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1386824Z return mod(**inputs) 2025-08-14T21:57:23.1387120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1387194Z outputs = self.roberta( 2025-08-14T21:57:23.1387476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1387608Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1387880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1387963Z layer_outputs = layer_module( 2025-08-14T21:57:23.1388192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1388272Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1388585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1388671Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1388949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1389027Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1389318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1389454Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1389712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1389801Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1389805Z 2025-08-14T21:57:23.1389906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1390106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1390180Z return mod(**inputs) 2025-08-14T21:57:23.1390446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1390522Z outputs = self.roberta( 2025-08-14T21:57:23.1390781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1390852Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1391117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1391186Z layer_outputs = layer_module( 2025-08-14T21:57:23.1391402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1391492Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1391766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1391860Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1392127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1392202Z return func(*args, **kwargs) 2025-08-14T21:57:23.1392482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1392555Z self_outputs = self.self( 2025-08-14T21:57:23.1392816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1392896Z return func(*args, **kwargs) 2025-08-14T21:57:23.1393172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1393418Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1393422Z 2025-08-14T21:57:23.1393532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1393750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1393847Z return mod(**inputs) 2025-08-14T21:57:23.1394129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1394206Z outputs = self.roberta( 2025-08-14T21:57:23.1394482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1394557Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1394859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1394937Z layer_outputs = layer_module( 2025-08-14T21:57:23.1395187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1395293Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1395572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1395667Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1395997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1396077Z return func(*args, **kwargs) 2025-08-14T21:57:23.1396360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1396434Z self_outputs = self.self( 2025-08-14T21:57:23.1396706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1396779Z return func(*args, **kwargs) 2025-08-14T21:57:23.1397053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1397136Z self.key(current_states) 2025-08-14T21:57:23.1397141Z 2025-08-14T21:57:23.1397250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1397468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1397548Z return mod(**inputs) 2025-08-14T21:57:23.1397825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1397906Z outputs = self.roberta( 2025-08-14T21:57:23.1398180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1398259Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1398541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1398617Z layer_outputs = layer_module( 2025-08-14T21:57:23.1398848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1398940Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1399210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1399306Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1399565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1399639Z return func(*args, **kwargs) 2025-08-14T21:57:23.1399944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1400017Z self_outputs = self.self( 2025-08-14T21:57:23.1400283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1400355Z return func(*args, **kwargs) 2025-08-14T21:57:23.1400644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1400726Z self.value(current_states) 2025-08-14T21:57:23.1400730Z 2025-08-14T21:57:23.1400815Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1400925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1401147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1401221Z return mod(**inputs) 2025-08-14T21:57:23.1401549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1401623Z outputs = self.roberta( 2025-08-14T21:57:23.1401912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1401998Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1402268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1402343Z layer_outputs = layer_module( 2025-08-14T21:57:23.1402579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1402661Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1402942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1403035Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1403289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1403374Z return func(*args, **kwargs) 2025-08-14T21:57:23.1403648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1403735Z self_outputs = self.self( 2025-08-14T21:57:23.1404004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1404079Z return func(*args, **kwargs) 2025-08-14T21:57:23.1404361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1404504Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1404507Z 2025-08-14T21:57:23.1404620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1404837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1404910Z return mod(**inputs) 2025-08-14T21:57:23.1405206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1405283Z outputs = self.roberta( 2025-08-14T21:57:23.1405555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1405642Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1405916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1405994Z layer_outputs = layer_module( 2025-08-14T21:57:23.1406233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1406338Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1406615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1406699Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1406951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1407049Z return func(*args, **kwargs) 2025-08-14T21:57:23.1407321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1407463Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1407738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1407826Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1407831Z 2025-08-14T21:57:23.1407964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1408171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1408255Z return mod(**inputs) 2025-08-14T21:57:23.1408548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1408621Z outputs = self.roberta( 2025-08-14T21:57:23.1409046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1409129Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1409401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1409484Z layer_outputs = layer_module( 2025-08-14T21:57:23.1409715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1409806Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1410078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1410167Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1410442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1410524Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1410836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1410971Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1411244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1411338Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1411342Z 2025-08-14T21:57:23.1411451Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1411659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1411736Z return mod(**inputs) 2025-08-14T21:57:23.1412016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1412094Z outputs = self.roberta( 2025-08-14T21:57:23.1412369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1412444Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1412725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1412799Z layer_outputs = layer_module( 2025-08-14T21:57:23.1413080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1413171Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1413444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1413567Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1413835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1413914Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1414226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1414352Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1414663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1414787Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1415036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1415121Z return self.act(input) 2025-08-14T21:57:23.1415127Z 2025-08-14T21:57:23.1415236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1415443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1415518Z return mod(**inputs) 2025-08-14T21:57:23.1415795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1415872Z outputs = self.roberta( 2025-08-14T21:57:23.1416146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1416236Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1416503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1416576Z layer_outputs = layer_module( 2025-08-14T21:57:23.1416807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1416887Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1417137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1417226Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1417470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1417543Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1417833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1417961Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1418223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1418303Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1418306Z 2025-08-14T21:57:23.1418404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1418603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1418666Z return mod(**inputs) 2025-08-14T21:57:23.1418924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1418992Z outputs = self.roberta( 2025-08-14T21:57:23.1419246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1419344Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1419594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1419663Z layer_outputs = layer_module( 2025-08-14T21:57:23.1419902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1419978Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1420234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1420313Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1420543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1420622Z return func(*args, **kwargs) 2025-08-14T21:57:23.1420893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1420965Z self_outputs = self.self( 2025-08-14T21:57:23.1421229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1421301Z return func(*args, **kwargs) 2025-08-14T21:57:23.1421562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1421772Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1421776Z 2025-08-14T21:57:23.1421879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1422084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1422151Z return mod(**inputs) 2025-08-14T21:57:23.1422427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1422494Z outputs = self.roberta( 2025-08-14T21:57:23.1422761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1422842Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1423106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1423175Z layer_outputs = layer_module( 2025-08-14T21:57:23.1423402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1423478Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1423748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1423832Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1424075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1424150Z return func(*args, **kwargs) 2025-08-14T21:57:23.1424415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1424491Z self_outputs = self.self( 2025-08-14T21:57:23.1424736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1424804Z return func(*args, **kwargs) 2025-08-14T21:57:23.1425074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1425144Z self.key(current_states) 2025-08-14T21:57:23.1425147Z 2025-08-14T21:57:23.1425271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1425473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1425538Z return mod(**inputs) 2025-08-14T21:57:23.1425805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1425901Z outputs = self.roberta( 2025-08-14T21:57:23.1426158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1426238Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1426497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1426567Z layer_outputs = layer_module( 2025-08-14T21:57:23.1426797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1426896Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1427164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1427259Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1427496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1427573Z return func(*args, **kwargs) 2025-08-14T21:57:23.1427827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1427901Z self_outputs = self.self( 2025-08-14T21:57:23.1428146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1428211Z return func(*args, **kwargs) 2025-08-14T21:57:23.1428469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1428541Z self.value(current_states) 2025-08-14T21:57:23.1428545Z 2025-08-14T21:57:23.1428624Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1428736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1428931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1429004Z return mod(**inputs) 2025-08-14T21:57:23.1429262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1429329Z outputs = self.roberta( 2025-08-14T21:57:23.1429590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1429662Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1429917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1429996Z layer_outputs = layer_module( 2025-08-14T21:57:23.1430211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1430297Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1430563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1430646Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1430928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1430999Z return func(*args, **kwargs) 2025-08-14T21:57:23.1431276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1431369Z self_outputs = self.self( 2025-08-14T21:57:23.1431629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1431709Z return func(*args, **kwargs) 2025-08-14T21:57:23.1431979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1432135Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1432147Z 2025-08-14T21:57:23.1432254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1432462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1432538Z return mod(**inputs) 2025-08-14T21:57:23.1432835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1432906Z outputs = self.roberta( 2025-08-14T21:57:23.1433200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1433278Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1433571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1433648Z layer_outputs = layer_module( 2025-08-14T21:57:23.1433879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1433969Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1434244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1434329Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1434589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1434663Z return func(*args, **kwargs) 2025-08-14T21:57:23.1434951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1435085Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1435361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1435459Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1435464Z 2025-08-14T21:57:23.1435574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1435863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1435943Z return mod(**inputs) 2025-08-14T21:57:23.1436244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1436327Z outputs = self.roberta( 2025-08-14T21:57:23.1436605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1436683Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1436972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1437048Z layer_outputs = layer_module( 2025-08-14T21:57:23.1437288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1437369Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1437642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1437738Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1438011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1438119Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1438433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1438561Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1438866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1438952Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1438955Z 2025-08-14T21:57:23.1439060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1439276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1439344Z return mod(**inputs) 2025-08-14T21:57:23.1439647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1439722Z outputs = self.roberta( 2025-08-14T21:57:23.1439994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1440091Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1440363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1440439Z layer_outputs = layer_module( 2025-08-14T21:57:23.1440673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1440755Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1441032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1441117Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1441387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1441474Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1441777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1441910Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1442179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1442297Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1442525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1442597Z return self.act(input) 2025-08-14T21:57:23.1442601Z 2025-08-14T21:57:23.1442708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1442925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1442995Z return mod(**inputs) 2025-08-14T21:57:23.1443283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1443356Z outputs = self.roberta( 2025-08-14T21:57:23.1443626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1443709Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1443980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1444063Z layer_outputs = layer_module( 2025-08-14T21:57:23.1444293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1444395Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1444673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1444760Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1445028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1445133Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1445446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1445584Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1445842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1445924Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1445928Z 2025-08-14T21:57:23.1446067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1446260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1446331Z return mod(**inputs) 2025-08-14T21:57:23.1446601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1446671Z outputs = self.roberta( 2025-08-14T21:57:23.1446926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1446996Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1447246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1447321Z layer_outputs = layer_module( 2025-08-14T21:57:23.1447534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1447617Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1447874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1447954Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1448205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1448274Z return func(*args, **kwargs) 2025-08-14T21:57:23.1448530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1448607Z self_outputs = self.self( 2025-08-14T21:57:23.1448850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1448926Z return func(*args, **kwargs) 2025-08-14T21:57:23.1449188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1449395Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1449400Z 2025-08-14T21:57:23.1449508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1449707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1449789Z return mod(**inputs) 2025-08-14T21:57:23.1450045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1450110Z outputs = self.roberta( 2025-08-14T21:57:23.1450367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1450437Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1450715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1450790Z layer_outputs = layer_module( 2025-08-14T21:57:23.1451002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1451087Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1451358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1451436Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1451675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1451742Z return func(*args, **kwargs) 2025-08-14T21:57:23.1452005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1452095Z self_outputs = self.self( 2025-08-14T21:57:23.1452334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1452409Z return func(*args, **kwargs) 2025-08-14T21:57:23.1452681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1452753Z self.key(current_states) 2025-08-14T21:57:23.1452757Z 2025-08-14T21:57:23.1452866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1453062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1453133Z return mod(**inputs) 2025-08-14T21:57:23.1453405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1453469Z outputs = self.roberta( 2025-08-14T21:57:23.1453728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1453796Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1454049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1454127Z layer_outputs = layer_module( 2025-08-14T21:57:23.1454338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1454420Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1454671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1454748Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1454986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1455054Z return func(*args, **kwargs) 2025-08-14T21:57:23.1455310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1455378Z self_outputs = self.self( 2025-08-14T21:57:23.1455608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1455683Z return func(*args, **kwargs) 2025-08-14T21:57:23.1455934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1456003Z self.value(current_states) 2025-08-14T21:57:23.1456014Z 2025-08-14T21:57:23.1456094Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1456195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1456397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1456481Z return mod(**inputs) 2025-08-14T21:57:23.1456748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1456821Z outputs = self.roberta( 2025-08-14T21:57:23.1457084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1457173Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1457437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1457506Z layer_outputs = layer_module( 2025-08-14T21:57:23.1457730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1457806Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1458080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1458173Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1458429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1458506Z return func(*args, **kwargs) 2025-08-14T21:57:23.1458764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1458835Z self_outputs = self.self( 2025-08-14T21:57:23.1459077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1459145Z return func(*args, **kwargs) 2025-08-14T21:57:23.1459402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1459541Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1459546Z 2025-08-14T21:57:23.1459648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1459849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1459915Z return mod(**inputs) 2025-08-14T21:57:23.1460178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1460254Z outputs = self.roberta( 2025-08-14T21:57:23.1460511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1460594Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1460853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1460923Z layer_outputs = layer_module( 2025-08-14T21:57:23.1461147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1461226Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1461484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1461572Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1461810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1461885Z return func(*args, **kwargs) 2025-08-14T21:57:23.1462142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1462269Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1462541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1462644Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1462648Z 2025-08-14T21:57:23.1462753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1462944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1463009Z return mod(**inputs) 2025-08-14T21:57:23.1463289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1463356Z outputs = self.roberta( 2025-08-14T21:57:23.1463607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1463685Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1463946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1464025Z layer_outputs = layer_module( 2025-08-14T21:57:23.1464269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1464352Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1464645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1464736Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1465003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1465092Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1465402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1465535Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1465808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1465896Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1465900Z 2025-08-14T21:57:23.1466016Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1466239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1466315Z return mod(**inputs) 2025-08-14T21:57:23.1466578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1466645Z outputs = self.roberta( 2025-08-14T21:57:23.1466911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1466983Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1467245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1467326Z layer_outputs = layer_module( 2025-08-14T21:57:23.1467558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1467648Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1467925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1468012Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1468291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1468367Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1468685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1468812Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1469105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1469228Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1469451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1469544Z return self.act(input) 2025-08-14T21:57:23.1469556Z 2025-08-14T21:57:23.1469663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1469874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1469949Z return mod(**inputs) 2025-08-14T21:57:23.1470230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1470302Z outputs = self.roberta( 2025-08-14T21:57:23.1470601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1470681Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1470979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1471055Z layer_outputs = layer_module( 2025-08-14T21:57:23.1471287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1471377Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1471648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1471735Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1472015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1472097Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1472414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1472557Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1472833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1472933Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1472937Z 2025-08-14T21:57:23.1473049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1473272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1473343Z return mod(**inputs) 2025-08-14T21:57:23.1473628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1473712Z outputs = self.roberta( 2025-08-14T21:57:23.1473993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1474072Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1474363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1474441Z layer_outputs = layer_module( 2025-08-14T21:57:23.1474686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1474770Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1475052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1475153Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1475417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1475514Z return func(*args, **kwargs) 2025-08-14T21:57:23.1475877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1475965Z self_outputs = self.self( 2025-08-14T21:57:23.1476239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1476340Z return func(*args, **kwargs) 2025-08-14T21:57:23.1476621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1476855Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1476859Z 2025-08-14T21:57:23.1476973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1477199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1477299Z return mod(**inputs) 2025-08-14T21:57:23.1477561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1477656Z outputs = self.roberta( 2025-08-14T21:57:23.1477915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1477991Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1478252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1478324Z layer_outputs = layer_module( 2025-08-14T21:57:23.1478548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1478624Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1478884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1478973Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1479216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1479292Z return func(*args, **kwargs) 2025-08-14T21:57:23.1479548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1479617Z self_outputs = self.self( 2025-08-14T21:57:23.1479858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1479927Z return func(*args, **kwargs) 2025-08-14T21:57:23.1480181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1480260Z self.key(current_states) 2025-08-14T21:57:23.1480265Z 2025-08-14T21:57:23.1480368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1480570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1480638Z return mod(**inputs) 2025-08-14T21:57:23.1480896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1480973Z outputs = self.roberta( 2025-08-14T21:57:23.1481232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1481310Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1481564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1481634Z layer_outputs = layer_module( 2025-08-14T21:57:23.1481880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1481958Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1482220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1482308Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1482565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1482640Z return func(*args, **kwargs) 2025-08-14T21:57:23.1482899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1482967Z self_outputs = self.self( 2025-08-14T21:57:23.1483210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1483279Z return func(*args, **kwargs) 2025-08-14T21:57:23.1483553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1483637Z self.value(current_states) 2025-08-14T21:57:23.1483641Z 2025-08-14T21:57:23.1483742Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1483858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1484066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1484135Z return mod(**inputs) 2025-08-14T21:57:23.1484418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1484488Z outputs = self.roberta( 2025-08-14T21:57:23.1484759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1484852Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1485110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1485185Z layer_outputs = layer_module( 2025-08-14T21:57:23.1485401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1485477Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1485739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1485819Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1486060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1486128Z return func(*args, **kwargs) 2025-08-14T21:57:23.1486384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1486463Z self_outputs = self.self( 2025-08-14T21:57:23.1486697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1486765Z return func(*args, **kwargs) 2025-08-14T21:57:23.1487027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1487162Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1487166Z 2025-08-14T21:57:23.1487280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1487487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1487556Z return mod(**inputs) 2025-08-14T21:57:23.1487855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1487980Z outputs = self.roberta( 2025-08-14T21:57:23.1488259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1488333Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1488608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1488705Z layer_outputs = layer_module( 2025-08-14T21:57:23.1488926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1489003Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1489268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1489349Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1489607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1489678Z return func(*args, **kwargs) 2025-08-14T21:57:23.1489971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1490117Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1490389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1490485Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1490489Z 2025-08-14T21:57:23.1490595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1490800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1490878Z return mod(**inputs) 2025-08-14T21:57:23.1491164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1491237Z outputs = self.roberta( 2025-08-14T21:57:23.1491515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1491592Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1491869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1491944Z layer_outputs = layer_module( 2025-08-14T21:57:23.1492180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1492265Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1492519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1492602Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1492863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1492939Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1493235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1493362Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1493630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1493724Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1493728Z 2025-08-14T21:57:23.1493836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1494045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1494112Z return mod(**inputs) 2025-08-14T21:57:23.1494410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1494490Z outputs = self.roberta( 2025-08-14T21:57:23.1494762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1494836Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1495131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1495205Z layer_outputs = layer_module( 2025-08-14T21:57:23.1495443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1495524Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1495804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1495920Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1496174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1496271Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1496565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1496687Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1496953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1497065Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1497277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1497353Z return self.act(input) 2025-08-14T21:57:23.1497358Z 2025-08-14T21:57:23.1497461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1497666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1497730Z return mod(**inputs) 2025-08-14T21:57:23.1497992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1498070Z outputs = self.roberta( 2025-08-14T21:57:23.1498328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1498406Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1498662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1498732Z layer_outputs = layer_module( 2025-08-14T21:57:23.1498959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1499039Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1499297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1499389Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1499646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1499730Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1500017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1500149Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1500415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1500517Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1500521Z 2025-08-14T21:57:23.1500635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1500845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1500913Z return mod(**inputs) 2025-08-14T21:57:23.1501194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1501294Z outputs = self.roberta( 2025-08-14T21:57:23.1501563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1501644Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1501914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1501994Z layer_outputs = layer_module( 2025-08-14T21:57:23.1502244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1502330Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1502631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1502718Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1502979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1503051Z return func(*args, **kwargs) 2025-08-14T21:57:23.1503321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1503399Z self_outputs = self.self( 2025-08-14T21:57:23.1503636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1503708Z return func(*args, **kwargs) 2025-08-14T21:57:23.1503971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:57:23.1504178Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:57:23.1504181Z 2025-08-14T21:57:23.1504296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1504501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1504568Z return mod(**inputs) 2025-08-14T21:57:23.1504853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1504925Z outputs = self.roberta( 2025-08-14T21:57:23.1505202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1505282Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1505550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1505635Z layer_outputs = layer_module( 2025-08-14T21:57:23.1505865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1505948Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1506227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1506314Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1506573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1506646Z return func(*args, **kwargs) 2025-08-14T21:57:23.1506917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1507018Z self_outputs = self.self( 2025-08-14T21:57:23.1507278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1507352Z return func(*args, **kwargs) 2025-08-14T21:57:23.1507642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:57:23.1507732Z self.key(current_states) 2025-08-14T21:57:23.1507737Z 2025-08-14T21:57:23.1507853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1508057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1508124Z return mod(**inputs) 2025-08-14T21:57:23.1508405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1508493Z outputs = self.roberta( 2025-08-14T21:57:23.1508933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1509059Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1509336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1509424Z layer_outputs = layer_module( 2025-08-14T21:57:23.1509654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1509737Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1510019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1510107Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1510366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1510439Z return func(*args, **kwargs) 2025-08-14T21:57:23.1510714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1510796Z self_outputs = self.self( 2025-08-14T21:57:23.1511052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1511123Z return func(*args, **kwargs) 2025-08-14T21:57:23.1511407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:57:23.1511482Z self.value(current_states) 2025-08-14T21:57:23.1511486Z 2025-08-14T21:57:23.1511583Z cudagraph partition due to non gpu ops 2025-08-14T21:57:23.1511691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1511903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1511981Z return mod(**inputs) 2025-08-14T21:57:23.1512261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1512341Z outputs = self.roberta( 2025-08-14T21:57:23.1512614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1512691Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1512970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1513046Z layer_outputs = layer_module( 2025-08-14T21:57:23.1513283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1513376Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1513699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1513792Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1514046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1514146Z return func(*args, **kwargs) 2025-08-14T21:57:23.1514422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:57:23.1514494Z self_outputs = self.self( 2025-08-14T21:57:23.1514743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1514822Z return func(*args, **kwargs) 2025-08-14T21:57:23.1515094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:57:23.1515267Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:57:23.1515271Z 2025-08-14T21:57:23.1515378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1515599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1515679Z return mod(**inputs) 2025-08-14T21:57:23.1516220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1516308Z outputs = self.roberta( 2025-08-14T21:57:23.1516588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1516667Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1516953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1517033Z layer_outputs = layer_module( 2025-08-14T21:57:23.1517270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1517362Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1517650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:57:23.1517744Z self_attention_outputs = self.attention( 2025-08-14T21:57:23.1517997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:57:23.1518069Z return func(*args, **kwargs) 2025-08-14T21:57:23.1518349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:57:23.1518483Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:57:23.1518760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:57:23.1518849Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1518853Z 2025-08-14T21:57:23.1518959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1519183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1519249Z return mod(**inputs) 2025-08-14T21:57:23.1519501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1519575Z outputs = self.roberta( 2025-08-14T21:57:23.1519824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1519901Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1520154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1520252Z layer_outputs = layer_module( 2025-08-14T21:57:23.1520483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1520571Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1520828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1520941Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1521186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1521268Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1521546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1521661Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1521933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:57:23.1522013Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1522016Z 2025-08-14T21:57:23.1522138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1522334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1522402Z return mod(**inputs) 2025-08-14T21:57:23.1522670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1522738Z outputs = self.roberta( 2025-08-14T21:57:23.1522994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1523072Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1523330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1523409Z layer_outputs = layer_module( 2025-08-14T21:57:23.1523629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1523705Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1523969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1524050Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1524312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1524387Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1524677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:57:23.1524806Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:57:23.1525062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:57:23.1525175Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:57:23.1525393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:23.1525463Z return self.act(input) 2025-08-14T21:57:23.1525467Z 2025-08-14T21:57:23.1525575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1525772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1525836Z return mod(**inputs) 2025-08-14T21:57:23.1526111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:57:23.1526200Z outputs = self.roberta( 2025-08-14T21:57:23.1526455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:57:23.1526525Z encoder_outputs = self.encoder( 2025-08-14T21:57:23.1526776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:57:23.1526869Z layer_outputs = layer_module( 2025-08-14T21:57:23.1527082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:23.1527160Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:23.1527423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:57:23.1527506Z layer_output = apply_chunking_to_forward( 2025-08-14T21:57:23.1527765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:57:23.1527859Z return forward_fn(*input_tensors) 2025-08-14T21:57:23.1528164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:57:23.1528323Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:57:23.1528585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:57:23.1528673Z hidden_states = self.dense(hidden_states) 2025-08-14T21:57:23.1528676Z 2025-08-14T21:57:23.1528778Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1528970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1529043Z return mod(**inputs) 2025-08-14T21:57:23.1529303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1530, in forward 2025-08-14T21:57:23.1529389Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:57:23.1529400Z 2025-08-14T21:57:23.1529503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1529698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1529771Z return mod(**inputs) 2025-08-14T21:57:23.1530031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in forward 2025-08-14T21:57:23.1530135Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:57:23.1530139Z 2025-08-14T21:57:23.1530246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:23.1530436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:23.1530507Z return mod(**inputs) 2025-08-14T21:57:23.1530767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in forward 2025-08-14T21:57:23.1530861Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:57:23.1530865Z 2025-08-14T21:57:31.0277958Z Compilation time (from dynamo_timed): 14.298801969 2025-08-14T21:57:31.0278439Z pass 2025-08-14T21:57:31.0278849Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:31.0279876Z TIMING: _recursive_pre_grad_passes:0.00721 _recursive_joint_graph_passes:0.65081 _recursive_post_grad_passes:0.09104 async_compile.wait:0.0029 code_gen:6.80144 inductor_compile:8.00533 backend_compile:11.17536 gc:0.00135 entire_frame_compile:14.2988 total_wall_time:14.2988 2025-08-14T21:57:31.0280857Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:12465 | FakeTensor.__torch_dispatch__:4777 | ProxyTorchDispatchMode.__torch_dispatch__:4566 2025-08-14T21:57:31.0281385Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-08-14T21:57:36.3635701Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:57:36.3636899Z from pkg_resources import resource_filename 2025-08-14T21:57:36.9354527Z 2025-08-14T21:57:38.0832972Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:57:38.0833292Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:57:38.0849147Z cpu eval T5ForConditionalGeneration 2025-08-14T21:57:39.4354992Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:39.8474024Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:40.3177260Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:50.1969847Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.1970224Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.1970740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.1971087Z return mod(**inputs) 2025-08-14T21:57:50.1971478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.1971893Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.1972320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.1972711Z layer_outputs = layer_module( 2025-08-14T21:57:50.1973067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.1973436Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.1973827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.1974207Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.1974666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.1975075Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.1975524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward 2025-08-14T21:57:50.1975932Z position_bias = position_bias + causal_mask 2025-08-14T21:57:50.1976105Z 2025-08-14T21:57:50.1976223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.1976634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.1976991Z return mod(**inputs) 2025-08-14T21:57:50.1977386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.1977791Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.1978201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.1978605Z layer_outputs = layer_module( 2025-08-14T21:57:50.1978988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.1979392Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.1979803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.1980234Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.1980640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.1981081Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.1981556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.1981955Z return self.weight * hidden_states 2025-08-14T21:57:50.1982097Z 2025-08-14T21:57:50.1982217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.1982608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.1982991Z return mod(**inputs) 2025-08-14T21:57:50.1983364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.1983763Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.1984123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.1984492Z layer_outputs = layer_module( 2025-08-14T21:57:50.1984847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.1985258Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.1985658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.1986082Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.1986479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.1986876Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.1987301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.1987698Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.1987840Z 2025-08-14T21:57:50.1987960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.1988331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.1988677Z return mod(**inputs) 2025-08-14T21:57:50.1989048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.1989434Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.1989824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.1990222Z layer_outputs = layer_module( 2025-08-14T21:57:50.1990726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.1991113Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.1991527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.1991924Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.1992328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.1992737Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.1993155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.1993566Z key_states = self.k(current_states) 2025-08-14T21:57:50.1993709Z 2025-08-14T21:57:50.1993836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.1994218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.1994568Z return mod(**inputs) 2025-08-14T21:57:50.1994947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.1995347Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.1995934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.1996384Z layer_outputs = layer_module( 2025-08-14T21:57:50.1996789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.1997187Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.1997652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.1998086Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.1998476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.1998877Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.1999269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.1999720Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.1999909Z 2025-08-14T21:57:50.2000019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2000420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2000769Z return mod(**inputs) 2025-08-14T21:57:50.2001151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2001534Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2001915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2002317Z layer_outputs = layer_module( 2025-08-14T21:57:50.2002651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2003012Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2003376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2003751Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2004115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2004487Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2004857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2005303Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2005535Z 2025-08-14T21:57:50.2005643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2006026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2006364Z return mod(**inputs) 2025-08-14T21:57:50.2006719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2007116Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2007495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2007860Z layer_outputs = layer_module( 2025-08-14T21:57:50.2008197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2008557Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2009088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2009461Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2009829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2010204Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2010574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2010989Z value_states = self.v(current_states) 2025-08-14T21:57:50.2011134Z 2025-08-14T21:57:50.2011238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2011601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2011918Z return mod(**inputs) 2025-08-14T21:57:50.2012293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2012678Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2013060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2013440Z layer_outputs = layer_module( 2025-08-14T21:57:50.2013806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2014185Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2014592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2014958Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2015347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2015729Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2016110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2016535Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2016717Z 2025-08-14T21:57:50.2016846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2017227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2017577Z return mod(**inputs) 2025-08-14T21:57:50.2017958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2018414Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2018776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2019163Z layer_outputs = layer_module( 2025-08-14T21:57:50.2019527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2019908Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2020288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2020680Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2021069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2021462Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2021847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2022268Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2022437Z 2025-08-14T21:57:50.2022554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2022933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2023283Z return mod(**inputs) 2025-08-14T21:57:50.2023661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2024066Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2024450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2024852Z layer_outputs = layer_module( 2025-08-14T21:57:50.2025219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2025621Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2026020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2026421Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2026841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2027244Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2027630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2028054Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2028226Z 2025-08-14T21:57:50.2028340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2028747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2029088Z return mod(**inputs) 2025-08-14T21:57:50.2029450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2029848Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2030233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2030627Z layer_outputs = layer_module( 2025-08-14T21:57:50.2031003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2031397Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2031801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2032218Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2032628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2033046Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2033449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2033866Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2034014Z 2025-08-14T21:57:50.2034128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2034521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2034885Z return mod(**inputs) 2025-08-14T21:57:50.2035272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2035672Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2036159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2036583Z layer_outputs = layer_module( 2025-08-14T21:57:50.2036961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2037364Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2037768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2038191Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2038650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2039068Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2039482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2039894Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2040046Z 2025-08-14T21:57:50.2040185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2040579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2040989Z return mod(**inputs) 2025-08-14T21:57:50.2041366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2041798Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2042199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2042594Z layer_outputs = layer_module( 2025-08-14T21:57:50.2042960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2043349Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2043748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2044242Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2044651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2045170Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2045584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2045990Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2046145Z 2025-08-14T21:57:50.2046259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2046664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2047017Z return mod(**inputs) 2025-08-14T21:57:50.2047394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2047797Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2048195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2048588Z layer_outputs = layer_module( 2025-08-14T21:57:50.2048968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2049360Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2049764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2050164Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2050569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2050975Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2051367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2051782Z key_states = self.k(current_states) 2025-08-14T21:57:50.2051948Z 2025-08-14T21:57:50.2052061Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2052444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2052780Z return mod(**inputs) 2025-08-14T21:57:50.2053143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2053541Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2053926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2054325Z layer_outputs = layer_module( 2025-08-14T21:57:50.2054705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2055103Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2055528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2055941Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2056348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2056770Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2057169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2057608Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2057797Z 2025-08-14T21:57:50.2057911Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2058279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2058619Z return mod(**inputs) 2025-08-14T21:57:50.2058996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2059389Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2059784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2060174Z layer_outputs = layer_module( 2025-08-14T21:57:50.2060541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2060916Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2061312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2061710Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2062104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2062499Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2062899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2063381Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2063602Z 2025-08-14T21:57:50.2063717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2064093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2064437Z return mod(**inputs) 2025-08-14T21:57:50.2064814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2065221Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2065664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2066074Z layer_outputs = layer_module( 2025-08-14T21:57:50.2066455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2066844Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2067235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2067641Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2068029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2068439Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2068839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2069320Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2069543Z 2025-08-14T21:57:50.2069651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2070053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2070394Z return mod(**inputs) 2025-08-14T21:57:50.2070758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2071140Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2071542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2071940Z layer_outputs = layer_module( 2025-08-14T21:57:50.2072293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2072683Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2073076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2073487Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2073901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2074312Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2074770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2075250Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2075478Z 2025-08-14T21:57:50.2075589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2076070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2076424Z return mod(**inputs) 2025-08-14T21:57:50.2076787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2077189Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2077592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2077990Z layer_outputs = layer_module( 2025-08-14T21:57:50.2078358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2078749Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2079157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2079561Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2079962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2080369Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2080773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2081173Z value_states = self.v(current_states) 2025-08-14T21:57:50.2081331Z 2025-08-14T21:57:50.2081444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2081835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2082184Z return mod(**inputs) 2025-08-14T21:57:50.2082554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2082958Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2083364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2083766Z layer_outputs = layer_module( 2025-08-14T21:57:50.2084153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2084558Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2084963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2085409Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2085825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2086247Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2086690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2087213Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2087383Z 2025-08-14T21:57:50.2087498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2087876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2088207Z return mod(**inputs) 2025-08-14T21:57:50.2088572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2089874Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2090272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2090688Z layer_outputs = layer_module( 2025-08-14T21:57:50.2091060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2091447Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2091829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2092227Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2092618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2093006Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2093402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2093826Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2093999Z 2025-08-14T21:57:50.2094114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2094489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2094831Z return mod(**inputs) 2025-08-14T21:57:50.2095190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2095637Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2096019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2096410Z layer_outputs = layer_module( 2025-08-14T21:57:50.2096772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2097153Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2097546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2097954Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2098369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2098760Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2099154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2099578Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2099749Z 2025-08-14T21:57:50.2099858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2100245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2100619Z return mod(**inputs) 2025-08-14T21:57:50.2100985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2101368Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2101759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2102172Z layer_outputs = layer_module( 2025-08-14T21:57:50.2102551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2102938Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2103339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2103747Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2104142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2104569Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2104960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2105388Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2105526Z 2025-08-14T21:57:50.2105614Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2105865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2106242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2106575Z return mod(**inputs) 2025-08-14T21:57:50.2106941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2107334Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2107716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2108108Z layer_outputs = layer_module( 2025-08-14T21:57:50.2108470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2109069Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2109467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2109894Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2110307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2110732Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2111144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2111551Z return self.weight * hidden_states 2025-08-14T21:57:50.2111704Z 2025-08-14T21:57:50.2111830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2112232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2112588Z return mod(**inputs) 2025-08-14T21:57:50.2112971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2113384Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2113778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2114186Z layer_outputs = layer_module( 2025-08-14T21:57:50.2114567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2114992Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2115428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2116005Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2116460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2116944Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2117438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2117914Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2118066Z 2025-08-14T21:57:50.2118189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2118582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2118949Z return mod(**inputs) 2025-08-14T21:57:50.2119320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2119709Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2120115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2120566Z layer_outputs = layer_module( 2025-08-14T21:57:50.2120939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2121328Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2121730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2122148Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2122562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2123001Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2123442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2123851Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2123996Z 2025-08-14T21:57:50.2124104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2124487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2124828Z return mod(**inputs) 2025-08-14T21:57:50.2125192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2125579Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2125968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2126357Z layer_outputs = layer_module( 2025-08-14T21:57:50.2126718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2127103Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2127475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2127883Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2128279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2128718Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2129152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2129554Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2129695Z 2025-08-14T21:57:50.2129780Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2130034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2130411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2130770Z return mod(**inputs) 2025-08-14T21:57:50.2131114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2131483Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2131844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2132226Z layer_outputs = layer_module( 2025-08-14T21:57:50.2132574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2132940Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2133311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2133695Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2134100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2134510Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2134940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2135340Z return self.weight * hidden_states 2025-08-14T21:57:50.2135481Z 2025-08-14T21:57:50.2135599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2135979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2136315Z return mod(**inputs) 2025-08-14T21:57:50.2136681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2137074Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2137459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2137857Z layer_outputs = layer_module( 2025-08-14T21:57:50.2138225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2138614Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2139005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2139408Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2139805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2140199Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2140598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2140995Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2141135Z 2025-08-14T21:57:50.2141254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2141629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2141974Z return mod(**inputs) 2025-08-14T21:57:50.2142341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2142731Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2143122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2143517Z layer_outputs = layer_module( 2025-08-14T21:57:50.2143884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2144262Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2144656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2145088Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2145481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2145875Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2146268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2146669Z key_states = self.k(current_states) 2025-08-14T21:57:50.2146808Z 2025-08-14T21:57:50.2146915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2147291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2147629Z return mod(**inputs) 2025-08-14T21:57:50.2147985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2148367Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2148772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2149160Z layer_outputs = layer_module( 2025-08-14T21:57:50.2149535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2149915Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2150302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2150699Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2151079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2151476Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2151871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2152321Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2152518Z 2025-08-14T21:57:50.2152626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2153002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2153342Z return mod(**inputs) 2025-08-14T21:57:50.2153701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2154089Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2154472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2154861Z layer_outputs = layer_module( 2025-08-14T21:57:50.2155217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2155599Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2156090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2156513Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2156940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2157351Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2157750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2158195Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2158416Z 2025-08-14T21:57:50.2158522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2158885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2159213Z return mod(**inputs) 2025-08-14T21:57:50.2159583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2159951Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2160313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2160672Z layer_outputs = layer_module( 2025-08-14T21:57:50.2161033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2161394Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2161767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2162133Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2162502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2162881Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2163264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2163734Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2163951Z 2025-08-14T21:57:50.2164054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2164413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2164726Z return mod(**inputs) 2025-08-14T21:57:50.2165073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2165503Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2165847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2166257Z layer_outputs = layer_module( 2025-08-14T21:57:50.2166599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2166955Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2167315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2167690Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2168054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2168425Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2168783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2169221Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2169428Z 2025-08-14T21:57:50.2169544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2169916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2170254Z return mod(**inputs) 2025-08-14T21:57:50.2170595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2170962Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2171326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2171685Z layer_outputs = layer_module( 2025-08-14T21:57:50.2172022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2172363Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2172724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2173119Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2173482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2173842Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2174210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2174587Z value_states = self.v(current_states) 2025-08-14T21:57:50.2174720Z 2025-08-14T21:57:50.2174828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2175166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2175478Z return mod(**inputs) 2025-08-14T21:57:50.2175811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2176172Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2176546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2176902Z layer_outputs = layer_module( 2025-08-14T21:57:50.2177251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2177600Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2177971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2178343Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2178708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2179084Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2179452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2179855Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2180021Z 2025-08-14T21:57:50.2180125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2180487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2180814Z return mod(**inputs) 2025-08-14T21:57:50.2181160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2181521Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2181886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2182250Z layer_outputs = layer_module( 2025-08-14T21:57:50.2182593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2182945Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2183317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2183691Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2184054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2184427Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2184800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2185194Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2185362Z 2025-08-14T21:57:50.2185465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2185820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2186149Z return mod(**inputs) 2025-08-14T21:57:50.2186503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2186921Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2187308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2187712Z layer_outputs = layer_module( 2025-08-14T21:57:50.2188076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2188461Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2188830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2189201Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2189572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2189949Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2190344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2190739Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2190908Z 2025-08-14T21:57:50.2191043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2191424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2191771Z return mod(**inputs) 2025-08-14T21:57:50.2192122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2192495Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2192865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2193258Z layer_outputs = layer_module( 2025-08-14T21:57:50.2193630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2194029Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2194416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2194826Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2195219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2195624Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2196088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2196499Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2196637Z 2025-08-14T21:57:50.2196757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2197129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2197452Z return mod(**inputs) 2025-08-14T21:57:50.2197800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2198182Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2198566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2198956Z layer_outputs = layer_module( 2025-08-14T21:57:50.2199307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2199692Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2200079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2200491Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2200894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2201322Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2201726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2202118Z return self.weight * hidden_states 2025-08-14T21:57:50.2202259Z 2025-08-14T21:57:50.2202404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2202769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2203107Z return mod(**inputs) 2025-08-14T21:57:50.2203465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2203848Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2204222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2204610Z layer_outputs = layer_module( 2025-08-14T21:57:50.2204987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2205360Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2205768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2206183Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2206587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2207015Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2207451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2207848Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2207990Z 2025-08-14T21:57:50.2208100Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2208484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2209009Z return mod(**inputs) 2025-08-14T21:57:50.2209386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2209770Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2210159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2210550Z layer_outputs = layer_module( 2025-08-14T21:57:50.2210911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2211294Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2211688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2212103Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2212501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2212939Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2213357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2213725Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2213858Z 2025-08-14T21:57:50.2213960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2214308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2214625Z return mod(**inputs) 2025-08-14T21:57:50.2214961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2215323Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2215745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2216112Z layer_outputs = layer_module( 2025-08-14T21:57:50.2216450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2216846Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2217216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2217595Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2217977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2218383Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2218806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2219227Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2219369Z 2025-08-14T21:57:50.2219448Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2219679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2220051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2220359Z return mod(**inputs) 2025-08-14T21:57:50.2220694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2221052Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2221404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2221770Z layer_outputs = layer_module( 2025-08-14T21:57:50.2222113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2222474Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2222835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2223220Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2223599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2224001Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2224405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2224770Z return self.weight * hidden_states 2025-08-14T21:57:50.2224902Z 2025-08-14T21:57:50.2225013Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2225359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2225693Z return mod(**inputs) 2025-08-14T21:57:50.2226055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2226419Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2226774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2227139Z layer_outputs = layer_module( 2025-08-14T21:57:50.2227485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2227835Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2228204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2228579Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2228956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2229354Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2229719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2230082Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2230211Z 2025-08-14T21:57:50.2230310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2230672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2230985Z return mod(**inputs) 2025-08-14T21:57:50.2231317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2231667Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2232019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2232374Z layer_outputs = layer_module( 2025-08-14T21:57:50.2232719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2233079Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2233474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2233846Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2234207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2234611Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2235003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2235409Z key_states = self.k(current_states) 2025-08-14T21:57:50.2235538Z 2025-08-14T21:57:50.2235641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2236084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2236424Z return mod(**inputs) 2025-08-14T21:57:50.2236784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2237193Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2237596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2237994Z layer_outputs = layer_module( 2025-08-14T21:57:50.2238355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2238741Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2239130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2239499Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2239880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2240256Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2240630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2241050Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2241239Z 2025-08-14T21:57:50.2241344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2241701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2242025Z return mod(**inputs) 2025-08-14T21:57:50.2242363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2242731Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2243094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2243475Z layer_outputs = layer_module( 2025-08-14T21:57:50.2243826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2244194Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2244582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2244947Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2245670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2246120Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2246606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2247181Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2247417Z 2025-08-14T21:57:50.2247544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2247984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2248408Z return mod(**inputs) 2025-08-14T21:57:50.2248843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2249274Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2249732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2250169Z layer_outputs = layer_module( 2025-08-14T21:57:50.2250593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2251005Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2251479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2251933Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2252355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2252802Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2253265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2253792Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2254020Z 2025-08-14T21:57:50.2254145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2254590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2254989Z return mod(**inputs) 2025-08-14T21:57:50.2255405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2271822Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2272516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2272942Z layer_outputs = layer_module( 2025-08-14T21:57:50.2273331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2273735Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2274148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2274558Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2274953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2275361Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2275948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2276430Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2276668Z 2025-08-14T21:57:50.2276791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2277237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2277591Z return mod(**inputs) 2025-08-14T21:57:50.2277957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2278356Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2278747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2279128Z layer_outputs = layer_module( 2025-08-14T21:57:50.2279533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2279926Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2280349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2280751Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2281151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2282139Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2282542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2282927Z value_states = self.v(current_states) 2025-08-14T21:57:50.2283082Z 2025-08-14T21:57:50.2283198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2283588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2283933Z return mod(**inputs) 2025-08-14T21:57:50.2284301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2284703Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2285093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2285483Z layer_outputs = layer_module( 2025-08-14T21:57:50.2285860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2286225Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2286589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2286968Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2287368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2287765Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2288155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2288583Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2288757Z 2025-08-14T21:57:50.2288877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2289263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2289600Z return mod(**inputs) 2025-08-14T21:57:50.2289968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2290362Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2290740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2291152Z layer_outputs = layer_module( 2025-08-14T21:57:50.2291519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2291903Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2292285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2292734Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2293123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2293520Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2293917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2294346Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2294518Z 2025-08-14T21:57:50.2294657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2295034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2295379Z return mod(**inputs) 2025-08-14T21:57:50.2295763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2296160Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2296539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2296935Z layer_outputs = layer_module( 2025-08-14T21:57:50.2297301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2297676Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2298070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2298473Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2298868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2299270Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2299664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2300090Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2300278Z 2025-08-14T21:57:50.2300389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2300771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2301113Z return mod(**inputs) 2025-08-14T21:57:50.2301483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2301851Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2302248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2302649Z layer_outputs = layer_module( 2025-08-14T21:57:50.2303019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2303415Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2303869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2304249Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2304620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2305005Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2305383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2305824Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2305963Z 2025-08-14T21:57:50.2306050Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2306306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2306681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2307015Z return mod(**inputs) 2025-08-14T21:57:50.2307368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2307772Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2308177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2308577Z layer_outputs = layer_module( 2025-08-14T21:57:50.2309121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2309595Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2309964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2310387Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2310777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2311181Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2311590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2311987Z return self.weight * hidden_states 2025-08-14T21:57:50.2312132Z 2025-08-14T21:57:50.2312280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2312661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2313002Z return mod(**inputs) 2025-08-14T21:57:50.2313368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2313772Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2314163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2314568Z layer_outputs = layer_module( 2025-08-14T21:57:50.2314957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2315349Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2315731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2316249Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2316662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2317118Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2317553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2317926Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2318073Z 2025-08-14T21:57:50.2318180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2318547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2318891Z return mod(**inputs) 2025-08-14T21:57:50.2319268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2319677Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2320069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2320502Z layer_outputs = layer_module( 2025-08-14T21:57:50.2320878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2321270Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2321664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2322133Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2322542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2322984Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2323417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2323827Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2323973Z 2025-08-14T21:57:50.2324119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2324474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2324787Z return mod(**inputs) 2025-08-14T21:57:50.2325148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2325520Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2325884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2326259Z layer_outputs = layer_module( 2025-08-14T21:57:50.2326611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2326981Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2327351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2327746Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2328133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2328542Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2328955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2329342Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2329484Z 2025-08-14T21:57:50.2329578Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2329820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2330188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2330519Z return mod(**inputs) 2025-08-14T21:57:50.2330867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2331248Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2331619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2331994Z layer_outputs = layer_module( 2025-08-14T21:57:50.2332341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2332714Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2333090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2333476Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2333852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2334263Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2334680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2335042Z return self.weight * hidden_states 2025-08-14T21:57:50.2335183Z 2025-08-14T21:57:50.2335286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2335643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2335988Z return mod(**inputs) 2025-08-14T21:57:50.2336325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2336696Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2337061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2337423Z layer_outputs = layer_module( 2025-08-14T21:57:50.2337775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2338183Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2338570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2338954Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2339327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2339711Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2340091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2340462Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2340605Z 2025-08-14T21:57:50.2340711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2341075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2341398Z return mod(**inputs) 2025-08-14T21:57:50.2341753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2342133Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2342514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2342886Z layer_outputs = layer_module( 2025-08-14T21:57:50.2343238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2343604Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2343985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2344360Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2344736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2345126Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2345527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2345930Z key_states = self.k(current_states) 2025-08-14T21:57:50.2346072Z 2025-08-14T21:57:50.2346178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2346544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2346864Z return mod(**inputs) 2025-08-14T21:57:50.2347214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2347592Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2347957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2348332Z layer_outputs = layer_module( 2025-08-14T21:57:50.2348698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2349058Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2349422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2349810Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2350181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2350554Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2350940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2351388Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2351578Z 2025-08-14T21:57:50.2351694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2352079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2352445Z return mod(**inputs) 2025-08-14T21:57:50.2352824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2353218Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2353597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2353986Z layer_outputs = layer_module( 2025-08-14T21:57:50.2354356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2354753Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2355157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2355573Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2356062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2356475Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2356884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2357380Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2357614Z 2025-08-14T21:57:50.2357722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2358064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2358383Z return mod(**inputs) 2025-08-14T21:57:50.2358721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2359075Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2359433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2359791Z layer_outputs = layer_module( 2025-08-14T21:57:50.2360125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2360474Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2360838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2361202Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2361554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2361920Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2362283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2362739Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2362942Z 2025-08-14T21:57:50.2363043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2363391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2363705Z return mod(**inputs) 2025-08-14T21:57:50.2364060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2364410Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2364771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2365142Z layer_outputs = layer_module( 2025-08-14T21:57:50.2365470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2365820Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2366195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2366570Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2366946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2367325Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2367696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2368129Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2368340Z 2025-08-14T21:57:50.2368443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2368814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2369128Z return mod(**inputs) 2025-08-14T21:57:50.2369467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2369834Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2370198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2370568Z layer_outputs = layer_module( 2025-08-14T21:57:50.2370910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2371269Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2371638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2372003Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2372376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2372753Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2373130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2373494Z value_states = self.v(current_states) 2025-08-14T21:57:50.2373637Z 2025-08-14T21:57:50.2373741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2374107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2374420Z return mod(**inputs) 2025-08-14T21:57:50.2374764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2375132Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2375496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2375857Z layer_outputs = layer_module( 2025-08-14T21:57:50.2376246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2376612Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2376981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2377366Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2377751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2378122Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2378484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2378607Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2378612Z 2025-08-14T21:57:50.2378720Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2378952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2379026Z return mod(**inputs) 2025-08-14T21:57:50.2379280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2379386Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2379632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2379711Z layer_outputs = layer_module( 2025-08-14T21:57:50.2379948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2380032Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2380280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2380374Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2380608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2380698Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2380926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2381033Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2381044Z 2025-08-14T21:57:50.2381148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2381344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2381416Z return mod(**inputs) 2025-08-14T21:57:50.2381647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2381718Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2381960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2382033Z layer_outputs = layer_module( 2025-08-14T21:57:50.2382258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2382337Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2382566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2382654Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2382882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2382962Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2383197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2383304Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2383326Z 2025-08-14T21:57:50.2383438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2383633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2383700Z return mod(**inputs) 2025-08-14T21:57:50.2383938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2384034Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2384272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2384344Z layer_outputs = layer_module( 2025-08-14T21:57:50.2384559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2384642Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2384887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2384971Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2385207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2385304Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2385539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2385619Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2385623Z 2025-08-14T21:57:50.2385725Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2385928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2385993Z return mod(**inputs) 2025-08-14T21:57:50.2386223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2386304Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2386536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2386615Z layer_outputs = layer_module( 2025-08-14T21:57:50.2386844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2386922Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2387153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2387232Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2387462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-08-14T21:57:50.2387590Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:57:50.2387594Z 2025-08-14T21:57:50.2387682Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2387800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2388008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2388079Z return mod(**inputs) 2025-08-14T21:57:50.2388329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2388407Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2388658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2388731Z layer_outputs = layer_module( 2025-08-14T21:57:50.2388959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2389049Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2389292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2389410Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2389658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2389761Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2390031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2390113Z return self.weight * hidden_states 2025-08-14T21:57:50.2390117Z 2025-08-14T21:57:50.2390223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2390441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2390509Z return mod(**inputs) 2025-08-14T21:57:50.2390763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2390855Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2391102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2391186Z layer_outputs = layer_module( 2025-08-14T21:57:50.2391434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2391520Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2391775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2391872Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2392123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2392249Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2392492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2392586Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2392590Z 2025-08-14T21:57:50.2392697Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2392914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2392984Z return mod(**inputs) 2025-08-14T21:57:50.2393231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2393313Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2393559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2393634Z layer_outputs = layer_module( 2025-08-14T21:57:50.2393870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2393955Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2394206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2394304Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2394546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2394677Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2394922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2395009Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2395019Z 2025-08-14T21:57:50.2395128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2395336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2395435Z return mod(**inputs) 2025-08-14T21:57:50.2395686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2395826Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2396095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2396221Z layer_outputs = layer_module( 2025-08-14T21:57:50.2396467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2396553Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2396807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2396910Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2397162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2397306Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2397564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2397691Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2397695Z 2025-08-14T21:57:50.2397790Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2397897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2398108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2398184Z return mod(**inputs) 2025-08-14T21:57:50.2398428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2398503Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2398760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2398840Z layer_outputs = layer_module( 2025-08-14T21:57:50.2399077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2399161Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2399403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2399499Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2399743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2399854Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2400081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2400161Z return self.weight * hidden_states 2025-08-14T21:57:50.2400166Z 2025-08-14T21:57:50.2400281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2400490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2400559Z return mod(**inputs) 2025-08-14T21:57:50.2400813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2400889Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2401139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2401213Z layer_outputs = layer_module( 2025-08-14T21:57:50.2401441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2401531Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2401775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2401881Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2402129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2402216Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2402465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2402566Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2402570Z 2025-08-14T21:57:50.2402677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2402891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2402959Z return mod(**inputs) 2025-08-14T21:57:50.2403224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2403301Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2403571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2403656Z layer_outputs = layer_module( 2025-08-14T21:57:50.2403921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2404007Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2404275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2404364Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2404628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2404718Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2404974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2405070Z key_states = self.k(current_states) 2025-08-14T21:57:50.2405075Z 2025-08-14T21:57:50.2405187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2405409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2405482Z return mod(**inputs) 2025-08-14T21:57:50.2405734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2405819Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2406082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2406161Z layer_outputs = layer_module( 2025-08-14T21:57:50.2406401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2406487Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2406745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2406833Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2407100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2407200Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2407457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2407598Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2407609Z 2025-08-14T21:57:50.2407720Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2407931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2408010Z return mod(**inputs) 2025-08-14T21:57:50.2408293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2408368Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2408625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2408825Z layer_outputs = layer_module( 2025-08-14T21:57:50.2409133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2409216Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2409466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2409559Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2409809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2409898Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2410180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2410370Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2410375Z 2025-08-14T21:57:50.2410491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2410697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2410766Z return mod(**inputs) 2025-08-14T21:57:50.2411020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2411094Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2411347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2411430Z layer_outputs = layer_module( 2025-08-14T21:57:50.2411662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2411751Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2412001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2412085Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2412333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2412419Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2412667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2412826Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2412829Z 2025-08-14T21:57:50.2412938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2413154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2413223Z return mod(**inputs) 2025-08-14T21:57:50.2413467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2413550Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2413796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2413872Z layer_outputs = layer_module( 2025-08-14T21:57:50.2414078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2414153Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2414381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2414484Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2414721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2414804Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2415041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2415207Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2415210Z 2025-08-14T21:57:50.2415307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2415495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2415566Z return mod(**inputs) 2025-08-14T21:57:50.2415793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2415869Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2416114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2416184Z layer_outputs = layer_module( 2025-08-14T21:57:50.2416421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2416499Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2416723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2416807Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2417026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2417113Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2417333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2417412Z value_states = self.v(current_states) 2025-08-14T21:57:50.2417416Z 2025-08-14T21:57:50.2417523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2417718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2417789Z return mod(**inputs) 2025-08-14T21:57:50.2418019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2418103Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2418332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2418402Z layer_outputs = layer_module( 2025-08-14T21:57:50.2418611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2418695Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2418926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2419013Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2419241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2419322Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2419560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2419671Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2419675Z 2025-08-14T21:57:50.2419782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2419976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2420042Z return mod(**inputs) 2025-08-14T21:57:50.2420279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2420377Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2420624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2420699Z layer_outputs = layer_module( 2025-08-14T21:57:50.2420911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2421011Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2421236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2421314Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2421545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2421625Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2421875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2421992Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2421996Z 2025-08-14T21:57:50.2422108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2422307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2422374Z return mod(**inputs) 2025-08-14T21:57:50.2422607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2422685Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2422914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2422992Z layer_outputs = layer_module( 2025-08-14T21:57:50.2423211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2423290Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2423529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2423610Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2423837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2423929Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2424157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2424281Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2424284Z 2025-08-14T21:57:50.2424383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2424572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2424647Z return mod(**inputs) 2025-08-14T21:57:50.2424870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2424941Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2425171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2425243Z layer_outputs = layer_module( 2025-08-14T21:57:50.2425458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2425534Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2425758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2425844Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2426072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2426186Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2426415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2426491Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2426514Z 2025-08-14T21:57:50.2426603Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2426704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2426898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2426971Z return mod(**inputs) 2025-08-14T21:57:50.2427201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2427287Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2427526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2427597Z layer_outputs = layer_module( 2025-08-14T21:57:50.2427820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2427911Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2428140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2428239Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2428466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2428569Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2428801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2428877Z return self.weight * hidden_states 2025-08-14T21:57:50.2428882Z 2025-08-14T21:57:50.2428991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2429189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2429261Z return mod(**inputs) 2025-08-14T21:57:50.2429495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2429568Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2429801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2429872Z layer_outputs = layer_module( 2025-08-14T21:57:50.2430087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2430172Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2430401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2430499Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2430730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2430852Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2431103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2431186Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2431189Z 2025-08-14T21:57:50.2431301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2431507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2431576Z return mod(**inputs) 2025-08-14T21:57:50.2431825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2431921Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2432170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2432252Z layer_outputs = layer_module( 2025-08-14T21:57:50.2432469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2432575Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2432804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2432892Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2433128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2433245Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2433489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2433579Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2433583Z 2025-08-14T21:57:50.2433698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2433902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2433969Z return mod(**inputs) 2025-08-14T21:57:50.2434199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2434277Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2434507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2434585Z layer_outputs = layer_module( 2025-08-14T21:57:50.2434799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2434880Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2435119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2435215Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2435456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2435586Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2435904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2436003Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2436007Z 2025-08-14T21:57:50.2436092Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2436200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2436425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2436498Z return mod(**inputs) 2025-08-14T21:57:50.2436752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2436844Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2437109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2437193Z layer_outputs = layer_module( 2025-08-14T21:57:50.2437426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2437510Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2437765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2437851Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2438164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2438276Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2438524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2438612Z return self.weight * hidden_states 2025-08-14T21:57:50.2438633Z 2025-08-14T21:57:50.2438741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2438949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2439025Z return mod(**inputs) 2025-08-14T21:57:50.2439267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2439349Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2439609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2439687Z layer_outputs = layer_module( 2025-08-14T21:57:50.2439923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2440022Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2440267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2440372Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2440603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2440694Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2440924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2441002Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2441008Z 2025-08-14T21:57:50.2441117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2441314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2441386Z return mod(**inputs) 2025-08-14T21:57:50.2441620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2441693Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2441931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2442000Z layer_outputs = layer_module( 2025-08-14T21:57:50.2442220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2442310Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2442553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2442647Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2442888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2442976Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2443222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2443305Z key_states = self.k(current_states) 2025-08-14T21:57:50.2443310Z 2025-08-14T21:57:50.2443422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2443628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2443695Z return mod(**inputs) 2025-08-14T21:57:50.2443947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2444042Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2444285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2444368Z layer_outputs = layer_module( 2025-08-14T21:57:50.2444596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2444714Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2444972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2445059Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2445314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2445402Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2445658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2445824Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2445829Z 2025-08-14T21:57:50.2445941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2446174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2446259Z return mod(**inputs) 2025-08-14T21:57:50.2446514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2446597Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2446843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2446925Z layer_outputs = layer_module( 2025-08-14T21:57:50.2447166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2447250Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2447511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2447597Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2447848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2447944Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2448204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2448375Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2448379Z 2025-08-14T21:57:50.2448488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2448698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2448779Z return mod(**inputs) 2025-08-14T21:57:50.2449037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2449113Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2449376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2449453Z layer_outputs = layer_module( 2025-08-14T21:57:50.2449695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2449779Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2450033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2450125Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2450375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2453376Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2453649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2453813Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2453820Z 2025-08-14T21:57:50.2453941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2454176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2454245Z return mod(**inputs) 2025-08-14T21:57:50.2454501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2454578Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2454846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2454921Z layer_outputs = layer_module( 2025-08-14T21:57:50.2455205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2455298Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2455557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2455644Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2455891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2455977Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2456221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2456386Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2456390Z 2025-08-14T21:57:50.2456498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2456714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2456782Z return mod(**inputs) 2025-08-14T21:57:50.2457027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2457113Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2457359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2457441Z layer_outputs = layer_module( 2025-08-14T21:57:50.2457668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2457752Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2458000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2458084Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2458326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2458418Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2458661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2458754Z value_states = self.v(current_states) 2025-08-14T21:57:50.2458758Z 2025-08-14T21:57:50.2458863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2459069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2459145Z return mod(**inputs) 2025-08-14T21:57:50.2459388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2459463Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2459715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2459866Z layer_outputs = layer_module( 2025-08-14T21:57:50.2460104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2460185Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2460451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2460541Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2460782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2460877Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2461115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2461247Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2461253Z 2025-08-14T21:57:50.2461367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2461592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2461663Z return mod(**inputs) 2025-08-14T21:57:50.2461912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2461985Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2462243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2462317Z layer_outputs = layer_module( 2025-08-14T21:57:50.2462549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2462638Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2462883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2462968Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2463222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2463308Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2463563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2463674Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2463679Z 2025-08-14T21:57:50.2463785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2464003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2464070Z return mod(**inputs) 2025-08-14T21:57:50.2464329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2464404Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2464655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2464740Z layer_outputs = layer_module( 2025-08-14T21:57:50.2464974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2465059Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2465315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2465393Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2465631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2465713Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2465944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2466093Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2466097Z 2025-08-14T21:57:50.2466199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2466402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2466485Z return mod(**inputs) 2025-08-14T21:57:50.2466719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2466797Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2467028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2467099Z layer_outputs = layer_module( 2025-08-14T21:57:50.2467337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2467420Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2467683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2467765Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2467997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2468087Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2468314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2468391Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2468401Z 2025-08-14T21:57:50.2468501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2468707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2468781Z return mod(**inputs) 2025-08-14T21:57:50.2469012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2469083Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2469322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2469393Z layer_outputs = layer_module( 2025-08-14T21:57:50.2469615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2469692Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2469924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2470008Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2470235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-08-14T21:57:50.2470367Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:57:50.2470370Z 2025-08-14T21:57:50.2470458Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2470560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2470760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2470826Z return mod(**inputs) 2025-08-14T21:57:50.2471055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2471132Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2471367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2471438Z layer_outputs = layer_module( 2025-08-14T21:57:50.2471661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2471765Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2472002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2472095Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2472341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2472445Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2472673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2472756Z return self.weight * hidden_states 2025-08-14T21:57:50.2472760Z 2025-08-14T21:57:50.2472859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2473055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2473147Z return mod(**inputs) 2025-08-14T21:57:50.2473398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2473493Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2473745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2473820Z layer_outputs = layer_module( 2025-08-14T21:57:50.2474055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2474136Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2474376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2474477Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2474720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2474843Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2475094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2475179Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2475185Z 2025-08-14T21:57:50.2475299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2475511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2475579Z return mod(**inputs) 2025-08-14T21:57:50.2475932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2476015Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2476275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2476357Z layer_outputs = layer_module( 2025-08-14T21:57:50.2476592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2476686Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2476939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2477038Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2477303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2477422Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2477667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2477751Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2477755Z 2025-08-14T21:57:50.2477895Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2478112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2478180Z return mod(**inputs) 2025-08-14T21:57:50.2478437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2478531Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2478778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2478859Z layer_outputs = layer_module( 2025-08-14T21:57:50.2479091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2479169Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2479405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2479512Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2479752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2479885Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2480119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2480203Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2480206Z 2025-08-14T21:57:50.2480286Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2480388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2480594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2480659Z return mod(**inputs) 2025-08-14T21:57:50.2480900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:57:50.2480976Z encoder_outputs = self.encoder( 2025-08-14T21:57:50.2481210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1128, in forward 2025-08-14T21:57:50.2481325Z hidden_states = self.final_layer_norm(hidden_states) 2025-08-14T21:57:50.2481560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2481644Z return self.weight * hidden_states 2025-08-14T21:57:50.2481647Z 2025-08-14T21:57:50.2481751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2481944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2482016Z return mod(**inputs) 2025-08-14T21:57:50.2482252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2482328Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2482568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2482640Z layer_outputs = layer_module( 2025-08-14T21:57:50.2482867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2482947Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2483182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2483271Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2483512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2483601Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2483856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2483958Z key_states = self.k(current_states) 2025-08-14T21:57:50.2483962Z 2025-08-14T21:57:50.2484077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2484284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2484371Z return mod(**inputs) 2025-08-14T21:57:50.2484622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2484696Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2484950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2485025Z layer_outputs = layer_module( 2025-08-14T21:57:50.2485255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2485359Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2485606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2485691Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2485959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2486050Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2486304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2486441Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2486445Z 2025-08-14T21:57:50.2486551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2486766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2486833Z return mod(**inputs) 2025-08-14T21:57:50.2487088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2487165Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2487422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2487506Z layer_outputs = layer_module( 2025-08-14T21:57:50.2487735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2487816Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2488073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2488155Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2488461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2488555Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2488796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2488964Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2488968Z 2025-08-14T21:57:50.2489075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2489290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2489358Z return mod(**inputs) 2025-08-14T21:57:50.2489613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2489694Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2489947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2490022Z layer_outputs = layer_module( 2025-08-14T21:57:50.2490282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2490362Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2490619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2490721Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2490969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2491062Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2491313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2491393Z value_states = self.v(current_states) 2025-08-14T21:57:50.2491397Z 2025-08-14T21:57:50.2491510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2491741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2491819Z return mod(**inputs) 2025-08-14T21:57:50.2492097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2492173Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2492428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2492502Z layer_outputs = layer_module( 2025-08-14T21:57:50.2492733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2492822Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2493074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2493164Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2493408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2493496Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2493755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2493871Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2493874Z 2025-08-14T21:57:50.2493988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2494195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2494263Z return mod(**inputs) 2025-08-14T21:57:50.2494520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2494595Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2494852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2494937Z layer_outputs = layer_module( 2025-08-14T21:57:50.2495168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2495258Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2495502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2495587Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2495836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2495923Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2496182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2496297Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2496320Z 2025-08-14T21:57:50.2496430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2496648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2496717Z return mod(**inputs) 2025-08-14T21:57:50.2496983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2497077Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2497310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2497387Z layer_outputs = layer_module( 2025-08-14T21:57:50.2497603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2497679Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2497934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2498016Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2498259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2498351Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2498578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2498692Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2498696Z 2025-08-14T21:57:50.2498795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2498992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2499066Z return mod(**inputs) 2025-08-14T21:57:50.2499298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2499377Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2499608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2499679Z layer_outputs = layer_module( 2025-08-14T21:57:50.2499904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2499979Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2500206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2500293Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2500521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2500610Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2500843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2500919Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2500922Z 2025-08-14T21:57:50.2501011Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2501112Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2501309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2501380Z return mod(**inputs) 2025-08-14T21:57:50.2501609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2501688Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2501919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2501990Z layer_outputs = layer_module( 2025-08-14T21:57:50.2502217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2503147Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2503384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2503475Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2503726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2503830Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2504055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2504132Z return self.weight * hidden_states 2025-08-14T21:57:50.2504135Z 2025-08-14T21:57:50.2504244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2504459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2504534Z return mod(**inputs) 2025-08-14T21:57:50.2504784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2504859Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2505103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2505176Z layer_outputs = layer_module( 2025-08-14T21:57:50.2505406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2505496Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2505736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2505840Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2506086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2506211Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2506460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2506544Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2506547Z 2025-08-14T21:57:50.2506657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2506865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2506933Z return mod(**inputs) 2025-08-14T21:57:50.2507186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2507259Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2507493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2507572Z layer_outputs = layer_module( 2025-08-14T21:57:50.2507790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2507874Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2508104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2508192Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2508431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2508544Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2509002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2509093Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2509151Z 2025-08-14T21:57:50.2509261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2509478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2509550Z return mod(**inputs) 2025-08-14T21:57:50.2509796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2509911Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2510157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2510239Z layer_outputs = layer_module( 2025-08-14T21:57:50.2510468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2510553Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2510826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2510923Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2511187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2511319Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2511563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2511656Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2511660Z 2025-08-14T21:57:50.2511767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2511980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2512057Z return mod(**inputs) 2025-08-14T21:57:50.2512310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2512399Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2512652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2512731Z layer_outputs = layer_module( 2025-08-14T21:57:50.2512977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2513063Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2513312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2513409Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2513657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2513787Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2514030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2514110Z return self.weight * hidden_states 2025-08-14T21:57:50.2514114Z 2025-08-14T21:57:50.2514230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2514433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2514510Z return mod(**inputs) 2025-08-14T21:57:50.2514755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2514832Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2515082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2515158Z layer_outputs = layer_module( 2025-08-14T21:57:50.2515387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2515499Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2515743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2515894Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2516148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2516272Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2516532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2516617Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2516621Z 2025-08-14T21:57:50.2516732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2516955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2517025Z return mod(**inputs) 2025-08-14T21:57:50.2517312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2517394Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2517663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2517760Z layer_outputs = layer_module( 2025-08-14T21:57:50.2517989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2518073Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2518322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2518405Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2518652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2518740Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2518980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2519069Z key_states = self.k(current_states) 2025-08-14T21:57:50.2519073Z 2025-08-14T21:57:50.2519180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2519392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2519459Z return mod(**inputs) 2025-08-14T21:57:50.2519704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2519789Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2520031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2520103Z layer_outputs = layer_module( 2025-08-14T21:57:50.2520342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2520425Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2520680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2520765Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2521006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2521100Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2521340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2521481Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2521485Z 2025-08-14T21:57:50.2521593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2521822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2521895Z return mod(**inputs) 2025-08-14T21:57:50.2522140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2522215Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2522485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2522559Z layer_outputs = layer_module( 2025-08-14T21:57:50.2522797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2522880Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2523126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2523217Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2523480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2523567Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2523835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2523987Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2523991Z 2025-08-14T21:57:50.2524095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2524282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2524344Z return mod(**inputs) 2025-08-14T21:57:50.2524583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2524653Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2524888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2524957Z layer_outputs = layer_module( 2025-08-14T21:57:50.2525168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2525255Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2525477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2525555Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2525785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2525865Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2526094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2526172Z value_states = self.v(current_states) 2025-08-14T21:57:50.2526175Z 2025-08-14T21:57:50.2526272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2526471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2526534Z return mod(**inputs) 2025-08-14T21:57:50.2526768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2526837Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2527062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2527138Z layer_outputs = layer_module( 2025-08-14T21:57:50.2527349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2527424Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2527655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2527750Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2527977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2528073Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2528292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2528403Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2528406Z 2025-08-14T21:57:50.2528512Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2528703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2528772Z return mod(**inputs) 2025-08-14T21:57:50.2529013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2529092Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2529334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2529405Z layer_outputs = layer_module( 2025-08-14T21:57:50.2529626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2529702Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2529932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2530009Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2530235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2530322Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2530550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2530652Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2530655Z 2025-08-14T21:57:50.2530762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2530953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2531027Z return mod(**inputs) 2025-08-14T21:57:50.2531255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2531324Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2531557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2531626Z layer_outputs = layer_module( 2025-08-14T21:57:50.2531841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2531924Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2532152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2532236Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2532459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2532535Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2532763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2532866Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2532870Z 2025-08-14T21:57:50.2532975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2533166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2533249Z return mod(**inputs) 2025-08-14T21:57:50.2533483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2533555Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2533780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2533874Z layer_outputs = layer_module( 2025-08-14T21:57:50.2534093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2534177Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2534413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2534492Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2534779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2534862Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2535111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2535197Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2535203Z 2025-08-14T21:57:50.2535283Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2535392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2535590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2535654Z return mod(**inputs) 2025-08-14T21:57:50.2535899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2535970Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2536212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2536284Z layer_outputs = layer_module( 2025-08-14T21:57:50.2536508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2536593Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2536819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2536897Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2537130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:57:50.2537234Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2537464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2537541Z return self.weight * hidden_states 2025-08-14T21:57:50.2537546Z 2025-08-14T21:57:50.2537647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2537849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2537913Z return mod(**inputs) 2025-08-14T21:57:50.2538140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2538221Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2538460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2538538Z layer_outputs = layer_module( 2025-08-14T21:57:50.2538756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2538844Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2539082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2539187Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2539418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2539502Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2539746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2539832Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2539836Z 2025-08-14T21:57:50.2539938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2540134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2540205Z return mod(**inputs) 2025-08-14T21:57:50.2540435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2540530Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2540765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2540853Z layer_outputs = layer_module( 2025-08-14T21:57:50.2541079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2541157Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2541385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2541470Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2541701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2541791Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2542021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2542098Z key_states = self.k(current_states) 2025-08-14T21:57:50.2542101Z 2025-08-14T21:57:50.2542210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2542408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2542481Z return mod(**inputs) 2025-08-14T21:57:50.2542710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2542783Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2543020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2543091Z layer_outputs = layer_module( 2025-08-14T21:57:50.2543309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2543395Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2543626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2543719Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2543963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2544052Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2544308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2544435Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2544438Z 2025-08-14T21:57:50.2544547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2544744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2544809Z return mod(**inputs) 2025-08-14T21:57:50.2545072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2545144Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2545375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2545470Z layer_outputs = layer_module( 2025-08-14T21:57:50.2545685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2545769Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2545996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2546074Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2546313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2546413Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2546644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2546825Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2546829Z 2025-08-14T21:57:50.2546933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2547134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2547199Z return mod(**inputs) 2025-08-14T21:57:50.2547428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2547508Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2547738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2547814Z layer_outputs = layer_module( 2025-08-14T21:57:50.2548034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2548108Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2548346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2548428Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2548659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2548749Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2548978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2549060Z value_states = self.v(current_states) 2025-08-14T21:57:50.2549064Z 2025-08-14T21:57:50.2549164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2549363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2549435Z return mod(**inputs) 2025-08-14T21:57:50.2549670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2549747Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2549981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2550051Z layer_outputs = layer_module( 2025-08-14T21:57:50.2550271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2550350Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2550599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2550690Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2550974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2551068Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2551317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2551447Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2551451Z 2025-08-14T21:57:50.2551564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2551769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2551836Z return mod(**inputs) 2025-08-14T21:57:50.2552092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2552167Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2552434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2552513Z layer_outputs = layer_module( 2025-08-14T21:57:50.2552761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2552852Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2553095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2553188Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2553439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2553524Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2553777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2553890Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2553895Z 2025-08-14T21:57:50.2554003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2554221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2554288Z return mod(**inputs) 2025-08-14T21:57:50.2554544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2554621Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2554874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2554956Z layer_outputs = layer_module( 2025-08-14T21:57:50.2555183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2555264Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2555521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2555608Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2556165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2556260Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2556518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2556640Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2556644Z 2025-08-14T21:57:50.2556753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2556974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2557044Z return mod(**inputs) 2025-08-14T21:57:50.2557297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2557441Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2557702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2557776Z layer_outputs = layer_module( 2025-08-14T21:57:50.2558032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2558114Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2558377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2558458Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2558711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2558806Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2559069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2559153Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2559165Z 2025-08-14T21:57:50.2559275Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2559385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2559606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2559675Z return mod(**inputs) 2025-08-14T21:57:50.2559928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2560012Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2560253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2560335Z layer_outputs = layer_module( 2025-08-14T21:57:50.2560563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2560645Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2560895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2560991Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2561234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2561346Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2561586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2561673Z return self.weight * hidden_states 2025-08-14T21:57:50.2561676Z 2025-08-14T21:57:50.2561782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2561992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2562069Z return mod(**inputs) 2025-08-14T21:57:50.2562317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2562392Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2562646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2562720Z layer_outputs = layer_module( 2025-08-14T21:57:50.2562955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2563037Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2563279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2563384Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2563646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2563776Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2564025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2564125Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2564129Z 2025-08-14T21:57:50.2564243Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2564447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2564514Z return mod(**inputs) 2025-08-14T21:57:50.2564765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2564840Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2565106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2565183Z layer_outputs = layer_module( 2025-08-14T21:57:50.2565433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2565524Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2565769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2565870Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2566110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2566231Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2566485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2566568Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2566574Z 2025-08-14T21:57:50.2566676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2566882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2566947Z return mod(**inputs) 2025-08-14T21:57:50.2567187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2567261Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2567494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2567572Z layer_outputs = layer_module( 2025-08-14T21:57:50.2567792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2567871Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2568111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2568204Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2568443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2568557Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2568786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2568872Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2568876Z 2025-08-14T21:57:50.2568956Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2569064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2569260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2569324Z return mod(**inputs) 2025-08-14T21:57:50.2569564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2569658Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2569895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2569974Z layer_outputs = layer_module( 2025-08-14T21:57:50.2570208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2570293Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2570528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2570609Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2570849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2570969Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2571200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2571283Z return self.weight * hidden_states 2025-08-14T21:57:50.2571301Z 2025-08-14T21:57:50.2571403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2571607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2571672Z return mod(**inputs) 2025-08-14T21:57:50.2571913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2571992Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2572231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2572309Z layer_outputs = layer_module( 2025-08-14T21:57:50.2572530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2572607Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2572856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2572937Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2573173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2573265Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2573501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2573584Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2573588Z 2025-08-14T21:57:50.2573688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2573889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2573963Z return mod(**inputs) 2025-08-14T21:57:50.2574201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2574273Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2574519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2574588Z layer_outputs = layer_module( 2025-08-14T21:57:50.2574814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2574890Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2575128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2575216Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2575451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2575559Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2575786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2575862Z key_states = self.k(current_states) 2025-08-14T21:57:50.2575881Z 2025-08-14T21:57:50.2575991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2576185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2576248Z return mod(**inputs) 2025-08-14T21:57:50.2576492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2576563Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2576798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2576887Z layer_outputs = layer_module( 2025-08-14T21:57:50.2577106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2577209Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2577441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2577522Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2577756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2577837Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2578072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2578201Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2578205Z 2025-08-14T21:57:50.2578309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2578509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2578572Z return mod(**inputs) 2025-08-14T21:57:50.2578810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2578883Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2579112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2579188Z layer_outputs = layer_module( 2025-08-14T21:57:50.2579402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2579480Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2579719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2579803Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2580036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2580119Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2580350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2580510Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2580514Z 2025-08-14T21:57:50.2580615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2580817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2580883Z return mod(**inputs) 2025-08-14T21:57:50.2581117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2581215Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2581450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2581520Z layer_outputs = layer_module( 2025-08-14T21:57:50.2581748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2581852Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2582087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2582168Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2582401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2582489Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2582743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2582824Z value_states = self.v(current_states) 2025-08-14T21:57:50.2582834Z 2025-08-14T21:57:50.2582935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2583146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2583222Z return mod(**inputs) 2025-08-14T21:57:50.2583454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2583528Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2583781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2583858Z layer_outputs = layer_module( 2025-08-14T21:57:50.2584095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2584177Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2584420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2584513Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2584764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2584847Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2585081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2585186Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2585189Z 2025-08-14T21:57:50.2585297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2585499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2585561Z return mod(**inputs) 2025-08-14T21:57:50.2585799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2585869Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2586103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2586179Z layer_outputs = layer_module( 2025-08-14T21:57:50.2586389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2586471Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2586704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2586783Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2587018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2587120Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2587354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2587460Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2587464Z 2025-08-14T21:57:50.2587566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2587787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2587852Z return mod(**inputs) 2025-08-14T21:57:50.2588084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2588162Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2588394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2588471Z layer_outputs = layer_module( 2025-08-14T21:57:50.2588702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2588782Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2589039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2589123Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2589359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2589440Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2589668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2589782Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2589786Z 2025-08-14T21:57:50.2589886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2590086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2590161Z return mod(**inputs) 2025-08-14T21:57:50.2590405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2590487Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2590737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2590811Z layer_outputs = layer_module( 2025-08-14T21:57:50.2591045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2591126Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2591365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2591456Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2591698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2591814Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2592063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2592140Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2592143Z 2025-08-14T21:57:50.2592251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2592445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2592515Z return mod(**inputs) 2025-08-14T21:57:50.2592760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2592836Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2593093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2593186Z layer_outputs = layer_module( 2025-08-14T21:57:50.2593420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2593509Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2593792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2593883Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2594135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-08-14T21:57:50.2594271Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:57:50.2594274Z 2025-08-14T21:57:50.2594365Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2594470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2594710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2594781Z return mod(**inputs) 2025-08-14T21:57:50.2595063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2595159Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2595405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2595478Z layer_outputs = layer_module( 2025-08-14T21:57:50.2595714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2595865Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2596143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2596230Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2596482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:57:50.2596602Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2596862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2596949Z return self.weight * hidden_states 2025-08-14T21:57:50.2596953Z 2025-08-14T21:57:50.2597072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2597284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2597363Z return mod(**inputs) 2025-08-14T21:57:50.2597622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2597700Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2597972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2598045Z layer_outputs = layer_module( 2025-08-14T21:57:50.2598261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2598346Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2598591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2598686Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2598948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2599039Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2599310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2599394Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2599428Z 2025-08-14T21:57:50.2599547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2599760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2599831Z return mod(**inputs) 2025-08-14T21:57:50.2600093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2600189Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2600452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2600534Z layer_outputs = layer_module( 2025-08-14T21:57:50.2600769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2600860Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2601145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2601233Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2601507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2601598Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2601851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2601934Z key_states = self.k(current_states) 2025-08-14T21:57:50.2601938Z 2025-08-14T21:57:50.2602046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2602267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2602337Z return mod(**inputs) 2025-08-14T21:57:50.2602587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2602676Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2602926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2603011Z layer_outputs = layer_module( 2025-08-14T21:57:50.2603247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2603332Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2603587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2603672Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2603919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2604017Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2604262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2604410Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2604414Z 2025-08-14T21:57:50.2604528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2604740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2604819Z return mod(**inputs) 2025-08-14T21:57:50.2605069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2605155Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2605406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2605482Z layer_outputs = layer_module( 2025-08-14T21:57:50.2605723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2605832Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2606080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2606176Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2606426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2606553Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2606800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2606965Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2606969Z 2025-08-14T21:57:50.2607084Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2607298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2607390Z return mod(**inputs) 2025-08-14T21:57:50.2607624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2607722Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2607971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2608048Z layer_outputs = layer_module( 2025-08-14T21:57:50.2608277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2608365Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2608608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2608831Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2609089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2609178Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2609430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2609512Z value_states = self.v(current_states) 2025-08-14T21:57:50.2609518Z 2025-08-14T21:57:50.2609625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2609842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2609912Z return mod(**inputs) 2025-08-14T21:57:50.2610167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2610243Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2610490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2610572Z layer_outputs = layer_module( 2025-08-14T21:57:50.2610790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2610874Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2611105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2611186Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2611421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2611505Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2611735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2611852Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2611856Z 2025-08-14T21:57:50.2611959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2612213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2612278Z return mod(**inputs) 2025-08-14T21:57:50.2612514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2612621Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2612857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2612930Z layer_outputs = layer_module( 2025-08-14T21:57:50.2613153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2613227Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2613467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2613574Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2613808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2613925Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2614153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2614270Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2614273Z 2025-08-14T21:57:50.2614376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2614571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2614642Z return mod(**inputs) 2025-08-14T21:57:50.2614871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2614942Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2615183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2615255Z layer_outputs = layer_module( 2025-08-14T21:57:50.2615480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2615558Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2615792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2615880Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2616109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2616192Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2616431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2616539Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2616543Z 2025-08-14T21:57:50.2616652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2616851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2616916Z return mod(**inputs) 2025-08-14T21:57:50.2617155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2617225Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2617463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2617533Z layer_outputs = layer_module( 2025-08-14T21:57:50.2617748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2617833Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2618084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2618168Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2618408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2618508Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2618742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2618819Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2618823Z 2025-08-14T21:57:50.2618901Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2619009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2619204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2619269Z return mod(**inputs) 2025-08-14T21:57:50.2619524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2619596Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2619848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2619923Z layer_outputs = layer_module( 2025-08-14T21:57:50.2620140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2620219Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2620437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2620530Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2620747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2620841Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2621063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2621137Z return self.weight * hidden_states 2025-08-14T21:57:50.2621140Z 2025-08-14T21:57:50.2621237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2621430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2621489Z return mod(**inputs) 2025-08-14T21:57:50.2621715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2621781Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2621998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2622073Z layer_outputs = layer_module( 2025-08-14T21:57:50.2622280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2622352Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2622576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2622665Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2622885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2622997Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2623218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2623301Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2623304Z 2025-08-14T21:57:50.2623401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2623617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2623680Z return mod(**inputs) 2025-08-14T21:57:50.2623905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2623986Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2624238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2624306Z layer_outputs = layer_module( 2025-08-14T21:57:50.2624516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2624590Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2624815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2624899Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2625130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2625250Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2625481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2625570Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2625574Z 2025-08-14T21:57:50.2625671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2625854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2625922Z return mod(**inputs) 2025-08-14T21:57:50.2626142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2626210Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2626441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2626510Z layer_outputs = layer_module( 2025-08-14T21:57:50.2626721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2626792Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2627013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2627106Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2627321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2627429Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2627653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2627731Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2627734Z 2025-08-14T21:57:50.2627818Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2627916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2628105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2628178Z return mod(**inputs) 2025-08-14T21:57:50.2628399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2628475Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2628694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2628762Z layer_outputs = layer_module( 2025-08-14T21:57:50.2628973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2629052Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2629298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2629384Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2629602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2629755Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2629970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2630041Z return self.weight * hidden_states 2025-08-14T21:57:50.2630044Z 2025-08-14T21:57:50.2630147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2630333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2630401Z return mod(**inputs) 2025-08-14T21:57:50.2630636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2630705Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2630943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2631015Z layer_outputs = layer_module( 2025-08-14T21:57:50.2631222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2631303Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2631518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2631601Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2631815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2631894Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2632125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2632201Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2632205Z 2025-08-14T21:57:50.2632305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2632506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2632570Z return mod(**inputs) 2025-08-14T21:57:50.2632813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2632883Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2633114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2633192Z layer_outputs = layer_module( 2025-08-14T21:57:50.2633410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2633489Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2633728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2633808Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2634041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2634122Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2634353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2634436Z key_states = self.k(current_states) 2025-08-14T21:57:50.2634440Z 2025-08-14T21:57:50.2634542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2634743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2634825Z return mod(**inputs) 2025-08-14T21:57:50.2635063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2635139Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2635376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2635461Z layer_outputs = layer_module( 2025-08-14T21:57:50.2635698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2635857Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2636136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2636224Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2636512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2636614Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2636891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2637040Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2637046Z 2025-08-14T21:57:50.2637158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2637373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2637452Z return mod(**inputs) 2025-08-14T21:57:50.2637760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2637851Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2638099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2638175Z layer_outputs = layer_module( 2025-08-14T21:57:50.2638404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2638487Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2638721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2638814Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2639047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2639130Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2639368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2639525Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2639531Z 2025-08-14T21:57:50.2639646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2639843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2639913Z return mod(**inputs) 2025-08-14T21:57:50.2640160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2640236Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2640476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2640550Z layer_outputs = layer_module( 2025-08-14T21:57:50.2640771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2640861Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2641098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2641199Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2641439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2641519Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2641771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2641850Z value_states = self.v(current_states) 2025-08-14T21:57:50.2641854Z 2025-08-14T21:57:50.2641957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2642160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2642226Z return mod(**inputs) 2025-08-14T21:57:50.2642465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2642554Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2642790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2642882Z layer_outputs = layer_module( 2025-08-14T21:57:50.2643105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2643186Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2643429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2643509Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2643748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2643829Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2644063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2644187Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2644191Z 2025-08-14T21:57:50.2644294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2644492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2644563Z return mod(**inputs) 2025-08-14T21:57:50.2644798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2644877Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2645110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2645182Z layer_outputs = layer_module( 2025-08-14T21:57:50.2645408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2645489Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2645730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2645809Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2646039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2646129Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2646360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2646467Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2646470Z 2025-08-14T21:57:50.2646577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2646775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2646848Z return mod(**inputs) 2025-08-14T21:57:50.2647101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2647173Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2647412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2647501Z layer_outputs = layer_module( 2025-08-14T21:57:50.2647719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2647802Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2648033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2648120Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2648352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2648452Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2648691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2648822Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2648826Z 2025-08-14T21:57:50.2648937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2649133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2649197Z return mod(**inputs) 2025-08-14T21:57:50.2649435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2649505Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2649737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2649825Z layer_outputs = layer_module( 2025-08-14T21:57:50.2650040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2650120Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2650344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2650422Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2650653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2650731Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2650955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2651036Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2651040Z 2025-08-14T21:57:50.2651117Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2651223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2651416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2651478Z return mod(**inputs) 2025-08-14T21:57:50.2651714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2651784Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2652015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2652083Z layer_outputs = layer_module( 2025-08-14T21:57:50.2652293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2652376Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2652600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2652708Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2652938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:57:50.2653040Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2653268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2653360Z return self.weight * hidden_states 2025-08-14T21:57:50.2653364Z 2025-08-14T21:57:50.2653463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2653659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2653723Z return mod(**inputs) 2025-08-14T21:57:50.2653947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2654024Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2654268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2654346Z layer_outputs = layer_module( 2025-08-14T21:57:50.2654578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2654656Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2654884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2654962Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2655199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2655289Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2655540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2655632Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2655636Z 2025-08-14T21:57:50.2655743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2655954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2656031Z return mod(**inputs) 2025-08-14T21:57:50.2656277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2656360Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2656611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2656685Z layer_outputs = layer_module( 2025-08-14T21:57:50.2656930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2657009Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2657243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2657331Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2657576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2657666Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2657888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2657962Z key_states = self.k(current_states) 2025-08-14T21:57:50.2657965Z 2025-08-14T21:57:50.2658071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2658260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2658329Z return mod(**inputs) 2025-08-14T21:57:50.2658558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2658653Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2658890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2658960Z layer_outputs = layer_module( 2025-08-14T21:57:50.2659193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2659279Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2659508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2659593Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2659823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2659904Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2660162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2660291Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2660316Z 2025-08-14T21:57:50.2660426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2660628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2660692Z return mod(**inputs) 2025-08-14T21:57:50.2660931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2661002Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2661232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2661308Z layer_outputs = layer_module( 2025-08-14T21:57:50.2661522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2661608Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2661839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2661919Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2662163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2662246Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2662481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2662641Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2662645Z 2025-08-14T21:57:50.2662747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2662953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2663022Z return mod(**inputs) 2025-08-14T21:57:50.2663261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2663341Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2663578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2663654Z layer_outputs = layer_module( 2025-08-14T21:57:50.2663875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2663952Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2664195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2664275Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2664531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2664623Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2664854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2664956Z value_states = self.v(current_states) 2025-08-14T21:57:50.2664960Z 2025-08-14T21:57:50.2665060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2665258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2665326Z return mod(**inputs) 2025-08-14T21:57:50.2665559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2665629Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2665887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2665960Z layer_outputs = layer_module( 2025-08-14T21:57:50.2666212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2666287Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2666509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2666596Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2666821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2666910Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2667136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2667243Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2667248Z 2025-08-14T21:57:50.2667359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2667556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2667622Z return mod(**inputs) 2025-08-14T21:57:50.2667862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2667935Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2668176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2668248Z layer_outputs = layer_module( 2025-08-14T21:57:50.2668462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2668545Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2668775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2668863Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2669089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2669171Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2669406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2669511Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2669514Z 2025-08-14T21:57:50.2669616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2669820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2669885Z return mod(**inputs) 2025-08-14T21:57:50.2670122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2670215Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2670447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2670527Z layer_outputs = layer_module( 2025-08-14T21:57:50.2670742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2670833Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2671071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2671149Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2671385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2671466Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2671721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2671839Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2671843Z 2025-08-14T21:57:50.2671961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2672167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2672235Z return mod(**inputs) 2025-08-14T21:57:50.2672473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2672554Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2672800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2672874Z layer_outputs = layer_module( 2025-08-14T21:57:50.2673110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2673195Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2673442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2673526Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2673769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2673866Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2674108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2674192Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2674204Z 2025-08-14T21:57:50.2674313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2674527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2674606Z return mod(**inputs) 2025-08-14T21:57:50.2674860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2674938Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2675202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2675279Z layer_outputs = layer_module( 2025-08-14T21:57:50.2675529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2675614Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2675944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2676048Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2676309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward 2025-08-14T21:57:50.2676478Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:57:50.2676491Z 2025-08-14T21:57:50.2676580Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2676693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2676924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2677018Z return mod(**inputs) 2025-08-14T21:57:50.2677276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2677361Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2677621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2677694Z layer_outputs = layer_module( 2025-08-14T21:57:50.2677944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2678053Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2678294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2678404Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2678636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2678740Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2678968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2679052Z return self.weight * hidden_states 2025-08-14T21:57:50.2679055Z 2025-08-14T21:57:50.2679157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2679355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2679429Z return mod(**inputs) 2025-08-14T21:57:50.2679662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2679733Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2679971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2680043Z layer_outputs = layer_module( 2025-08-14T21:57:50.2680265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2680340Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2680570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2680667Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2680898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2681026Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2681256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2681337Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2681340Z 2025-08-14T21:57:50.2681453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2681648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2681713Z return mod(**inputs) 2025-08-14T21:57:50.2681954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2682027Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2682265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2682337Z layer_outputs = layer_module( 2025-08-14T21:57:50.2682575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2682661Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2682895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2683003Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2683242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2683356Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2683594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2683675Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2683678Z 2025-08-14T21:57:50.2683799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2684009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2684073Z return mod(**inputs) 2025-08-14T21:57:50.2684337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2684412Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2684646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2684726Z layer_outputs = layer_module( 2025-08-14T21:57:50.2684943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2685022Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2685260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2685352Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2685590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2685703Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2685933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2686024Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2686028Z 2025-08-14T21:57:50.2686108Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2686221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2686419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2686483Z return mod(**inputs) 2025-08-14T21:57:50.2686723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2686797Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2687031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2687111Z layer_outputs = layer_module( 2025-08-14T21:57:50.2687330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2687417Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2687654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2687735Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2687976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2688082Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2688315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2688430Z return self.weight * hidden_states 2025-08-14T21:57:50.2688434Z 2025-08-14T21:57:50.2688544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2688756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2688844Z return mod(**inputs) 2025-08-14T21:57:50.2689091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2689174Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2689423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2689497Z layer_outputs = layer_module( 2025-08-14T21:57:50.2689734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2689835Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2690083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2690194Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2690427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2690519Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2690750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2690838Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2690842Z 2025-08-14T21:57:50.2690952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2691161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2691237Z return mod(**inputs) 2025-08-14T21:57:50.2691487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2691563Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2691819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2691894Z layer_outputs = layer_module( 2025-08-14T21:57:50.2692131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2692213Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2692463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2692555Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2692798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2692895Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2693137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2693219Z key_states = self.k(current_states) 2025-08-14T21:57:50.2693222Z 2025-08-14T21:57:50.2693336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2693545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2693612Z return mod(**inputs) 2025-08-14T21:57:50.2693867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2693942Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2694197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2694270Z layer_outputs = layer_module( 2025-08-14T21:57:50.2694500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2694607Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2694849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2694960Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2695214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2695300Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2695548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2695682Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2695686Z 2025-08-14T21:57:50.2695795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2696027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2696099Z return mod(**inputs) 2025-08-14T21:57:50.2696371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2696449Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2696696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2696777Z layer_outputs = layer_module( 2025-08-14T21:57:50.2697006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2697088Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2697337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2697423Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2697675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2697761Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2698006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2698177Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2698181Z 2025-08-14T21:57:50.2698290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2698504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2698573Z return mod(**inputs) 2025-08-14T21:57:50.2698821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2698902Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2699135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2699208Z layer_outputs = layer_module( 2025-08-14T21:57:50.2699438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2699516Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2699753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2699832Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2700064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2700153Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2700380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2700458Z value_states = self.v(current_states) 2025-08-14T21:57:50.2700490Z 2025-08-14T21:57:50.2700594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2700792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2700864Z return mod(**inputs) 2025-08-14T21:57:50.2701094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2701185Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2701424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2701494Z layer_outputs = layer_module( 2025-08-14T21:57:50.2701728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2701806Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2702052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2702143Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2702384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2702467Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2702704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2702812Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2702815Z 2025-08-14T21:57:50.2702921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2703115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2703181Z return mod(**inputs) 2025-08-14T21:57:50.2703421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2703498Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2703742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2703825Z layer_outputs = layer_module( 2025-08-14T21:57:50.2704052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2704144Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2704384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2704468Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2704717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2704812Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2705052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2705160Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2705164Z 2025-08-14T21:57:50.2705266Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2705466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2705533Z return mod(**inputs) 2025-08-14T21:57:50.2705762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2705843Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2706081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2706163Z layer_outputs = layer_module( 2025-08-14T21:57:50.2706392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2706495Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2706746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2706832Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2707071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2707181Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2707425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2707539Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2707542Z 2025-08-14T21:57:50.2707642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2707837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2707925Z return mod(**inputs) 2025-08-14T21:57:50.2708158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2708251Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2708483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2708553Z layer_outputs = layer_module( 2025-08-14T21:57:50.2708905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2708996Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2709238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2709328Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2709572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2709668Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2709912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2709993Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2709997Z 2025-08-14T21:57:50.2710091Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2710199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2710413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2710483Z return mod(**inputs) 2025-08-14T21:57:50.2710728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2710811Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2711058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2711134Z layer_outputs = layer_module( 2025-08-14T21:57:50.2711369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2711453Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2711707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2711796Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2712039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:57:50.2712157Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2712399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2712479Z return self.weight * hidden_states 2025-08-14T21:57:50.2712482Z 2025-08-14T21:57:50.2712643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2712853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2712930Z return mod(**inputs) 2025-08-14T21:57:50.2713174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2713276Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2713527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2713600Z layer_outputs = layer_module( 2025-08-14T21:57:50.2713830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2713921Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2714187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2714284Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2714529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2714644Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2714902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2714988Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2714991Z 2025-08-14T21:57:50.2715109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2715320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2715390Z return mod(**inputs) 2025-08-14T21:57:50.2715649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2715730Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2716041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2716128Z layer_outputs = layer_module( 2025-08-14T21:57:50.2716371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2716465Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2716713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2716801Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2717059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2717152Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2717408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2717494Z key_states = self.k(current_states) 2025-08-14T21:57:50.2717499Z 2025-08-14T21:57:50.2717611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2717833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2717907Z return mod(**inputs) 2025-08-14T21:57:50.2718165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2718251Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2718496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2718580Z layer_outputs = layer_module( 2025-08-14T21:57:50.2718813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2718895Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2719184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2719271Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2719514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2719628Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2719867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2720011Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2720015Z 2025-08-14T21:57:50.2720124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2720334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2720409Z return mod(**inputs) 2025-08-14T21:57:50.2720673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2720757Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2721019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2721096Z layer_outputs = layer_module( 2025-08-14T21:57:50.2721334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2721415Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2721659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2721751Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2721996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2722093Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2722338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2722505Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2722509Z 2025-08-14T21:57:50.2722628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2722836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2722909Z return mod(**inputs) 2025-08-14T21:57:50.2723156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2723231Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2723484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2723558Z layer_outputs = layer_module( 2025-08-14T21:57:50.2723790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2723878Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2724126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2724216Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2724459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2724552Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2724781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2724855Z value_states = self.v(current_states) 2025-08-14T21:57:50.2724859Z 2025-08-14T21:57:50.2724958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2725157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2725269Z return mod(**inputs) 2025-08-14T21:57:50.2725505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2725574Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2725827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2725902Z layer_outputs = layer_module( 2025-08-14T21:57:50.2726110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2726190Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2726410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2726488Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2726736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2726817Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2727054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2727169Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2727173Z 2025-08-14T21:57:50.2727270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2727469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2727533Z return mod(**inputs) 2025-08-14T21:57:50.2727757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2727836Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2728063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2728133Z layer_outputs = layer_module( 2025-08-14T21:57:50.2728356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2728428Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2728653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2728726Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2728944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2729029Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2729249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2729358Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2729362Z 2025-08-14T21:57:50.2729460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2729650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2729719Z return mod(**inputs) 2025-08-14T21:57:50.2729942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2730014Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2730248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2730317Z layer_outputs = layer_module( 2025-08-14T21:57:50.2730527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2730599Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2730818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2730919Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2731137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2731216Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2731464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2731567Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2731571Z 2025-08-14T21:57:50.2731675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2731867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2731929Z return mod(**inputs) 2025-08-14T21:57:50.2732181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2732255Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2732484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2732869Z layer_outputs = layer_module( 2025-08-14T21:57:50.2733091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2733179Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2733408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2733486Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2733725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2733809Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2734044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2734125Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2734128Z 2025-08-14T21:57:50.2734210Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2734321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2734516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2734581Z return mod(**inputs) 2025-08-14T21:57:50.2734818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2734888Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2735133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2735201Z layer_outputs = layer_module( 2025-08-14T21:57:50.2735411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2735494Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2735721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2735815Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2736042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2736134Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2736363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2736435Z return self.weight * hidden_states 2025-08-14T21:57:50.2736439Z 2025-08-14T21:57:50.2736539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2736739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2736822Z return mod(**inputs) 2025-08-14T21:57:50.2737055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2737125Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2737352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2737455Z layer_outputs = layer_module( 2025-08-14T21:57:50.2737665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2737737Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2737962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2738048Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2738290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2738404Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2738644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2738730Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2738734Z 2025-08-14T21:57:50.2738831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2739023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2739084Z return mod(**inputs) 2025-08-14T21:57:50.2739303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2739378Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2739597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2739665Z layer_outputs = layer_module( 2025-08-14T21:57:50.2739882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2739959Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2740187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2740275Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2740505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2740621Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2740836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2740920Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2740925Z 2025-08-14T21:57:50.2741025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2741214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2741285Z return mod(**inputs) 2025-08-14T21:57:50.2741508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2741579Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2741812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2741880Z layer_outputs = layer_module( 2025-08-14T21:57:50.2742095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2742170Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2742395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2742505Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2742728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2742837Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2743082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2743160Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2743163Z 2025-08-14T21:57:50.2743268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2743460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2743523Z return mod(**inputs) 2025-08-14T21:57:50.2743756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2743842Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2744078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2744162Z layer_outputs = layer_module( 2025-08-14T21:57:50.2744377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2744465Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2744689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2744776Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2745009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-08-14T21:57:50.2745132Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-08-14T21:57:50.2745136Z 2025-08-14T21:57:50.2745223Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2745322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2745514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2745585Z return mod(**inputs) 2025-08-14T21:57:50.2745809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2745886Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2746113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2746182Z layer_outputs = layer_module( 2025-08-14T21:57:50.2746397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2746473Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2746700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2746791Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2747019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:57:50.2747130Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2747361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2747437Z return self.weight * hidden_states 2025-08-14T21:57:50.2747441Z 2025-08-14T21:57:50.2747551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2747746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2747811Z return mod(**inputs) 2025-08-14T21:57:50.2748050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2748149Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2748388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2748460Z layer_outputs = layer_module( 2025-08-14T21:57:50.2748677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2748779Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2749008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2749097Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2749327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2749410Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2749664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2749749Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2749753Z 2025-08-14T21:57:50.2749877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2750093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2750163Z return mod(**inputs) 2025-08-14T21:57:50.2750415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2750485Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2750715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2750793Z layer_outputs = layer_module( 2025-08-14T21:57:50.2751006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2751086Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2751321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2751402Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2751643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2751733Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2751974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2752062Z key_states = self.k(current_states) 2025-08-14T21:57:50.2752065Z 2025-08-14T21:57:50.2752171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2752385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2752453Z return mod(**inputs) 2025-08-14T21:57:50.2752699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2752780Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2753025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2753099Z layer_outputs = layer_module( 2025-08-14T21:57:50.2753334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2753415Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2753664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2753746Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2753985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2754102Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2754350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2754486Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2754498Z 2025-08-14T21:57:50.2754624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2754838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2754914Z return mod(**inputs) 2025-08-14T21:57:50.2755167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2755243Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2755501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2755579Z layer_outputs = layer_module( 2025-08-14T21:57:50.2755921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2756018Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2756288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2756389Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2756640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2756729Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2756989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2757161Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2757166Z 2025-08-14T21:57:50.2757289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2757509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2757585Z return mod(**inputs) 2025-08-14T21:57:50.2757823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2757897Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2758143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2758220Z layer_outputs = layer_module( 2025-08-14T21:57:50.2758457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2758550Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2758803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2758890Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2759148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2759237Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2759493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2759580Z value_states = self.v(current_states) 2025-08-14T21:57:50.2759584Z 2025-08-14T21:57:50.2759704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2759931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2760001Z return mod(**inputs) 2025-08-14T21:57:50.2760252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2760337Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2760590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2760696Z layer_outputs = layer_module( 2025-08-14T21:57:50.2760937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2761021Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2761297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2761384Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2761638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2761726Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2761973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2762123Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2762129Z 2025-08-14T21:57:50.2762241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2762468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2762548Z return mod(**inputs) 2025-08-14T21:57:50.2762804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2762890Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2763140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2763217Z layer_outputs = layer_module( 2025-08-14T21:57:50.2763460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2763545Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2763798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2763894Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2764147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2764245Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2764497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2764613Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2764616Z 2025-08-14T21:57:50.2764735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2764950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2765027Z return mod(**inputs) 2025-08-14T21:57:50.2765284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2765361Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2765616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2765685Z layer_outputs = layer_module( 2025-08-14T21:57:50.2765904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2765986Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2766216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2766302Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2766531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2766613Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2766868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2766974Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2766977Z 2025-08-14T21:57:50.2767086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2767279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2767360Z return mod(**inputs) 2025-08-14T21:57:50.2767596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2767667Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2767897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2767974Z layer_outputs = layer_module( 2025-08-14T21:57:50.2768203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2768290Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2768531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:57:50.2768615Z self_attention_outputs = self.layer[0]( 2025-08-14T21:57:50.2768855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:57:50.2768938Z attention_output = self.SelfAttention( 2025-08-14T21:57:50.2769164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2769248Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2769252Z 2025-08-14T21:57:50.2769332Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2769441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2769637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2769703Z return mod(**inputs) 2025-08-14T21:57:50.2769944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2770015Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2770247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2770324Z layer_outputs = layer_module( 2025-08-14T21:57:50.2770540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2770624Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2770851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2770931Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2771171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:57:50.2771276Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2771515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2771594Z return self.weight * hidden_states 2025-08-14T21:57:50.2771597Z 2025-08-14T21:57:50.2771698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2771900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2771965Z return mod(**inputs) 2025-08-14T21:57:50.2772199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2772279Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2772512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2772611Z layer_outputs = layer_module( 2025-08-14T21:57:50.2772833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2772909Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2773165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2773244Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2773478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2773561Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2773787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:57:50.2773873Z query_states = self.q(hidden_states) 2025-08-14T21:57:50.2773879Z 2025-08-14T21:57:50.2773997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2774193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2774283Z return mod(**inputs) 2025-08-14T21:57:50.2774518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2774597Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2774827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2774897Z layer_outputs = layer_module( 2025-08-14T21:57:50.2775121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2775207Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2775427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2775511Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2775729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2775817Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2776042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:57:50.2776119Z key_states = self.k(current_states) 2025-08-14T21:57:50.2776123Z 2025-08-14T21:57:50.2776233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2776430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2776501Z return mod(**inputs) 2025-08-14T21:57:50.2776730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2776805Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2777046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2777118Z layer_outputs = layer_module( 2025-08-14T21:57:50.2777334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2777421Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2777653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2777739Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2777976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2778057Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2778287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:57:50.2778434Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:57:50.2778438Z 2025-08-14T21:57:50.2778544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2778737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2778817Z return mod(**inputs) 2025-08-14T21:57:50.2779048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2779117Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2779340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2779416Z layer_outputs = layer_module( 2025-08-14T21:57:50.2779624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2779731Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2779953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2780048Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2780278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2780358Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2780575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:57:50.2780725Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:57:50.2780728Z 2025-08-14T21:57:50.2780824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2781017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2781082Z return mod(**inputs) 2025-08-14T21:57:50.2781306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2781381Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2781610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2781687Z layer_outputs = layer_module( 2025-08-14T21:57:50.2781896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2781971Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2782207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2782286Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2782515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2782607Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2782843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:57:50.2782927Z value_states = self.v(current_states) 2025-08-14T21:57:50.2782931Z 2025-08-14T21:57:50.2783030Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2783225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2783297Z return mod(**inputs) 2025-08-14T21:57:50.2783527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2783599Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2783837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2783908Z layer_outputs = layer_module( 2025-08-14T21:57:50.2784155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2784230Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2784462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2784591Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2784812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2784901Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2785126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2785230Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2785234Z 2025-08-14T21:57:50.2785342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2785553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2785619Z return mod(**inputs) 2025-08-14T21:57:50.2785873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2785947Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2786185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2786257Z layer_outputs = layer_module( 2025-08-14T21:57:50.2786480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2786562Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2786784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2786863Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2787100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2787183Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2787419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:57:50.2787528Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:57:50.2787532Z 2025-08-14T21:57:50.2787635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2787837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2787901Z return mod(**inputs) 2025-08-14T21:57:50.2788141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2788215Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2788461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2788545Z layer_outputs = layer_module( 2025-08-14T21:57:50.2788773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2788859Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2789097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2789175Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2789411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2789495Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2789735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:57:50.2789856Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:57:50.2789879Z 2025-08-14T21:57:50.2789987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2790200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2790269Z return mod(**inputs) 2025-08-14T21:57:50.2790533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2790614Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2790858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2790932Z layer_outputs = layer_module( 2025-08-14T21:57:50.2791169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2791252Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2791522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:57:50.2791609Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:57:50.2791867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:57:50.2791965Z attention_output = self.EncDecAttention( 2025-08-14T21:57:50.2792205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:57:50.2792285Z attn_output = self.o(attn_output) 2025-08-14T21:57:50.2792296Z 2025-08-14T21:57:50.2792381Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2792489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2792702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2792769Z return mod(**inputs) 2025-08-14T21:57:50.2793013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2793097Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2793342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2793417Z layer_outputs = layer_module( 2025-08-14T21:57:50.2793651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2793731Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2793977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2794074Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2794316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:57:50.2794425Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:57:50.2794670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:57:50.2794757Z return self.weight * hidden_states 2025-08-14T21:57:50.2794761Z 2025-08-14T21:57:50.2794867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2795077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2795150Z return mod(**inputs) 2025-08-14T21:57:50.2795393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2795467Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2795723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2795869Z layer_outputs = layer_module( 2025-08-14T21:57:50.2796124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2796236Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2796493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2796601Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2796871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2797008Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2797263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:57:50.2797347Z hidden_states = self.wi(hidden_states) 2025-08-14T21:57:50.2797351Z 2025-08-14T21:57:50.2797465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2797695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2797768Z return mod(**inputs) 2025-08-14T21:57:50.2798044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2798123Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2798379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2798454Z layer_outputs = layer_module( 2025-08-14T21:57:50.2798684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2798773Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2799016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2799113Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2799361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2799483Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2799733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:57:50.2799821Z hidden_states = self.act(hidden_states) 2025-08-14T21:57:50.2799824Z 2025-08-14T21:57:50.2799932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2800151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2800219Z return mod(**inputs) 2025-08-14T21:57:50.2800471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:57:50.2800546Z decoder_outputs = self.decoder( 2025-08-14T21:57:50.2800793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:57:50.2800877Z layer_outputs = layer_module( 2025-08-14T21:57:50.2801109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:50.2801193Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:50.2801444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:57:50.2801538Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:57:50.2801786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:57:50.2801907Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:57:50.2802146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:57:50.2802238Z hidden_states = self.wo(hidden_states) 2025-08-14T21:57:50.2802262Z 2025-08-14T21:57:50.2802349Z cudagraph partition due to non gpu ops 2025-08-14T21:57:50.2802463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2802670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2802757Z return mod(**inputs) 2025-08-14T21:57:50.2803013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1789, in forward 2025-08-14T21:57:50.2803142Z sequence_output = sequence_output * (self.model_dim**-0.5) 2025-08-14T21:57:50.2803146Z 2025-08-14T21:57:50.2803254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2803467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2803534Z return mod(**inputs) 2025-08-14T21:57:50.2803805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1791, in forward 2025-08-14T21:57:50.2803902Z lm_logits = self.lm_head(sequence_output) 2025-08-14T21:57:50.2803907Z 2025-08-14T21:57:50.2804016Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2804252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2804322Z return mod(**inputs) 2025-08-14T21:57:50.2804572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-08-14T21:57:50.2804728Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:57:50.2804732Z 2025-08-14T21:57:50.2804839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2805053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2805122Z return mod(**inputs) 2025-08-14T21:57:50.2805377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-08-14T21:57:50.2805529Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:57:50.2805533Z 2025-08-14T21:57:50.2805641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:50.2805853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:50.2805922Z return mod(**inputs) 2025-08-14T21:57:50.2806175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-08-14T21:57:50.2806321Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:57:50.2806324Z 2025-08-14T21:58:00.0685492Z Compilation time (from dynamo_timed): 18.188576337 2025-08-14T21:58:00.0788410Z pass 2025-08-14T21:58:00.0788847Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:00.0789759Z TIMING: _recursive_pre_grad_passes:0.01221 _recursive_joint_graph_passes:0.58348 _recursive_post_grad_passes:0.1973 async_compile.wait:0.7779 code_gen:9.46624 inductor_compile:11.10452 backend_compile:15.26453 gc:0.00038 entire_frame_compile:18.18858 total_wall_time:18.18858 2025-08-14T21:58:00.0790866Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:20429 | FakeTensor.__torch_dispatch__:5656 | ProxyTorchDispatchMode.__torch_dispatch__:7292 2025-08-14T21:58:00.0791413Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-08-14T21:58:05.5855496Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:58:05.5857043Z from pkg_resources import resource_filename 2025-08-14T21:58:06.1703139Z 2025-08-14T21:58:07.3468916Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:58:07.3469390Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:58:07.3477333Z cpu eval T5Small 2025-08-14T21:58:08.6551591Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:09.0707377Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:09.5010351Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:20.6297926Z Compilation time (from dynamo_timed): 9.524283846 2025-08-14T21:58:20.6459976Z pass 2025-08-14T21:58:20.6460730Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:20.6461559Z TIMING: _recursive_pre_grad_passes:0.01249 async_compile.wait:0.00647 backend_compile:6.59828 gc:0.00248 entire_frame_compile:9.52428 total_wall_time:9.52428 2025-08-14T21:58:20.6463172Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:2289 | FakeTensor.__torch_dispatch__:17 2025-08-14T21:58:20.6463869Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-08-14T21:58:25.7673405Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:58:25.7674430Z from pkg_resources import resource_filename 2025-08-14T21:58:26.3643732Z 2025-08-14T21:58:29.1180019Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:58:29.1180320Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:58:29.1196438Z cpu eval TrOCRForCausalLM 2025-08-14T21:58:29.2740440Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2025-08-14T21:58:29.3110907Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:29.5746324Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:29.8177879Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:37.6842857Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6846123Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6846474Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6846805Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6850799Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6851108Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6853939Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6854175Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6854392Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6854665Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6854891Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6855225Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6855483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6855902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6856282Z return mod(**inputs) 2025-08-14T21:58:37.6856749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6857209Z outputs = self.model.decoder( 2025-08-14T21:58:37.6857663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6858087Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6858455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6859169Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6859613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6860107Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6860562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.6861149Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.6861331Z 2025-08-14T21:58:37.6861456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6861846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6862222Z return mod(**inputs) 2025-08-14T21:58:37.6862663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6863144Z outputs = self.model.decoder( 2025-08-14T21:58:37.6863629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6864149Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6864555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6864959Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6865394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6865856Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6866340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.6866779Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.6866938Z 2025-08-14T21:58:37.6867059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6867456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6867826Z return mod(**inputs) 2025-08-14T21:58:37.6868225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6868661Z outputs = self.model.decoder( 2025-08-14T21:58:37.6869089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6869515Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6869902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6870308Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6870733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6871183Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6871631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.6872081Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.6872241Z 2025-08-14T21:58:37.6872341Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6872566Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6872794Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6873051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6873489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6873844Z return mod(**inputs) 2025-08-14T21:58:37.6874238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6874667Z outputs = self.model.decoder( 2025-08-14T21:58:37.6875116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6875545Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6876011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6876435Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6876891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6877358Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6877804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.6878242Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.6878401Z 2025-08-14T21:58:37.6878519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6878936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6879283Z return mod(**inputs) 2025-08-14T21:58:37.6879697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6880126Z outputs = self.model.decoder( 2025-08-14T21:58:37.6880547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6880944Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6881311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6881696Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6882096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.6882559Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.6882751Z 2025-08-14T21:58:37.6882862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6883239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6883576Z return mod(**inputs) 2025-08-14T21:58:37.6883965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6884378Z outputs = self.model.decoder( 2025-08-14T21:58:37.6884778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6885209Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6885574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6885952Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6886354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.6886855Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.6887275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.6887641Z return self.act(input) 2025-08-14T21:58:37.6887759Z 2025-08-14T21:58:37.6887869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6888284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6888624Z return mod(**inputs) 2025-08-14T21:58:37.6888999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6889406Z outputs = self.model.decoder( 2025-08-14T21:58:37.6889807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6890231Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6890603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6890996Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6891414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.6891829Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.6891979Z 2025-08-14T21:58:37.6892088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6892464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6892794Z return mod(**inputs) 2025-08-14T21:58:37.6893169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6894576Z outputs = self.model.decoder( 2025-08-14T21:58:37.6895009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6895427Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6895795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6896179Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6896577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6897021Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6897446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.6897901Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.6898076Z 2025-08-14T21:58:37.6898189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6898567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6898910Z return mod(**inputs) 2025-08-14T21:58:37.6899290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6899694Z outputs = self.model.decoder( 2025-08-14T21:58:37.6900089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6900504Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6900868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6901258Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6901676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6902119Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6902550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.6902972Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.6903116Z 2025-08-14T21:58:37.6903233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6903617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6903962Z return mod(**inputs) 2025-08-14T21:58:37.6904351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6904764Z outputs = self.model.decoder( 2025-08-14T21:58:37.6905165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6905642Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6906016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6906410Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6906826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6907285Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6907714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.6908138Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.6908298Z 2025-08-14T21:58:37.6908386Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6908615Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6909224Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6909531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6909936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6910293Z return mod(**inputs) 2025-08-14T21:58:37.6910704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6911095Z outputs = self.model.decoder( 2025-08-14T21:58:37.6911496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6911959Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6912320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6912703Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6913114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6913543Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6913977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.6914390Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.6914534Z 2025-08-14T21:58:37.6914652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6915021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6915362Z return mod(**inputs) 2025-08-14T21:58:37.6915947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6916381Z outputs = self.model.decoder( 2025-08-14T21:58:37.6916788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6917211Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6917579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6917951Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6918344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.6918780Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.6918956Z 2025-08-14T21:58:37.6919069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6919425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6919751Z return mod(**inputs) 2025-08-14T21:58:37.6920117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6920498Z outputs = self.model.decoder( 2025-08-14T21:58:37.6920910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6921292Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6921641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6922023Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6922410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.6922858Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.6923268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.6923625Z return self.act(input) 2025-08-14T21:58:37.6923751Z 2025-08-14T21:58:37.6923860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6924252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6924586Z return mod(**inputs) 2025-08-14T21:58:37.6924995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6925383Z outputs = self.model.decoder( 2025-08-14T21:58:37.6925770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6926168Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6926529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6926894Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6927297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.6927717Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.6927877Z 2025-08-14T21:58:37.6927987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6928365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6928701Z return mod(**inputs) 2025-08-14T21:58:37.6929083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6929493Z outputs = self.model.decoder( 2025-08-14T21:58:37.6929895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6930296Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6930660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6931043Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6931449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6931886Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6932318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.6932769Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.6932942Z 2025-08-14T21:58:37.6933053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6933431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6933774Z return mod(**inputs) 2025-08-14T21:58:37.6934146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6934559Z outputs = self.model.decoder( 2025-08-14T21:58:37.6934957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6935428Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6935783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6936176Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6936585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6937043Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6937470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.6937894Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.6938035Z 2025-08-14T21:58:37.6938154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6938524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6938886Z return mod(**inputs) 2025-08-14T21:58:37.6939270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6939693Z outputs = self.model.decoder( 2025-08-14T21:58:37.6940092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6940499Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6940865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6941238Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6941651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6942089Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6942523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.6942953Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.6943108Z 2025-08-14T21:58:37.6943194Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6943422Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6943651Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6943882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6944241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6944573Z return mod(**inputs) 2025-08-14T21:58:37.6944927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6945315Z outputs = self.model.decoder( 2025-08-14T21:58:37.6945693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6946084Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6946427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6946789Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6947185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6947594Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6948006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.6948405Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.6948540Z 2025-08-14T21:58:37.6948652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6949004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6949330Z return mod(**inputs) 2025-08-14T21:58:37.6949714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6950122Z outputs = self.model.decoder( 2025-08-14T21:58:37.6950541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6950969Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6951333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6951712Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6952131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.6952611Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.6952789Z 2025-08-14T21:58:37.6952904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6953294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6953638Z return mod(**inputs) 2025-08-14T21:58:37.6954058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6954551Z outputs = self.model.decoder( 2025-08-14T21:58:37.6954978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6955517Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6955981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6956383Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6956808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.6957285Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.6957705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.6958047Z return self.act(input) 2025-08-14T21:58:37.6958167Z 2025-08-14T21:58:37.6958271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6958628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6958945Z return mod(**inputs) 2025-08-14T21:58:37.6959305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6959690Z outputs = self.model.decoder( 2025-08-14T21:58:37.6960057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6960462Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6960831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6961234Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6961633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.6962054Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.6962196Z 2025-08-14T21:58:37.6962298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6962651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6962963Z return mod(**inputs) 2025-08-14T21:58:37.6963317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6963709Z outputs = self.model.decoder( 2025-08-14T21:58:37.6964116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6964570Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6964939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6965332Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6965757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6966169Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6966611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.6967071Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.6967247Z 2025-08-14T21:58:37.6967355Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6967750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6968114Z return mod(**inputs) 2025-08-14T21:58:37.6968497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6968936Z outputs = self.model.decoder( 2025-08-14T21:58:37.6969348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6969767Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6970139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6970527Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6970930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6971338Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6971758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.6972158Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.6972293Z 2025-08-14T21:58:37.6972412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6972789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6973152Z return mod(**inputs) 2025-08-14T21:58:37.6973550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6973968Z outputs = self.model.decoder( 2025-08-14T21:58:37.6974367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6974781Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6975155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6975538Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6975955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6976403Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6976841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.6977270Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.6977431Z 2025-08-14T21:58:37.6977520Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6977752Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6977972Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.6978227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6978610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6978978Z return mod(**inputs) 2025-08-14T21:58:37.6979351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6979763Z outputs = self.model.decoder( 2025-08-14T21:58:37.6980161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6980580Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6980946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6981329Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6981741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.6982178Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.6982628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.6983052Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.6983195Z 2025-08-14T21:58:37.6983329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6983700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6984044Z return mod(**inputs) 2025-08-14T21:58:37.6984420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6984819Z outputs = self.model.decoder( 2025-08-14T21:58:37.6985220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6985625Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6985989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6986362Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6986767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.6987219Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.6987400Z 2025-08-14T21:58:37.6987511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6987889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6988229Z return mod(**inputs) 2025-08-14T21:58:37.6988610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6989009Z outputs = self.model.decoder( 2025-08-14T21:58:37.6989404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6989809Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6990165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6990547Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6990954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.6991409Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.6991808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.6992251Z return self.act(input) 2025-08-14T21:58:37.6992370Z 2025-08-14T21:58:37.6992487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6992864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6993201Z return mod(**inputs) 2025-08-14T21:58:37.6993584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6994026Z outputs = self.model.decoder( 2025-08-14T21:58:37.6994420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6994847Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6995214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6995607Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.6996103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.6996539Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.6996688Z 2025-08-14T21:58:37.6996809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.6997219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.6997577Z return mod(**inputs) 2025-08-14T21:58:37.6997980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.6998413Z outputs = self.model.decoder( 2025-08-14T21:58:37.6998818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.6999233Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.6999606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.6999994Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7000405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7000850Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7001297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.7001748Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.7001935Z 2025-08-14T21:58:37.7002049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7002436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7002811Z return mod(**inputs) 2025-08-14T21:58:37.7003194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7003614Z outputs = self.model.decoder( 2025-08-14T21:58:37.7004020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7004442Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7004817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7005207Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7005629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7006052Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7006478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.7006888Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.7007030Z 2025-08-14T21:58:37.7007147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7007516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7007858Z return mod(**inputs) 2025-08-14T21:58:37.7008244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7008817Z outputs = self.model.decoder( 2025-08-14T21:58:37.7009234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7009641Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7010003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7010433Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7010849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7011286Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7011715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.7012141Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.7012298Z 2025-08-14T21:58:37.7012432Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7012667Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7012886Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7013162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7013543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7013882Z return mod(**inputs) 2025-08-14T21:58:37.7014264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7014682Z outputs = self.model.decoder( 2025-08-14T21:58:37.7015089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7015492Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7015860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7016242Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7016651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7017076Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7017504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.7017919Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.7018062Z 2025-08-14T21:58:37.7018171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7018555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7018902Z return mod(**inputs) 2025-08-14T21:58:37.7019279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7019681Z outputs = self.model.decoder( 2025-08-14T21:58:37.7020081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7020489Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7020845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7021227Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7021634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7022089Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7022260Z 2025-08-14T21:58:37.7022362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7022715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7023033Z return mod(**inputs) 2025-08-14T21:58:37.7023430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7023805Z outputs = self.model.decoder( 2025-08-14T21:58:37.7024176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7024587Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7024945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7025324Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7025732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7026184Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7026604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.7026967Z return self.act(input) 2025-08-14T21:58:37.7027086Z 2025-08-14T21:58:37.7027206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7027573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7027896Z return mod(**inputs) 2025-08-14T21:58:37.7028257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7028640Z outputs = self.model.decoder( 2025-08-14T21:58:37.7029008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7029392Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7029738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7030099Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7030498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.7030913Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.7031058Z 2025-08-14T21:58:37.7031174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7031549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7031871Z return mod(**inputs) 2025-08-14T21:58:37.7032262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7032666Z outputs = self.model.decoder( 2025-08-14T21:58:37.7033070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7033471Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7033833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7034204Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7034609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7035042Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7035484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.7036011Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.7036201Z 2025-08-14T21:58:37.7036314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7036704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7037054Z return mod(**inputs) 2025-08-14T21:58:37.7037445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7037890Z outputs = self.model.decoder( 2025-08-14T21:58:37.7038273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7038692Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7039094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7039491Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7039921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7040378Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7040824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.7041269Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.7041444Z 2025-08-14T21:58:37.7041564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7041962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7042326Z return mod(**inputs) 2025-08-14T21:58:37.7042714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7043126Z outputs = self.model.decoder( 2025-08-14T21:58:37.7043536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7043958Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7044331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7044714Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7045143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7045543Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7045937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.7046336Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.7046487Z 2025-08-14T21:58:37.7046570Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7046784Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7046988Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7047226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7047580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7047917Z return mod(**inputs) 2025-08-14T21:58:37.7048308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7048693Z outputs = self.model.decoder( 2025-08-14T21:58:37.7049081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7049482Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7049845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7050207Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7050589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7050990Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7051392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.7051784Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.7051944Z 2025-08-14T21:58:37.7052048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7052407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7052726Z return mod(**inputs) 2025-08-14T21:58:37.7053076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7053456Z outputs = self.model.decoder( 2025-08-14T21:58:37.7053824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7054204Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7054540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7054905Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7055317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7055776Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7055960Z 2025-08-14T21:58:37.7056083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7056445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7056787Z return mod(**inputs) 2025-08-14T21:58:37.7057159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7057565Z outputs = self.model.decoder( 2025-08-14T21:58:37.7057963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7058364Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7058717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7059101Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7059507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7059961Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7060370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.7060708Z return self.act(input) 2025-08-14T21:58:37.7060819Z 2025-08-14T21:58:37.7061087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7061455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7061803Z return mod(**inputs) 2025-08-14T21:58:37.7062178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7062592Z outputs = self.model.decoder( 2025-08-14T21:58:37.7062989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7063395Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7063764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7064139Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7064545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.7064968Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.7065112Z 2025-08-14T21:58:37.7065228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7065600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7065942Z return mod(**inputs) 2025-08-14T21:58:37.7066325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7066764Z outputs = self.model.decoder( 2025-08-14T21:58:37.7067169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7067565Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7067940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7068295Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7068709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7069153Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7069592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.7070064Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.7070248Z 2025-08-14T21:58:37.7070358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7070752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7071074Z return mod(**inputs) 2025-08-14T21:58:37.7071429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7071830Z outputs = self.model.decoder( 2025-08-14T21:58:37.7072235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7072641Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7073014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7073407Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7073829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7074308Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7074766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.7075209Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.7075354Z 2025-08-14T21:58:37.7075465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7075937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7076308Z return mod(**inputs) 2025-08-14T21:58:37.7076709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7077131Z outputs = self.model.decoder( 2025-08-14T21:58:37.7077537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7077944Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7078305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7078660Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7079065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7079493Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7079910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.7080337Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.7080490Z 2025-08-14T21:58:37.7080574Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7080802Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7081052Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7081300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7081682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7082018Z return mod(**inputs) 2025-08-14T21:58:37.7082399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7082823Z outputs = self.model.decoder( 2025-08-14T21:58:37.7083219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7083627Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7083992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7084385Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7084811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7085249Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7085705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.7086128Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.7086273Z 2025-08-14T21:58:37.7086384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7086763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7087106Z return mod(**inputs) 2025-08-14T21:58:37.7087480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7087900Z outputs = self.model.decoder( 2025-08-14T21:58:37.7088303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7088708Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7089069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7089450Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7089860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7090320Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7090510Z 2025-08-14T21:58:37.7090620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7091001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7091344Z return mod(**inputs) 2025-08-14T21:58:37.7091717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7092131Z outputs = self.model.decoder( 2025-08-14T21:58:37.7092535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7092942Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7093300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7093686Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7094095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7094544Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7094949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.7095312Z return self.act(input) 2025-08-14T21:58:37.7095429Z 2025-08-14T21:58:37.7095549Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7095944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7096284Z return mod(**inputs) 2025-08-14T21:58:37.7096664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7097091Z outputs = self.model.decoder( 2025-08-14T21:58:37.7097482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7097885Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7098250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7098623Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7099032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.7099480Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.7099624Z 2025-08-14T21:58:37.7099740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7100123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7100465Z return mod(**inputs) 2025-08-14T21:58:37.7100846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7101242Z outputs = self.model.decoder( 2025-08-14T21:58:37.7101637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7102047Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7102409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7102779Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7103188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7103614Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7104062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.7104495Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.7104675Z 2025-08-14T21:58:37.7104783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7105157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7105488Z return mod(**inputs) 2025-08-14T21:58:37.7105865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7106268Z outputs = self.model.decoder( 2025-08-14T21:58:37.7106666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7107063Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7107428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7107807Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7108205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7108796Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7109254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.7109686Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.7109830Z 2025-08-14T21:58:37.7109940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7110338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7110793Z return mod(**inputs) 2025-08-14T21:58:37.7111187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7111608Z outputs = self.model.decoder( 2025-08-14T21:58:37.7112089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7112523Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7112899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7113299Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7127556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7128240Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7128885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.7129340Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.7129553Z 2025-08-14T21:58:37.7129650Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7129889Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7130112Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7130358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7130751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7131110Z return mod(**inputs) 2025-08-14T21:58:37.7131498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7131919Z outputs = self.model.decoder( 2025-08-14T21:58:37.7132337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7132748Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7133120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7133512Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7133931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7134366Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7134792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.7135209Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.7135355Z 2025-08-14T21:58:37.7135477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7135852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7136201Z return mod(**inputs) 2025-08-14T21:58:37.7136587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7136995Z outputs = self.model.decoder( 2025-08-14T21:58:37.7137388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7137792Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7138164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7138538Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7138945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7139404Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7139628Z 2025-08-14T21:58:37.7139749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7140122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7140467Z return mod(**inputs) 2025-08-14T21:58:37.7140856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7141307Z outputs = self.model.decoder( 2025-08-14T21:58:37.7141706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7142120Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7142489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7142878Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7143309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7143769Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7144201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.7144556Z return self.act(input) 2025-08-14T21:58:37.7144685Z 2025-08-14T21:58:37.7144796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7145178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7145513Z return mod(**inputs) 2025-08-14T21:58:37.7145893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7146307Z outputs = self.model.decoder( 2025-08-14T21:58:37.7146703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7147119Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7147484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7147876Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7148282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.7148694Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.7148847Z 2025-08-14T21:58:37.7148958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7149338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7149674Z return mod(**inputs) 2025-08-14T21:58:37.7150058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7150475Z outputs = self.model.decoder( 2025-08-14T21:58:37.7150889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7151297Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7151674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7152064Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7152486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7152945Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7153396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.7153869Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.7154048Z 2025-08-14T21:58:37.7154163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7154579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7154947Z return mod(**inputs) 2025-08-14T21:58:37.7155349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7155873Z outputs = self.model.decoder( 2025-08-14T21:58:37.7156347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7156783Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7157163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7157567Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7157974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7158406Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7158810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.7159227Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.7159364Z 2025-08-14T21:58:37.7159478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7159844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7160162Z return mod(**inputs) 2025-08-14T21:58:37.7160524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7160911Z outputs = self.model.decoder( 2025-08-14T21:58:37.7161282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7161670Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7162023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7162383Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7162761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7163172Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7163581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.7163977Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.7164128Z 2025-08-14T21:58:37.7164211Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7164429Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7164643Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7164875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7165237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7165566Z return mod(**inputs) 2025-08-14T21:58:37.7165919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7166305Z outputs = self.model.decoder( 2025-08-14T21:58:37.7166685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7167068Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7167407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7167780Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7168160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7168554Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7168976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.7169397Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.7169540Z 2025-08-14T21:58:37.7169659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7170061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7170408Z return mod(**inputs) 2025-08-14T21:58:37.7170790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7171191Z outputs = self.model.decoder( 2025-08-14T21:58:37.7171563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7171940Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7172301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7172655Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7173056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7173473Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7173650Z 2025-08-14T21:58:37.7173751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7174105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7174422Z return mod(**inputs) 2025-08-14T21:58:37.7174781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7175171Z outputs = self.model.decoder( 2025-08-14T21:58:37.7175571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7175967Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7176311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7176667Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7177056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7177473Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7177849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.7178179Z return self.act(input) 2025-08-14T21:58:37.7178288Z 2025-08-14T21:58:37.7178388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7178734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7179050Z return mod(**inputs) 2025-08-14T21:58:37.7179396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7179764Z outputs = self.model.decoder( 2025-08-14T21:58:37.7180127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7180503Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7180830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7181175Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7181550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.7181931Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.7182063Z 2025-08-14T21:58:37.7182163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7182556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7182869Z return mod(**inputs) 2025-08-14T21:58:37.7183210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7183598Z outputs = self.model.decoder( 2025-08-14T21:58:37.7183964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7184333Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7184659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7185005Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7185378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7185792Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7186186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.7186632Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.7186807Z 2025-08-14T21:58:37.7186925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7187290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7187631Z return mod(**inputs) 2025-08-14T21:58:37.7188010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7188395Z outputs = self.model.decoder( 2025-08-14T21:58:37.7188762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7189153Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7189521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7189902Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7190312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7190743Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7191171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.7191579Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.7191728Z 2025-08-14T21:58:37.7191838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7192213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7192555Z return mod(**inputs) 2025-08-14T21:58:37.7192925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7193332Z outputs = self.model.decoder( 2025-08-14T21:58:37.7193731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7194129Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7194491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7194871Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7195274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7195695Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7196222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.7196681Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.7196831Z 2025-08-14T21:58:37.7196919Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7197150Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7197376Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7197639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7198030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7198369Z return mod(**inputs) 2025-08-14T21:58:37.7198749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7199163Z outputs = self.model.decoder( 2025-08-14T21:58:37.7199553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7199951Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7200337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7200712Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7201135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7201569Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7201997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.7202413Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.7202564Z 2025-08-14T21:58:37.7202673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7203048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7203391Z return mod(**inputs) 2025-08-14T21:58:37.7203764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7204168Z outputs = self.model.decoder( 2025-08-14T21:58:37.7204569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7204971Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7205339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7205727Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7206132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7206589Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7206777Z 2025-08-14T21:58:37.7206885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7207261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7207601Z return mod(**inputs) 2025-08-14T21:58:37.7207989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7208420Z outputs = self.model.decoder( 2025-08-14T21:58:37.7209577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7210240Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7210652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7211060Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7211513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7212024Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7212643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.7213013Z return self.act(input) 2025-08-14T21:58:37.7213138Z 2025-08-14T21:58:37.7213262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7213676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7214092Z return mod(**inputs) 2025-08-14T21:58:37.7214492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7214913Z outputs = self.model.decoder( 2025-08-14T21:58:37.7215329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7215748Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7216119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7216551Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7216978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.7217448Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.7217599Z 2025-08-14T21:58:37.7217714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7218111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7218464Z return mod(**inputs) 2025-08-14T21:58:37.7218838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7219284Z outputs = self.model.decoder( 2025-08-14T21:58:37.7219709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7220098Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7220440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7220802Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7221191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7221604Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7222007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.7222434Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.7222601Z 2025-08-14T21:58:37.7222716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7223098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7223436Z return mod(**inputs) 2025-08-14T21:58:37.7223798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7224209Z outputs = self.model.decoder( 2025-08-14T21:58:37.7224614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7225057Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7225415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7225831Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7226223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7226631Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7227065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.7227525Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.7227676Z 2025-08-14T21:58:37.7227789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7228181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7228551Z return mod(**inputs) 2025-08-14T21:58:37.7228933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7229360Z outputs = self.model.decoder( 2025-08-14T21:58:37.7229770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7230152Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7230502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7230924Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7231369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7231823Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7232265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.7232699Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.7232848Z 2025-08-14T21:58:37.7232942Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7233164Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7233387Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7233636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7234025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7234366Z return mod(**inputs) 2025-08-14T21:58:37.7234744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7235149Z outputs = self.model.decoder( 2025-08-14T21:58:37.7235554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7236052Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7236437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7236834Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7237265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7237708Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7238145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.7238568Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.7238721Z 2025-08-14T21:58:37.7238834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7239219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7239559Z return mod(**inputs) 2025-08-14T21:58:37.7239953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7240376Z outputs = self.model.decoder( 2025-08-14T21:58:37.7240781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7241197Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7241566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7241960Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7242474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7242999Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7243191Z 2025-08-14T21:58:37.7243305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7243704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7244064Z return mod(**inputs) 2025-08-14T21:58:37.7244446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7244859Z outputs = self.model.decoder( 2025-08-14T21:58:37.7245257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7245668Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7246049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7246434Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7246855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7247314Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7247726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.7248091Z return self.act(input) 2025-08-14T21:58:37.7248207Z 2025-08-14T21:58:37.7248316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7248699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7249041Z return mod(**inputs) 2025-08-14T21:58:37.7249412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7249824Z outputs = self.model.decoder( 2025-08-14T21:58:37.7250226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7250641Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7250999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7251395Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7251778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.7252170Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.7252307Z 2025-08-14T21:58:37.7252410Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7252768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7253090Z return mod(**inputs) 2025-08-14T21:58:37.7253460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7253872Z outputs = self.model.decoder( 2025-08-14T21:58:37.7254270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7254685Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7255059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7255450Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7255869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7256307Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7256743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:58:37.7257195Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:58:37.7257361Z 2025-08-14T21:58:37.7257473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7257827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7258188Z return mod(**inputs) 2025-08-14T21:58:37.7258568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7258985Z outputs = self.model.decoder( 2025-08-14T21:58:37.7259374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7259779Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7260121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7260493Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7260884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7261006Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7261256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:58:37.7261340Z key_states = self.k_proj(current_states) 2025-08-14T21:58:37.7261343Z 2025-08-14T21:58:37.7261453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7261648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7261713Z return mod(**inputs) 2025-08-14T21:58:37.7261968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7262044Z outputs = self.model.decoder( 2025-08-14T21:58:37.7262299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7262370Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7262589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7262679Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7262925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7263022Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7263276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:58:37.7263364Z value_states = self.v_proj(current_states) 2025-08-14T21:58:37.7263367Z 2025-08-14T21:58:37.7263455Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7263537Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7263615Z cudagraph partition due to non gpu ops 2025-08-14T21:58:37.7263725Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7263925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7263997Z return mod(**inputs) 2025-08-14T21:58:37.7264251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7264324Z outputs = self.model.decoder( 2025-08-14T21:58:37.7264579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7264653Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7264869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7264956Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7265228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:58:37.7265334Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:58:37.7265582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:58:37.7265685Z attn_output = self.out_proj(attn_output) 2025-08-14T21:58:37.7265688Z 2025-08-14T21:58:37.7265797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7266017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7266089Z return mod(**inputs) 2025-08-14T21:58:37.7266383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7266461Z outputs = self.model.decoder( 2025-08-14T21:58:37.7266760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7266840Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7267108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7267205Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7267479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7267617Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7267621Z 2025-08-14T21:58:37.7267743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7267946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7268023Z return mod(**inputs) 2025-08-14T21:58:37.7268277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7268356Z outputs = self.model.decoder( 2025-08-14T21:58:37.7268618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7268692Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7268919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7269004Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7269281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:58:37.7269434Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:58:37.7269657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:37.7269740Z return self.act(input) 2025-08-14T21:58:37.7269747Z 2025-08-14T21:58:37.7269860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7270076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7270157Z return mod(**inputs) 2025-08-14T21:58:37.7270443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:58:37.7270525Z outputs = self.model.decoder( 2025-08-14T21:58:37.7270801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:58:37.7270879Z layer_outputs = decoder_layer( 2025-08-14T21:58:37.7271125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:37.7271211Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:37.7271489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:58:37.7271603Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:58:37.7271607Z 2025-08-14T21:58:37.7271717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7271936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7272031Z return mod(**inputs) 2025-08-14T21:58:37.7272302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 839, in forward 2025-08-14T21:58:37.7272411Z logits = self.output_projection(outputs[0]) 2025-08-14T21:58:37.7272414Z 2025-08-14T21:58:37.7272518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:37.7272725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:37.7272802Z return mod(**inputs) 2025-08-14T21:58:37.7273094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 844, in forward 2025-08-14T21:58:37.7273265Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:58:37.7273269Z 2025-08-14T21:58:46.4557435Z Compilation time (from dynamo_timed): 15.235323079 2025-08-14T21:58:46.4592022Z pass 2025-08-14T21:58:46.4595263Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:46.4596441Z TIMING: _recursive_pre_grad_passes:0.00841 _recursive_joint_graph_passes:0.76645 _recursive_post_grad_passes:0.08598 async_compile.wait:0.78709 code_gen:8.08782 inductor_compile:9.28719 backend_compile:12.84389 gc:0.00064 entire_frame_compile:15.23532 total_wall_time:15.23532 2025-08-14T21:58:46.4597455Z STATS: call_* op count: 443 | FakeTensorMode.__torch_dispatch__:14347 | FakeTensor.__torch_dispatch__:4678 | ProxyTorchDispatchMode.__torch_dispatch__:5467 2025-08-14T21:58:46.4598018Z Dynamo produced 1 graphs covering 443 ops with 0 graph breaks (0 unique) 2025-08-14T21:58:52.0495537Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:58:52.0496530Z from pkg_resources import resource_filename 2025-08-14T21:58:52.7091533Z 2025-08-14T21:58:59.2805612Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:58:59.2808106Z loading model: 0it [00:06, ?it/s] 2025-08-14T21:58:59.2828388Z cpu eval XGLMForCausalLM 2025-08-14T21:58:59.6641457Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2025-08-14T21:58:59.7592055Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:59:00.2992018Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:59:00.8432681Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:59:15.7163356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7163901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7164289Z return mod(**inputs) 2025-08-14T21:59:15.7164761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7165194Z outputs = self.model( 2025-08-14T21:59:15.7165597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7166071Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7166462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7167211Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7167640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7168093Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7168529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7169068Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7169263Z 2025-08-14T21:59:15.7169387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7169805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7170181Z return mod(**inputs) 2025-08-14T21:59:15.7170572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7171043Z outputs = self.model( 2025-08-14T21:59:15.7171546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7172040Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7172426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7172818Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7173239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7173683Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7174141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7174579Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7174744Z 2025-08-14T21:59:15.7174863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7175256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7175611Z return mod(**inputs) 2025-08-14T21:59:15.7176042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7176457Z outputs = self.model( 2025-08-14T21:59:15.7176858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7177266Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7177634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7178021Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7178429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7178867Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7179292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7179739Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7179915Z 2025-08-14T21:59:15.7180035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7180422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7180788Z return mod(**inputs) 2025-08-14T21:59:15.7181165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7181565Z outputs = self.model( 2025-08-14T21:59:15.7181938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7182410Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7182820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7183209Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7183613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7184069Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7184502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7185028Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7185238Z 2025-08-14T21:59:15.7185353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7185736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7186093Z return mod(**inputs) 2025-08-14T21:59:15.7186501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7186914Z outputs = self.model( 2025-08-14T21:59:15.7187309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7187716Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7188082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7188469Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7188877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7189305Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7189725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7190154Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7190308Z 2025-08-14T21:59:15.7190427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7190813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7191167Z return mod(**inputs) 2025-08-14T21:59:15.7191557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7191964Z outputs = self.model( 2025-08-14T21:59:15.7192350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7192769Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7193148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7193546Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7193964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7194412Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7194847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7195287Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7195456Z 2025-08-14T21:59:15.7196097Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7196505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7196846Z return mod(**inputs) 2025-08-14T21:59:15.7197233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7197650Z outputs = self.model( 2025-08-14T21:59:15.7198041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7198478Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7198846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7199240Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7199672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7200106Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7200541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7201034Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7201227Z 2025-08-14T21:59:15.7201345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7201749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7202106Z return mod(**inputs) 2025-08-14T21:59:15.7202509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7202911Z outputs = self.model( 2025-08-14T21:59:15.7203307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7203726Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7204100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7204483Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7204894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7205335Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7205766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7206189Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7206345Z 2025-08-14T21:59:15.7206457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7206846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7207192Z return mod(**inputs) 2025-08-14T21:59:15.7207575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7207988Z outputs = self.model( 2025-08-14T21:59:15.7208367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7209009Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7209398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7209797Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7210208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7210686Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7210885Z 2025-08-14T21:59:15.7210998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7211389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7211755Z return mod(**inputs) 2025-08-14T21:59:15.7212150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7212558Z outputs = self.model( 2025-08-14T21:59:15.7212924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7213385Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7213751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7214151Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7214546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7215025Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7215436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7215796Z return self.act(input) 2025-08-14T21:59:15.7215914Z 2025-08-14T21:59:15.7216024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7216402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7216765Z return mod(**inputs) 2025-08-14T21:59:15.7217185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7217589Z outputs = self.model( 2025-08-14T21:59:15.7217996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7218412Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7218778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7219162Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7219569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7219976Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7220130Z 2025-08-14T21:59:15.7220239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7220619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7220965Z return mod(**inputs) 2025-08-14T21:59:15.7221333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7221729Z outputs = self.model( 2025-08-14T21:59:15.7222117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7222518Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7222887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7223270Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7223681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7224101Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7224532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7224975Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7225149Z 2025-08-14T21:59:15.7225268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7225642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7225991Z return mod(**inputs) 2025-08-14T21:59:15.7226364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7226757Z outputs = self.model( 2025-08-14T21:59:15.7227137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7227540Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7227913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7228308Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7228717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7229149Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7229591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7230013Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7230164Z 2025-08-14T21:59:15.7230271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7230654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7230991Z return mod(**inputs) 2025-08-14T21:59:15.7231365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7231777Z outputs = self.model( 2025-08-14T21:59:15.7232157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7232630Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7233006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7233396Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7233797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7234244Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7234678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7235190Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7235367Z 2025-08-14T21:59:15.7235483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7235970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7236332Z return mod(**inputs) 2025-08-14T21:59:15.7236719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7237125Z outputs = self.model( 2025-08-14T21:59:15.7237511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7237925Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7238294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7238700Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7239115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7239561Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7239984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7240461Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7240672Z 2025-08-14T21:59:15.7240793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7241177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7241519Z return mod(**inputs) 2025-08-14T21:59:15.7241915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7242317Z outputs = self.model( 2025-08-14T21:59:15.7242697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7243142Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7243524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7243917Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7244320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7244780Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7245222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7245643Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7245807Z 2025-08-14T21:59:15.7245919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7246311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7246667Z return mod(**inputs) 2025-08-14T21:59:15.7247086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7247505Z outputs = self.model( 2025-08-14T21:59:15.7247907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7248301Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7248666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7249053Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7249454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7249883Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7250320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7250770Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7250927Z 2025-08-14T21:59:15.7251043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7251416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7251758Z return mod(**inputs) 2025-08-14T21:59:15.7252137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7252525Z outputs = self.model( 2025-08-14T21:59:15.7252925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7253345Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7253709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7254080Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7254483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7254914Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7255333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7255785Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7255980Z 2025-08-14T21:59:15.7256087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7256464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7256797Z return mod(**inputs) 2025-08-14T21:59:15.7257170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7257563Z outputs = self.model( 2025-08-14T21:59:15.7257937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7258367Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7258747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7259143Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7259559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7259986Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7260407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7260824Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7260967Z 2025-08-14T21:59:15.7261075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7261471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7261813Z return mod(**inputs) 2025-08-14T21:59:15.7262206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7262593Z outputs = self.model( 2025-08-14T21:59:15.7262966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7263370Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7263727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7264116Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7264516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7264967Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7265150Z 2025-08-14T21:59:15.7265261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7265653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7266000Z return mod(**inputs) 2025-08-14T21:59:15.7266369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7266773Z outputs = self.model( 2025-08-14T21:59:15.7267158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7267568Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7267937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7268330Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7268746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7269195Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7269600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7269971Z return self.act(input) 2025-08-14T21:59:15.7270093Z 2025-08-14T21:59:15.7270212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7270595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7270967Z return mod(**inputs) 2025-08-14T21:59:15.7271350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7271760Z outputs = self.model( 2025-08-14T21:59:15.7272151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7272570Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7272967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7273357Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7273772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7274225Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7274372Z 2025-08-14T21:59:15.7274489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7274866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7275239Z return mod(**inputs) 2025-08-14T21:59:15.7275625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7276142Z outputs = self.model( 2025-08-14T21:59:15.7276549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7276970Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7277366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7277766Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7278184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7278637Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7279074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7279532Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7279717Z 2025-08-14T21:59:15.7279830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7280218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7280563Z return mod(**inputs) 2025-08-14T21:59:15.7280951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7281359Z outputs = self.model( 2025-08-14T21:59:15.7281747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7282149Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7282526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7282916Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7283322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7283760Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7284194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7284599Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7284733Z 2025-08-14T21:59:15.7284836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7285191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7285517Z return mod(**inputs) 2025-08-14T21:59:15.7285869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7286233Z outputs = self.model( 2025-08-14T21:59:15.7286587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7286964Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7287305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7287725Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7288107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7288514Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7288922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7289337Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7289498Z 2025-08-14T21:59:15.7289607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7289962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7290277Z return mod(**inputs) 2025-08-14T21:59:15.7290647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7291025Z outputs = self.model( 2025-08-14T21:59:15.7291373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7291772Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7292119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7292498Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7292900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7293303Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7293702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7294147Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7294353Z 2025-08-14T21:59:15.7294464Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7294843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7295185Z return mod(**inputs) 2025-08-14T21:59:15.7295557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7295959Z outputs = self.model( 2025-08-14T21:59:15.7296335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7296738Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7297079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7297447Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7297829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7298228Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7298629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7299024Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7299169Z 2025-08-14T21:59:15.7299280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7299626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7299949Z return mod(**inputs) 2025-08-14T21:59:15.7300305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7300672Z outputs = self.model( 2025-08-14T21:59:15.7301028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7301435Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7301784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7302136Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7302538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7302990Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7303416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7303840Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7304002Z 2025-08-14T21:59:15.7304107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7304485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7304830Z return mod(**inputs) 2025-08-14T21:59:15.7305235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7305637Z outputs = self.model( 2025-08-14T21:59:15.7306036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7306440Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7306813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7307203Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7307609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7308032Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7308467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7309216Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7309410Z 2025-08-14T21:59:15.7309518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7309912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7310271Z return mod(**inputs) 2025-08-14T21:59:15.7310656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7311054Z outputs = self.model( 2025-08-14T21:59:15.7311436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7311846Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7312206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7312601Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7313009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7313451Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7313872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7314290Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7314436Z 2025-08-14T21:59:15.7314552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7314930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7315301Z return mod(**inputs) 2025-08-14T21:59:15.7315694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7316176Z outputs = self.model( 2025-08-14T21:59:15.7316634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7317049Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7317413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7317785Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7318216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7318673Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7318854Z 2025-08-14T21:59:15.7318970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7319339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7319684Z return mod(**inputs) 2025-08-14T21:59:15.7320092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7320457Z outputs = self.model( 2025-08-14T21:59:15.7320824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7321187Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7321514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7321845Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7322203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7322606Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7322978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7323303Z return self.act(input) 2025-08-14T21:59:15.7323423Z 2025-08-14T21:59:15.7323526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7323877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7324182Z return mod(**inputs) 2025-08-14T21:59:15.7324524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7324885Z outputs = self.model( 2025-08-14T21:59:15.7325229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7325587Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7325927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7326285Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7326666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7327054Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7327194Z 2025-08-14T21:59:15.7327293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7327647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7327947Z return mod(**inputs) 2025-08-14T21:59:15.7328281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7328632Z outputs = self.model( 2025-08-14T21:59:15.7328972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7329331Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7329666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7330018Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7330438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7330842Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7330985Z 2025-08-14T21:59:15.7331088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7331455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7331767Z return mod(**inputs) 2025-08-14T21:59:15.7332104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7332458Z outputs = self.model( 2025-08-14T21:59:15.7332786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7333149Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7333496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7333842Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7334213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7334602Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7334985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7335424Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7335597Z 2025-08-14T21:59:15.7335706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7336080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7336423Z return mod(**inputs) 2025-08-14T21:59:15.7336795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7337189Z outputs = self.model( 2025-08-14T21:59:15.7337539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7337915Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7338247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7338602Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7338986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7339408Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7339831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7340240Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7340387Z 2025-08-14T21:59:15.7340502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7340872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7341407Z return mod(**inputs) 2025-08-14T21:59:15.7341788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7342190Z outputs = self.model( 2025-08-14T21:59:15.7342559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7342961Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7343327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7343702Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7344107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7344562Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7344984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7345414Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7345616Z 2025-08-14T21:59:15.7345731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7346077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7346388Z return mod(**inputs) 2025-08-14T21:59:15.7346722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7347079Z outputs = self.model( 2025-08-14T21:59:15.7347426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7347803Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7348142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7348506Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7348880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7349264Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7349649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7350073Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7350253Z 2025-08-14T21:59:15.7350352Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7350702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7351020Z return mod(**inputs) 2025-08-14T21:59:15.7351370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7351738Z outputs = self.model( 2025-08-14T21:59:15.7352092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7352474Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7352808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7353166Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7353542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7353960Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7354377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7354796Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7354953Z 2025-08-14T21:59:15.7355065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7355443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7355831Z return mod(**inputs) 2025-08-14T21:59:15.7356217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7356628Z outputs = self.model( 2025-08-14T21:59:15.7357005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7357381Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7357715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7358093Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7358454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7358854Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7359248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7359649Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7359800Z 2025-08-14T21:59:15.7359899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7360244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7360557Z return mod(**inputs) 2025-08-14T21:59:15.7360889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7361249Z outputs = self.model( 2025-08-14T21:59:15.7361617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7361994Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7362345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7362710Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7363089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7363487Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7363886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7364317Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7364495Z 2025-08-14T21:59:15.7364603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7364983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7365394Z return mod(**inputs) 2025-08-14T21:59:15.7365750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7366124Z outputs = self.model( 2025-08-14T21:59:15.7366476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7366861Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7367208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7367564Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7367947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7368352Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7368752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7369132Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7369274Z 2025-08-14T21:59:15.7369377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7369732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7370050Z return mod(**inputs) 2025-08-14T21:59:15.7370404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7370772Z outputs = self.model( 2025-08-14T21:59:15.7371130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7371513Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7371862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7372250Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7372653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7373119Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7373359Z 2025-08-14T21:59:15.7373464Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7373820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7374133Z return mod(**inputs) 2025-08-14T21:59:15.7374489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7374863Z outputs = self.model( 2025-08-14T21:59:15.7375240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7375613Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7375990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7376352Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7376721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7377140Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7377523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7377862Z return self.act(input) 2025-08-14T21:59:15.7377978Z 2025-08-14T21:59:15.7378085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7378463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7378789Z return mod(**inputs) 2025-08-14T21:59:15.7379141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7379544Z outputs = self.model( 2025-08-14T21:59:15.7379913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7380299Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7380627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7380978Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7381348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7381718Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7381853Z 2025-08-14T21:59:15.7381954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7382301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7382618Z return mod(**inputs) 2025-08-14T21:59:15.7382953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7383315Z outputs = self.model( 2025-08-14T21:59:15.7383662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7384034Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7384366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7384716Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7385083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7385471Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7385888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7386297Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7386457Z 2025-08-14T21:59:15.7386564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7386923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7387237Z return mod(**inputs) 2025-08-14T21:59:15.7387579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7387934Z outputs = self.model( 2025-08-14T21:59:15.7388278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7388647Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7389015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7389368Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7389774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7390180Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7390597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7390998Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7391140Z 2025-08-14T21:59:15.7391242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7391598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7391914Z return mod(**inputs) 2025-08-14T21:59:15.7392284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7392674Z outputs = self.model( 2025-08-14T21:59:15.7393051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7393439Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7393804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7394184Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7394576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7394999Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7395417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7395950Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7396131Z 2025-08-14T21:59:15.7396242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7396629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7396982Z return mod(**inputs) 2025-08-14T21:59:15.7397365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7397753Z outputs = self.model( 2025-08-14T21:59:15.7398126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7398597Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7398955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7399347Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7399751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7400212Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7400630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7401097Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7401315Z 2025-08-14T21:59:15.7401435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7401810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7402146Z return mod(**inputs) 2025-08-14T21:59:15.7402516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7402920Z outputs = self.model( 2025-08-14T21:59:15.7403303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7403708Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7404089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7404487Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7404880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7405308Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7405733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7406139Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7406295Z 2025-08-14T21:59:15.7406405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7406781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7407134Z return mod(**inputs) 2025-08-14T21:59:15.7407479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7407863Z outputs = self.model( 2025-08-14T21:59:15.7408213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7408574Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7409090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7409444Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7409819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7410216Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7410626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7411026Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7411172Z 2025-08-14T21:59:15.7411279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7411619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7411938Z return mod(**inputs) 2025-08-14T21:59:15.7412286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7412648Z outputs = self.model( 2025-08-14T21:59:15.7412999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7413368Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7413707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7414105Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7414474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7414866Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7415246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7415710Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7415894Z 2025-08-14T21:59:15.7415996Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7416352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7416669Z return mod(**inputs) 2025-08-14T21:59:15.7417023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7417431Z outputs = self.model( 2025-08-14T21:59:15.7417781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7418172Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7418509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7418862Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7419223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7419613Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7420001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7420379Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7420511Z 2025-08-14T21:59:15.7420614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7420963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7421278Z return mod(**inputs) 2025-08-14T21:59:15.7421615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7421979Z outputs = self.model( 2025-08-14T21:59:15.7422322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7422687Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7423014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7423362Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7423729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7424140Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7424308Z 2025-08-14T21:59:15.7424407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7424749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7425063Z return mod(**inputs) 2025-08-14T21:59:15.7425399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7425760Z outputs = self.model( 2025-08-14T21:59:15.7426110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7426488Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7426824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7427182Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7427584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7427997Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7428390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7428737Z return self.act(input) 2025-08-14T21:59:15.7428846Z 2025-08-14T21:59:15.7428954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7429295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7429616Z return mod(**inputs) 2025-08-14T21:59:15.7430003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7430374Z outputs = self.model( 2025-08-14T21:59:15.7430756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7431139Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7431513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7431869Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7432253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7432642Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7432778Z 2025-08-14T21:59:15.7432890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7433244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7433571Z return mod(**inputs) 2025-08-14T21:59:15.7433925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7434299Z outputs = self.model( 2025-08-14T21:59:15.7434660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7435067Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7435440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7435894Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7436312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7436737Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7436872Z 2025-08-14T21:59:15.7436984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7437336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7437662Z return mod(**inputs) 2025-08-14T21:59:15.7438026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7438394Z outputs = self.model( 2025-08-14T21:59:15.7438750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7439131Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7439479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7439831Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7440208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7440671Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7441062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7441508Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7441679Z 2025-08-14T21:59:15.7441782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7442139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7442455Z return mod(**inputs) 2025-08-14T21:59:15.7442833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7443207Z outputs = self.model( 2025-08-14T21:59:15.7443552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7443933Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7444278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7444637Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7445033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7445438Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7445854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7446240Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7446372Z 2025-08-14T21:59:15.7446474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7446828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7447152Z return mod(**inputs) 2025-08-14T21:59:15.7447495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7447870Z outputs = self.model( 2025-08-14T21:59:15.7448225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7448600Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7448939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7449297Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7449677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7450073Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7450459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7450862Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7451022Z 2025-08-14T21:59:15.7451130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7451473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7451794Z return mod(**inputs) 2025-08-14T21:59:15.7452147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7452522Z outputs = self.model( 2025-08-14T21:59:15.7452870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7453248Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7453596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7453948Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7454332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7454735Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7455135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7455591Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7455784Z 2025-08-14T21:59:15.7455888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7456245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7456595Z return mod(**inputs) 2025-08-14T21:59:15.7456970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7457366Z outputs = self.model( 2025-08-14T21:59:15.7457742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7458137Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7458524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7458907Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7459327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7459744Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7460167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7460592Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7460742Z 2025-08-14T21:59:15.7460852Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7461227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7461576Z return mod(**inputs) 2025-08-14T21:59:15.7461928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7462292Z outputs = self.model( 2025-08-14T21:59:15.7462645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7463026Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7463372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7463752Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7464147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7464553Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7464963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7465390Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7465538Z 2025-08-14T21:59:15.7465650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7466004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7466320Z return mod(**inputs) 2025-08-14T21:59:15.7466672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7467045Z outputs = self.model( 2025-08-14T21:59:15.7467389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7467776Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7468139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7468518Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7468885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7469317Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7469713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7470139Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7470343Z 2025-08-14T21:59:15.7470445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7470807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7471159Z return mod(**inputs) 2025-08-14T21:59:15.7471541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7471954Z outputs = self.model( 2025-08-14T21:59:15.7472339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7472774Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7473148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7473552Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7473961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7474387Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7474815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7475233Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7475374Z 2025-08-14T21:59:15.7475490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7475956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7476309Z return mod(**inputs) 2025-08-14T21:59:15.7476701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7477104Z outputs = self.model( 2025-08-14T21:59:15.7477465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7477888Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7478239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7478591Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7478972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7479404Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7479578Z 2025-08-14T21:59:15.7479691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7480040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7480366Z return mod(**inputs) 2025-08-14T21:59:15.7480715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7481083Z outputs = self.model( 2025-08-14T21:59:15.7481439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7481815Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7482162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7482513Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7482892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7483318Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7483734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7484074Z return self.act(input) 2025-08-14T21:59:15.7484189Z 2025-08-14T21:59:15.7484292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7484645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7484998Z return mod(**inputs) 2025-08-14T21:59:15.7485355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7485735Z outputs = self.model( 2025-08-14T21:59:15.7486101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7486512Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7486877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7487244Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7487639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7488031Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7488169Z 2025-08-14T21:59:15.7488280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7488637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7488953Z return mod(**inputs) 2025-08-14T21:59:15.7489303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7489690Z outputs = self.model( 2025-08-14T21:59:15.7490026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7490405Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7490752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7491108Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7491480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7491882Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7492284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7492678Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7492843Z 2025-08-14T21:59:15.7492943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7493295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7493621Z return mod(**inputs) 2025-08-14T21:59:15.7493968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7494340Z outputs = self.model( 2025-08-14T21:59:15.7494692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7495073Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7495411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7495771Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7496152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7496552Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7496943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7497345Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7497474Z 2025-08-14T21:59:15.7497579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7497918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7498231Z return mod(**inputs) 2025-08-14T21:59:15.7498590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7498945Z outputs = self.model( 2025-08-14T21:59:15.7499288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7499651Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7499987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7500335Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7500733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7501134Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7501549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7501963Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7502134Z 2025-08-14T21:59:15.7502238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7502597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7502915Z return mod(**inputs) 2025-08-14T21:59:15.7503271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7503651Z outputs = self.model( 2025-08-14T21:59:15.7504009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7504382Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7504727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7505092Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7505466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7505868Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7506269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7506709Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7506896Z 2025-08-14T21:59:15.7506998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7507356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7507685Z return mod(**inputs) 2025-08-14T21:59:15.7508039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7508404Z outputs = self.model( 2025-08-14T21:59:15.7508900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7509290Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7509629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7509994Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7510373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7510778Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7511249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7511654Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7511804Z 2025-08-14T21:59:15.7511921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7512343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7512709Z return mod(**inputs) 2025-08-14T21:59:15.7513099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7513514Z outputs = self.model( 2025-08-14T21:59:15.7513903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7514324Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7514731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7515134Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7515575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7516074Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7516517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7516955Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7517123Z 2025-08-14T21:59:15.7517233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7517609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7517937Z return mod(**inputs) 2025-08-14T21:59:15.7518287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7518662Z outputs = self.model( 2025-08-14T21:59:15.7519016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7519429Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7519780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7520143Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7520521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7520915Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7521309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7521747Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7521925Z 2025-08-14T21:59:15.7522037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7522383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7522704Z return mod(**inputs) 2025-08-14T21:59:15.7523065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7523456Z outputs = self.model( 2025-08-14T21:59:15.7523834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7524287Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7524644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7525026Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7525427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7525882Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7526297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7526720Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7526897Z 2025-08-14T21:59:15.7527005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7527379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7527717Z return mod(**inputs) 2025-08-14T21:59:15.7528067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7528436Z outputs = self.model( 2025-08-14T21:59:15.7528787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7529175Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7529524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7529904Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7530275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7530693Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7530869Z 2025-08-14T21:59:15.7530972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7531324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7531640Z return mod(**inputs) 2025-08-14T21:59:15.7531991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7532362Z outputs = self.model( 2025-08-14T21:59:15.7532727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7533136Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7533479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7533834Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7534204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7534619Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7535001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7535340Z return self.act(input) 2025-08-14T21:59:15.7535451Z 2025-08-14T21:59:15.7535554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7535930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7536272Z return mod(**inputs) 2025-08-14T21:59:15.7536637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7537030Z outputs = self.model( 2025-08-14T21:59:15.7537403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7537802Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7538138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7538498Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7538866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7539315Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7539501Z 2025-08-14T21:59:15.7539603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7539950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7540264Z return mod(**inputs) 2025-08-14T21:59:15.7540603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7541002Z outputs = self.model( 2025-08-14T21:59:15.7541360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7541727Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7542085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7542444Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7542864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7543267Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7543407Z 2025-08-14T21:59:15.7543509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7543882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7544207Z return mod(**inputs) 2025-08-14T21:59:15.7544552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7544925Z outputs = self.model( 2025-08-14T21:59:15.7545277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7545651Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7545996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7546357Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7546735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7547131Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7547538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7547977Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7548149Z 2025-08-14T21:59:15.7548258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7548631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7548975Z return mod(**inputs) 2025-08-14T21:59:15.7549323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7549684Z outputs = self.model( 2025-08-14T21:59:15.7550037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7550108Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7550334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7550413Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7550658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7550754Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7550992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7551078Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7551081Z 2025-08-14T21:59:15.7551184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7551409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7551474Z return mod(**inputs) 2025-08-14T21:59:15.7551712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7551787Z outputs = self.model( 2025-08-14T21:59:15.7552046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7552122Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7552346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7552424Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7552671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7552769Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7553026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7553146Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7553169Z 2025-08-14T21:59:15.7553270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7553467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7553538Z return mod(**inputs) 2025-08-14T21:59:15.7553779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7553855Z outputs = self.model( 2025-08-14T21:59:15.7554145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7554222Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7554458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7554542Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7554813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7554914Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7555177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7555327Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7555331Z 2025-08-14T21:59:15.7555439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7555652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7555727Z return mod(**inputs) 2025-08-14T21:59:15.7556072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7556159Z outputs = self.model( 2025-08-14T21:59:15.7556422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7556500Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7556745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7556830Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7557102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7557217Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7557490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7557594Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7557630Z 2025-08-14T21:59:15.7557744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7557975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7558058Z return mod(**inputs) 2025-08-14T21:59:15.7558325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7558423Z outputs = self.model( 2025-08-14T21:59:15.7558672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7558745Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7558976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7559054Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7559319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7559426Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7559684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7559790Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7559796Z 2025-08-14T21:59:15.7559896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7560089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7560162Z return mod(**inputs) 2025-08-14T21:59:15.7560402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7560477Z outputs = self.model( 2025-08-14T21:59:15.7560720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7560794Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7561015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7561105Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7561337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7561439Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7561674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7561804Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7561807Z 2025-08-14T21:59:15.7561903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7562094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7562168Z return mod(**inputs) 2025-08-14T21:59:15.7562406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7562478Z outputs = self.model( 2025-08-14T21:59:15.7562716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7562787Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7563003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7563078Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7563308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7563407Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7563642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7563751Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7563754Z 2025-08-14T21:59:15.7563851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7564041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7564135Z return mod(**inputs) 2025-08-14T21:59:15.7564383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7564447Z outputs = self.model( 2025-08-14T21:59:15.7564686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7564754Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7564967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7565057Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7565286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7565434Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7565438Z 2025-08-14T21:59:15.7565536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7565731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7565794Z return mod(**inputs) 2025-08-14T21:59:15.7566031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7566102Z outputs = self.model( 2025-08-14T21:59:15.7566328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7566395Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7566607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7566680Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7566914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7567027Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7567223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7567297Z return self.act(input) 2025-08-14T21:59:15.7567300Z 2025-08-14T21:59:15.7567401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7567591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7567654Z return mod(**inputs) 2025-08-14T21:59:15.7567889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7567961Z outputs = self.model( 2025-08-14T21:59:15.7568195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7568265Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7568481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7568557Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7568798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7568878Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7568881Z 2025-08-14T21:59:15.7568978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7569185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7569269Z return mod(**inputs) 2025-08-14T21:59:15.7569497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7569568Z outputs = self.model( 2025-08-14T21:59:15.7569797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7569907Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7570115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7570191Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7570432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7570524Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7570783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7570891Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7570894Z 2025-08-14T21:59:15.7571006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7571200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7571265Z return mod(**inputs) 2025-08-14T21:59:15.7571493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7571564Z outputs = self.model( 2025-08-14T21:59:15.7571791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7571865Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7572069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7572144Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7572380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7572473Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7572701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7572786Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7572789Z 2025-08-14T21:59:15.7572883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7573074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7573135Z return mod(**inputs) 2025-08-14T21:59:15.7573363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7573436Z outputs = self.model( 2025-08-14T21:59:15.7573669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7573744Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7573947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7574020Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7574255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7574346Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7574579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7574692Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7574695Z 2025-08-14T21:59:15.7574793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7575010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7575074Z return mod(**inputs) 2025-08-14T21:59:15.7575310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7575405Z outputs = self.model( 2025-08-14T21:59:15.7575642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7575719Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7575931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7576005Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7576246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7576358Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7576593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7576751Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7576755Z 2025-08-14T21:59:15.7576856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7577052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7577114Z return mod(**inputs) 2025-08-14T21:59:15.7577348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7577431Z outputs = self.model( 2025-08-14T21:59:15.7577657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7577725Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7577938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7578011Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7578243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7578334Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7578560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7578651Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7578655Z 2025-08-14T21:59:15.7578751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7578947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7579009Z return mod(**inputs) 2025-08-14T21:59:15.7579244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7579317Z outputs = self.model( 2025-08-14T21:59:15.7579551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7579619Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7579836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7579910Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7580150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7580244Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7580475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7580575Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7580596Z 2025-08-14T21:59:15.7580693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7580889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7580953Z return mod(**inputs) 2025-08-14T21:59:15.7581184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7581275Z outputs = self.model( 2025-08-14T21:59:15.7581514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7581582Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7581804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7581883Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7582145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7582241Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7582516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7582649Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7582654Z 2025-08-14T21:59:15.7582752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7582947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7583010Z return mod(**inputs) 2025-08-14T21:59:15.7583248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7583322Z outputs = self.model( 2025-08-14T21:59:15.7583559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7583631Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7583853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7583928Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7584175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7584269Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7584508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7584594Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7584598Z 2025-08-14T21:59:15.7584695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7584890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7584962Z return mod(**inputs) 2025-08-14T21:59:15.7585200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7585274Z outputs = self.model( 2025-08-14T21:59:15.7585510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7585581Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7585805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7585879Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7586126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7586240Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7586244Z 2025-08-14T21:59:15.7586344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7586560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7586622Z return mod(**inputs) 2025-08-14T21:59:15.7586866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7586956Z outputs = self.model( 2025-08-14T21:59:15.7587191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7587267Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7587476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7587550Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7587796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7587940Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7588145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7588238Z return self.act(input) 2025-08-14T21:59:15.7588242Z 2025-08-14T21:59:15.7588744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7588942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7589005Z return mod(**inputs) 2025-08-14T21:59:15.7589243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7589317Z outputs = self.model( 2025-08-14T21:59:15.7589553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7589630Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7589845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7589920Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7590167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7590250Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7590254Z 2025-08-14T21:59:15.7590361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7590553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7590617Z return mod(**inputs) 2025-08-14T21:59:15.7590865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7590932Z outputs = self.model( 2025-08-14T21:59:15.7591179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7591259Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7591478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7591563Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7591808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7591886Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7591890Z 2025-08-14T21:59:15.7591997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7592191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7592256Z return mod(**inputs) 2025-08-14T21:59:15.7592513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7592608Z outputs = self.model( 2025-08-14T21:59:15.7592866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7592941Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7593165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7594184Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7594439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7594546Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7594799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7594915Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7594919Z 2025-08-14T21:59:15.7595055Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7595266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7595332Z return mod(**inputs) 2025-08-14T21:59:15.7595610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7595685Z outputs = self.model( 2025-08-14T21:59:15.7596030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7596113Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7596343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7596435Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7596699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7596812Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7597096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7597180Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7597184Z 2025-08-14T21:59:15.7597304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7597514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7597584Z return mod(**inputs) 2025-08-14T21:59:15.7597852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7597923Z outputs = self.model( 2025-08-14T21:59:15.7598193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7598269Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7598506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7598598Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7598863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7598965Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7599231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7599347Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7599351Z 2025-08-14T21:59:15.7599465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7599674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7599742Z return mod(**inputs) 2025-08-14T21:59:15.7600011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7600107Z outputs = self.model( 2025-08-14T21:59:15.7600368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7600443Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7600691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7600779Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7601038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7601145Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7601412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7601573Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7601577Z 2025-08-14T21:59:15.7601694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7601920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7601992Z return mod(**inputs) 2025-08-14T21:59:15.7602254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7602323Z outputs = self.model( 2025-08-14T21:59:15.7602577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7602663Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7602890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7602979Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7603235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7603337Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7603599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7603691Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7603695Z 2025-08-14T21:59:15.7603806Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7604011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7604080Z return mod(**inputs) 2025-08-14T21:59:15.7604340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7604411Z outputs = self.model( 2025-08-14T21:59:15.7604669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7604750Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7604968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7605054Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7605309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7605408Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7605671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7605778Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7605781Z 2025-08-14T21:59:15.7605891Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7606088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7606173Z return mod(**inputs) 2025-08-14T21:59:15.7606427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7606494Z outputs = self.model( 2025-08-14T21:59:15.7606739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7606839Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7607057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7607139Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7607382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7607476Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7607744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7607873Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7607877Z 2025-08-14T21:59:15.7607998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7608197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7608262Z return mod(**inputs) 2025-08-14T21:59:15.7608512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7608578Z outputs = self.model( 2025-08-14T21:59:15.7608970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7609056Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7609275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7609361Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7609603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7609699Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7609948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7610029Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7610032Z 2025-08-14T21:59:15.7610132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7610336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7610401Z return mod(**inputs) 2025-08-14T21:59:15.7610650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7610720Z outputs = self.model( 2025-08-14T21:59:15.7610961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7611041Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7611259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7611347Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7611587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7611703Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7611707Z 2025-08-14T21:59:15.7611815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7612013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7612078Z return mod(**inputs) 2025-08-14T21:59:15.7612375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7612443Z outputs = self.model( 2025-08-14T21:59:15.7612689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7612800Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7613018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7613106Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7613349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7613468Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7613689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7613787Z return self.act(input) 2025-08-14T21:59:15.7613792Z 2025-08-14T21:59:15.7613902Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7614121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7614188Z return mod(**inputs) 2025-08-14T21:59:15.7614440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7614508Z outputs = self.model( 2025-08-14T21:59:15.7614755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7614827Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7615047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7615131Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7615368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7615447Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7615458Z 2025-08-14T21:59:15.7615560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7615762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7615841Z return mod(**inputs) 2025-08-14T21:59:15.7616094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7616164Z outputs = self.model( 2025-08-14T21:59:15.7616426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7616501Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7616730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7616821Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7617073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7617183Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7617438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7617555Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7617559Z 2025-08-14T21:59:15.7617672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7617875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7617949Z return mod(**inputs) 2025-08-14T21:59:15.7618213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7618302Z outputs = self.model( 2025-08-14T21:59:15.7618550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7618622Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7618841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7618952Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7619212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7619324Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7619584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7619664Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7619668Z 2025-08-14T21:59:15.7619797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7619987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7620059Z return mod(**inputs) 2025-08-14T21:59:15.7620310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7620377Z outputs = self.model( 2025-08-14T21:59:15.7620616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7620685Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7620893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7620979Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7621217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7621319Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7621553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7621662Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7621667Z 2025-08-14T21:59:15.7621769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7621958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7622028Z return mod(**inputs) 2025-08-14T21:59:15.7622261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7622325Z outputs = self.model( 2025-08-14T21:59:15.7622564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7622634Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7622844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7622926Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7623162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7623263Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7623504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7623636Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7623640Z 2025-08-14T21:59:15.7623746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7623941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7624012Z return mod(**inputs) 2025-08-14T21:59:15.7624277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7624344Z outputs = self.model( 2025-08-14T21:59:15.7624602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7624689Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7624899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7624981Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7625215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7625313Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7625549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7625650Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7625654Z 2025-08-14T21:59:15.7625759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7625966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7626031Z return mod(**inputs) 2025-08-14T21:59:15.7626277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7626342Z outputs = self.model( 2025-08-14T21:59:15.7626583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7626654Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7626864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7626946Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7627180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7627283Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7627516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7627609Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7627612Z 2025-08-14T21:59:15.7627718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7627904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7627967Z return mod(**inputs) 2025-08-14T21:59:15.7628207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7628271Z outputs = self.model( 2025-08-14T21:59:15.7628514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7628586Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7628802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7628887Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7629123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7629215Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7629452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7629574Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7629578Z 2025-08-14T21:59:15.7629682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7629873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7629983Z return mod(**inputs) 2025-08-14T21:59:15.7630228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7630293Z outputs = self.model( 2025-08-14T21:59:15.7630553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7630624Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7630842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7630927Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7631167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7631262Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7631526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7631608Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7631612Z 2025-08-14T21:59:15.7631733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7631932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7631996Z return mod(**inputs) 2025-08-14T21:59:15.7632241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7632308Z outputs = self.model( 2025-08-14T21:59:15.7632570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7632647Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7632875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7632966Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7633220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7633343Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7633348Z 2025-08-14T21:59:15.7633463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7633668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7633743Z return mod(**inputs) 2025-08-14T21:59:15.7633999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7634073Z outputs = self.model( 2025-08-14T21:59:15.7634337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7634416Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7634641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7634733Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7634986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7635115Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7635334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7635406Z return self.act(input) 2025-08-14T21:59:15.7635410Z 2025-08-14T21:59:15.7635523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7635728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7635886Z return mod(**inputs) 2025-08-14T21:59:15.7636191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7636265Z outputs = self.model( 2025-08-14T21:59:15.7636535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7636635Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7636890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7636982Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7637255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7637348Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7637353Z 2025-08-14T21:59:15.7637459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7637704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7637787Z return mod(**inputs) 2025-08-14T21:59:15.7638071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7638146Z outputs = self.model( 2025-08-14T21:59:15.7638388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7638459Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7638684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7638760Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7639000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7639087Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7639092Z 2025-08-14T21:59:15.7639196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7639397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7639464Z return mod(**inputs) 2025-08-14T21:59:15.7639710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7639787Z outputs = self.model( 2025-08-14T21:59:15.7640033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7640107Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7640332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7640409Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7640660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7640760Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7641001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7641122Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7641127Z 2025-08-14T21:59:15.7641229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7641432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7641498Z return mod(**inputs) 2025-08-14T21:59:15.7641741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7641815Z outputs = self.model( 2025-08-14T21:59:15.7642056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7642149Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7642370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7642450Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7642701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7642815Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7643060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7643147Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7643151Z 2025-08-14T21:59:15.7643250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7643454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7643517Z return mod(**inputs) 2025-08-14T21:59:15.7643776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7643851Z outputs = self.model( 2025-08-14T21:59:15.7644110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7644186Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7644429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7644511Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7644782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7644892Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7645136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7645254Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7645258Z 2025-08-14T21:59:15.7645357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7645554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7645628Z return mod(**inputs) 2025-08-14T21:59:15.7645873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7645948Z outputs = self.model( 2025-08-14T21:59:15.7646192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7646262Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7646487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7646566Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7646814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7646909Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7647151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7647295Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7647298Z 2025-08-14T21:59:15.7647400Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7647611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7647687Z return mod(**inputs) 2025-08-14T21:59:15.7647944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7648022Z outputs = self.model( 2025-08-14T21:59:15.7648298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7648372Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7648611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7648722Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7648963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7649067Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7649312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7649405Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7649408Z 2025-08-14T21:59:15.7649507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7649718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7649792Z return mod(**inputs) 2025-08-14T21:59:15.7650051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7650126Z outputs = self.model( 2025-08-14T21:59:15.7650370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7650441Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7650661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7650739Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7650989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7651089Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7651323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7651423Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7651427Z 2025-08-14T21:59:15.7651524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7651712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7651782Z return mod(**inputs) 2025-08-14T21:59:15.7652013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7652085Z outputs = self.model( 2025-08-14T21:59:15.7652317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7652386Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7652603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7652680Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7652915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7653016Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7653248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7653376Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7653379Z 2025-08-14T21:59:15.7653475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7653663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7653734Z return mod(**inputs) 2025-08-14T21:59:15.7653969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7654066Z outputs = self.model( 2025-08-14T21:59:15.7654312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7654385Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7654632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7654709Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7654951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7655054Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7655294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7655382Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7655405Z 2025-08-14T21:59:15.7655506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7655697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7655801Z return mod(**inputs) 2025-08-14T21:59:15.7656055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7656121Z outputs = self.model( 2025-08-14T21:59:15.7656364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7656434Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7656649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7656726Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7656963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7657087Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7657091Z 2025-08-14T21:59:15.7657191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7657390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7657454Z return mod(**inputs) 2025-08-14T21:59:15.7657688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7657769Z outputs = self.model( 2025-08-14T21:59:15.7657996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7658066Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7658278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7658356Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7658598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7658713Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7658914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7658993Z return self.act(input) 2025-08-14T21:59:15.7658997Z 2025-08-14T21:59:15.7659093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7659304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7659366Z return mod(**inputs) 2025-08-14T21:59:15.7659601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7659671Z outputs = self.model( 2025-08-14T21:59:15.7659928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7659998Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7660214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7660313Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7660552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7660628Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7660632Z 2025-08-14T21:59:15.7660739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7660929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7660992Z return mod(**inputs) 2025-08-14T21:59:15.7661241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7661313Z outputs = self.model( 2025-08-14T21:59:15.7661561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7661638Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7661847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7661920Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7662162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7662255Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7662494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7662610Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7662616Z 2025-08-14T21:59:15.7662713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7662903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7662966Z return mod(**inputs) 2025-08-14T21:59:15.7663192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7663265Z outputs = self.model( 2025-08-14T21:59:15.7663490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7663565Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7663770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7663846Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7664088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7664184Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7664419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7664504Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7664509Z 2025-08-14T21:59:15.7664604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7664796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7664861Z return mod(**inputs) 2025-08-14T21:59:15.7665093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7665165Z outputs = self.model( 2025-08-14T21:59:15.7665398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7665510Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7665712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7665786Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7666019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7666130Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7666357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7666466Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7666469Z 2025-08-14T21:59:15.7666563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7666755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7666837Z return mod(**inputs) 2025-08-14T21:59:15.7667075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7667163Z outputs = self.model( 2025-08-14T21:59:15.7667408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7667483Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7667686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7667761Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7668002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7668095Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7668329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7668466Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7668469Z 2025-08-14T21:59:15.7668568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7668764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7668829Z return mod(**inputs) 2025-08-14T21:59:15.7669064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7669137Z outputs = self.model( 2025-08-14T21:59:15.7669373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7669444Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7669662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7669739Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7669987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7670080Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7670314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7670407Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7670411Z 2025-08-14T21:59:15.7670508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7670705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7670770Z return mod(**inputs) 2025-08-14T21:59:15.7671007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7671079Z outputs = self.model( 2025-08-14T21:59:15.7671335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7671405Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7671631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7671727Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7671973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7672068Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7672307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7672409Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7672413Z 2025-08-14T21:59:15.7672509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7672733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7672799Z return mod(**inputs) 2025-08-14T21:59:15.7673064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7673140Z outputs = self.model( 2025-08-14T21:59:15.7673394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7673469Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7673708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7673789Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7674055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7674157Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7674416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7674559Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7674563Z 2025-08-14T21:59:15.7674668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7674884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7674953Z return mod(**inputs) 2025-08-14T21:59:15.7675210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7675288Z outputs = self.model( 2025-08-14T21:59:15.7675547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7675622Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7675952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7676042Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7676309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7676413Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7676669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7676764Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7676767Z 2025-08-14T21:59:15.7676872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7677081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7677156Z return mod(**inputs) 2025-08-14T21:59:15.7677404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7677505Z outputs = self.model( 2025-08-14T21:59:15.7677747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7677824Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7678087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7678168Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7678434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7678557Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7678560Z 2025-08-14T21:59:15.7678666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7678910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7678983Z return mod(**inputs) 2025-08-14T21:59:15.7679244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7679339Z outputs = self.model( 2025-08-14T21:59:15.7679596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7679678Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7679905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7679985Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7680254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7680368Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7680583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7680655Z return self.act(input) 2025-08-14T21:59:15.7680658Z 2025-08-14T21:59:15.7680761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7680960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7681026Z return mod(**inputs) 2025-08-14T21:59:15.7681266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7681338Z outputs = self.model( 2025-08-14T21:59:15.7681586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7681668Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7681896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7681980Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7682243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7682331Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7682335Z 2025-08-14T21:59:15.7682442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7682656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7682724Z return mod(**inputs) 2025-08-14T21:59:15.7682985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7683056Z outputs = self.model( 2025-08-14T21:59:15.7683311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7683395Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7683647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7683728Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7683993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7684096Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7684100Z 2025-08-14T21:59:15.7684218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7684425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7684493Z return mod(**inputs) 2025-08-14T21:59:15.7684767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7684834Z outputs = self.model( 2025-08-14T21:59:15.7685097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7685170Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7685404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7685491Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7685733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7685828Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7686091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7686206Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7686210Z 2025-08-14T21:59:15.7686321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7686530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7686600Z return mod(**inputs) 2025-08-14T21:59:15.7686862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7686934Z outputs = self.model( 2025-08-14T21:59:15.7687195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7687272Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7687498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7687587Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7687846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7687948Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7688216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7688302Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7688305Z 2025-08-14T21:59:15.7688420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7688626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7688696Z return mod(**inputs) 2025-08-14T21:59:15.7688960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7689031Z outputs = self.model( 2025-08-14T21:59:15.7689284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7689368Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7689593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7689703Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7689956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7690056Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7690317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7690452Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7690456Z 2025-08-14T21:59:15.7690566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7690771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7690839Z return mod(**inputs) 2025-08-14T21:59:15.7691098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7691188Z outputs = self.model( 2025-08-14T21:59:15.7691447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7691531Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7691779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7691872Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7692125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7692226Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7692487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7692626Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7692630Z 2025-08-14T21:59:15.7692746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7692953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7693022Z return mod(**inputs) 2025-08-14T21:59:15.7693285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7693359Z outputs = self.model( 2025-08-14T21:59:15.7693609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7693692Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7693917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7694005Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7694258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7694361Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7694623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7694716Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7694720Z 2025-08-14T21:59:15.7694838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7695046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7695116Z return mod(**inputs) 2025-08-14T21:59:15.7695374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7695443Z outputs = self.model( 2025-08-14T21:59:15.7695694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7695776Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7696038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7696120Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7696363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7696485Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7696745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7696843Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7696847Z 2025-08-14T21:59:15.7696958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7697160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7697228Z return mod(**inputs) 2025-08-14T21:59:15.7697504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7697577Z outputs = self.model( 2025-08-14T21:59:15.7697848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7697932Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7698163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7698252Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7698514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7698609Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7698859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7698987Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7698991Z 2025-08-14T21:59:15.7699090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7699293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7699357Z return mod(**inputs) 2025-08-14T21:59:15.7699607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7699676Z outputs = self.model( 2025-08-14T21:59:15.7699936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7700020Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7700249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7700339Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7700598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7700701Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7700970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7701056Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7701060Z 2025-08-14T21:59:15.7701166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7701379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7701447Z return mod(**inputs) 2025-08-14T21:59:15.7701718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7701784Z outputs = self.model( 2025-08-14T21:59:15.7702029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7702130Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7702349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7702427Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7702709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7702824Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7702828Z 2025-08-14T21:59:15.7702934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7703129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7703194Z return mod(**inputs) 2025-08-14T21:59:15.7703457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7703525Z outputs = self.model( 2025-08-14T21:59:15.7703774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7703860Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7704075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7704162Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7704403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7704518Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7704736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7704804Z return self.act(input) 2025-08-14T21:59:15.7704808Z 2025-08-14T21:59:15.7704919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7705114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7705178Z return mod(**inputs) 2025-08-14T21:59:15.7705425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7705491Z outputs = self.model( 2025-08-14T21:59:15.7705740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7705810Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7706027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7706113Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7706354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7706439Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7706443Z 2025-08-14T21:59:15.7706552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7706748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7706822Z return mod(**inputs) 2025-08-14T21:59:15.7707067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7707133Z outputs = self.model( 2025-08-14T21:59:15.7707382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7707452Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7707668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7707753Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7708018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7708123Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7708366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7708494Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7708498Z 2025-08-14T21:59:15.7708604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7708984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7709064Z return mod(**inputs) 2025-08-14T21:59:15.7709300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7709367Z outputs = self.model( 2025-08-14T21:59:15.7709655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7709731Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7709974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7710060Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7710307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7710411Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7710658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7710738Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7710741Z 2025-08-14T21:59:15.7710857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7711053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7711128Z return mod(**inputs) 2025-08-14T21:59:15.7711371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7711439Z outputs = self.model( 2025-08-14T21:59:15.7711687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7711761Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7711975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7712064Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7712305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7712411Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7712655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7712768Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7712771Z 2025-08-14T21:59:15.7712880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7713075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7713142Z return mod(**inputs) 2025-08-14T21:59:15.7713402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7713472Z outputs = self.model( 2025-08-14T21:59:15.7713736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7713813Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7714044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7714164Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7714422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7714532Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7714817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7714958Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7714961Z 2025-08-14T21:59:15.7715075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7715281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7715349Z return mod(**inputs) 2025-08-14T21:59:15.7715632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7715705Z outputs = self.model( 2025-08-14T21:59:15.7716018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7716125Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7716364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7716458Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7716726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7716839Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7717102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7717195Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7717199Z 2025-08-14T21:59:15.7717319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7717531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7717608Z return mod(**inputs) 2025-08-14T21:59:15.7717848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7717915Z outputs = self.model( 2025-08-14T21:59:15.7718167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7718236Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7718446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7718528Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7718760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7718855Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7719098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7719189Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7719194Z 2025-08-14T21:59:15.7719298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7719487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7719551Z return mod(**inputs) 2025-08-14T21:59:15.7719794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7719859Z outputs = self.model( 2025-08-14T21:59:15.7720100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7720172Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7720410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7720492Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7720728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7720841Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7721092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7721215Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7721219Z 2025-08-14T21:59:15.7721326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7721516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7721580Z return mod(**inputs) 2025-08-14T21:59:15.7721847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7721916Z outputs = self.model( 2025-08-14T21:59:15.7722187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7722260Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7722470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7722554Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7722791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7722881Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7723122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7723203Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7723206Z 2025-08-14T21:59:15.7723313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7723503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7723570Z return mod(**inputs) 2025-08-14T21:59:15.7723818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7723884Z outputs = self.model( 2025-08-14T21:59:15.7724123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7724203Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7724417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7724502Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7724741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7724860Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7724864Z 2025-08-14T21:59:15.7724974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7725174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7725249Z return mod(**inputs) 2025-08-14T21:59:15.7725493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7725564Z outputs = self.model( 2025-08-14T21:59:15.7725816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7725889Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7726111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7726225Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7726468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7726588Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7726816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7726885Z return self.act(input) 2025-08-14T21:59:15.7726889Z 2025-08-14T21:59:15.7726997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7727190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7727260Z return mod(**inputs) 2025-08-14T21:59:15.7727500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7727585Z outputs = self.model( 2025-08-14T21:59:15.7727839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7727928Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7728149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7728237Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7728478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7728566Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7728570Z 2025-08-14T21:59:15.7728670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7728866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7728941Z return mod(**inputs) 2025-08-14T21:59:15.7729183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7729249Z outputs = self.model( 2025-08-14T21:59:15.7729497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7729570Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7729792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7729868Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7730107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7730196Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7730199Z 2025-08-14T21:59:15.7730301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7730505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7730571Z return mod(**inputs) 2025-08-14T21:59:15.7730812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7730885Z outputs = self.model( 2025-08-14T21:59:15.7731126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7731196Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7731417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7731493Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7731741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7731837Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7732106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7732223Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7732228Z 2025-08-14T21:59:15.7732327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7732548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7732613Z return mod(**inputs) 2025-08-14T21:59:15.7732853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7732928Z outputs = self.model( 2025-08-14T21:59:15.7733167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7733238Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7733493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7733573Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7733836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7733934Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7734177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7734263Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7734267Z 2025-08-14T21:59:15.7734366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7734562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7734635Z return mod(**inputs) 2025-08-14T21:59:15.7734877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7734955Z outputs = self.model( 2025-08-14T21:59:15.7735195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7735271Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7735495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7735575Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7735813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7735917Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7736156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7736275Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7736279Z 2025-08-14T21:59:15.7736380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7736573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7736648Z return mod(**inputs) 2025-08-14T21:59:15.7736889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7736965Z outputs = self.model( 2025-08-14T21:59:15.7737206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7737276Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7737497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7737573Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7737814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7737936Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7738178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7738317Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7738337Z 2025-08-14T21:59:15.7738439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7738634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7738711Z return mod(**inputs) 2025-08-14T21:59:15.7738951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7739028Z outputs = self.model( 2025-08-14T21:59:15.7739274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7739368Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7739584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7739677Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7739910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7740011Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7740249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7740341Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7740344Z 2025-08-14T21:59:15.7740444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7740636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7740714Z return mod(**inputs) 2025-08-14T21:59:15.7740949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7741026Z outputs = self.model( 2025-08-14T21:59:15.7741264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7741338Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7741556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7741636Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7741881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7741989Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7742233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7742338Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7742342Z 2025-08-14T21:59:15.7742444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7742642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7742720Z return mod(**inputs) 2025-08-14T21:59:15.7742963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7743034Z outputs = self.model( 2025-08-14T21:59:15.7743285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7743361Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7743588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7743671Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7743933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7744038Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7744280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7744432Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7744436Z 2025-08-14T21:59:15.7744536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7744732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7744809Z return mod(**inputs) 2025-08-14T21:59:15.7745049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7745119Z outputs = self.model( 2025-08-14T21:59:15.7745387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7745459Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7745709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7745790Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7746032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7746135Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7746377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7746465Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7746468Z 2025-08-14T21:59:15.7746570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7746767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7746838Z return mod(**inputs) 2025-08-14T21:59:15.7747092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7747161Z outputs = self.model( 2025-08-14T21:59:15.7747412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7747482Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7747704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7747782Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7748023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7748148Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7748153Z 2025-08-14T21:59:15.7748253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7748457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7748522Z return mod(**inputs) 2025-08-14T21:59:15.7748769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7748842Z outputs = self.model( 2025-08-14T21:59:15.7749079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7749148Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7749372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7749448Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7749697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7749833Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7750044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7750126Z return self.act(input) 2025-08-14T21:59:15.7750148Z 2025-08-14T21:59:15.7750249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7750443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7750515Z return mod(**inputs) 2025-08-14T21:59:15.7750754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7750826Z outputs = self.model( 2025-08-14T21:59:15.7751063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7751153Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7751378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7751472Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7751721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7751803Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7751806Z 2025-08-14T21:59:15.7751907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7752105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7752170Z return mod(**inputs) 2025-08-14T21:59:15.7752410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7752485Z outputs = self.model( 2025-08-14T21:59:15.7752727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7752804Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7753020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7753098Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7753348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7753445Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7753692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7753814Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7753817Z 2025-08-14T21:59:15.7753922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7754138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7754206Z return mod(**inputs) 2025-08-14T21:59:15.7754461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7754538Z outputs = self.model( 2025-08-14T21:59:15.7754795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7754877Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7755104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7755188Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7755448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7755552Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7755909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7756008Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7756014Z 2025-08-14T21:59:15.7756120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7756365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7756433Z return mod(**inputs) 2025-08-14T21:59:15.7756692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7756775Z outputs = self.model( 2025-08-14T21:59:15.7757037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7757115Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7757375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7757464Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7757749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7757853Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7758128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7758251Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7758255Z 2025-08-14T21:59:15.7758360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7758575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7758645Z return mod(**inputs) 2025-08-14T21:59:15.7758899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7758979Z outputs = self.model( 2025-08-14T21:59:15.7759230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7759305Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7759541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7759622Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7759882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7759984Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7760237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7760385Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7760392Z 2025-08-14T21:59:15.7760498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7760716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7760785Z return mod(**inputs) 2025-08-14T21:59:15.7761037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7761116Z outputs = self.model( 2025-08-14T21:59:15.7761372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7761446Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7761678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7761760Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7762019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7762138Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7762392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7762487Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7762511Z 2025-08-14T21:59:15.7762617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7762840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7762907Z return mod(**inputs) 2025-08-14T21:59:15.7763166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7763242Z outputs = self.model( 2025-08-14T21:59:15.7763520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7763598Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7763833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7763935Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7764210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7764313Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7764580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7764687Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7764690Z 2025-08-14T21:59:15.7764802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7765002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7765068Z return mod(**inputs) 2025-08-14T21:59:15.7765303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7765378Z outputs = self.model( 2025-08-14T21:59:15.7765613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7765685Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7765905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7765979Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7766223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7766315Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7766551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7766680Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7766682Z 2025-08-14T21:59:15.7766781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7766972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7767043Z return mod(**inputs) 2025-08-14T21:59:15.7767282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7767352Z outputs = self.model( 2025-08-14T21:59:15.7767588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7767658Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7767875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7767969Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7768208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7768299Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7768535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7768639Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7768642Z 2025-08-14T21:59:15.7768740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7768929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7768999Z return mod(**inputs) 2025-08-14T21:59:15.7769237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7769313Z outputs = self.model( 2025-08-14T21:59:15.7769580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7769652Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7769901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7769980Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7770221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7770343Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7770346Z 2025-08-14T21:59:15.7770448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7770653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7770721Z return mod(**inputs) 2025-08-14T21:59:15.7770968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7771052Z outputs = self.model( 2025-08-14T21:59:15.7771288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7771364Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7771575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7771649Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7771889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7772001Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7772205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7772281Z return self.act(input) 2025-08-14T21:59:15.7772287Z 2025-08-14T21:59:15.7772387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7772585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7772649Z return mod(**inputs) 2025-08-14T21:59:15.7772882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7772958Z outputs = self.model( 2025-08-14T21:59:15.7773191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7773267Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7773474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7773549Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7773788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7773890Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7773894Z 2025-08-14T21:59:15.7773993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7774190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7774283Z return mod(**inputs) 2025-08-14T21:59:15.7774548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7774617Z outputs = self.model( 2025-08-14T21:59:15.7774888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7774967Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7775189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7775285Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7775539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7775667Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7775672Z 2025-08-14T21:59:15.7775800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7776007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7776075Z return mod(**inputs) 2025-08-14T21:59:15.7776334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7776405Z outputs = self.model( 2025-08-14T21:59:15.7776662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7776736Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7776967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7777054Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7777307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7777412Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7777673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7777783Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7777786Z 2025-08-14T21:59:15.7777896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7778096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7778159Z return mod(**inputs) 2025-08-14T21:59:15.7778401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7778469Z outputs = self.model( 2025-08-14T21:59:15.7778705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7778783Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7778998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7779084Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7779323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7779419Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7779663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7779744Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7779770Z 2025-08-14T21:59:15.7779877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7780072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7780138Z return mod(**inputs) 2025-08-14T21:59:15.7780386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7780472Z outputs = self.model( 2025-08-14T21:59:15.7780721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7780798Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7781008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7781090Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7781340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7781439Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7781699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7781810Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7781813Z 2025-08-14T21:59:15.7781920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7782112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7782176Z return mod(**inputs) 2025-08-14T21:59:15.7782422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7782491Z outputs = self.model( 2025-08-14T21:59:15.7782730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7782809Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7783025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7783111Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7783351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7783443Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7783690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7783822Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7783825Z 2025-08-14T21:59:15.7783929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7784125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7784192Z return mod(**inputs) 2025-08-14T21:59:15.7784436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7784504Z outputs = self.model( 2025-08-14T21:59:15.7784741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7784821Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7785034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7785118Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7785356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7785452Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7785699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7785811Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7785814Z 2025-08-14T21:59:15.7785913Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7786115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7786201Z return mod(**inputs) 2025-08-14T21:59:15.7786451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7786517Z outputs = self.model( 2025-08-14T21:59:15.7786758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7786839Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7787051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7787154Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7787399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7787510Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7787762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7787858Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7787862Z 2025-08-14T21:59:15.7787961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7788166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7788232Z return mod(**inputs) 2025-08-14T21:59:15.7788480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7788551Z outputs = self.model( 2025-08-14T21:59:15.7788795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7788875Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7789092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7789170Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7789422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7789516Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7789764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7789893Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7789897Z 2025-08-14T21:59:15.7790002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7790223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7790290Z return mod(**inputs) 2025-08-14T21:59:15.7790556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7790628Z outputs = self.model( 2025-08-14T21:59:15.7790888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7790971Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7791200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7791283Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7791546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7791667Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7791929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7792015Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7792019Z 2025-08-14T21:59:15.7792155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7792367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7792435Z return mod(**inputs) 2025-08-14T21:59:15.7792697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7792767Z outputs = self.model( 2025-08-14T21:59:15.7793019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7793100Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7793363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7793449Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7793738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7793869Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7793873Z 2025-08-14T21:59:15.7793988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7794199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7794269Z return mod(**inputs) 2025-08-14T21:59:15.7794538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7794610Z outputs = self.model( 2025-08-14T21:59:15.7794879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7794959Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7795194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7795286Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7795547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7795671Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7796124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7796207Z return self.act(input) 2025-08-14T21:59:15.7796211Z 2025-08-14T21:59:15.7796330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7796549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7796621Z return mod(**inputs) 2025-08-14T21:59:15.7796899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7796970Z outputs = self.model( 2025-08-14T21:59:15.7797211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7797291Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7797513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7797605Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7797867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7797953Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7797957Z 2025-08-14T21:59:15.7798077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7798316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7798393Z return mod(**inputs) 2025-08-14T21:59:15.7798655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7798752Z outputs = self.model( 2025-08-14T21:59:15.7799024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7799101Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7799338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7799429Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7799697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7799832Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7800093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7800231Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7800235Z 2025-08-14T21:59:15.7800353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7800564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7800636Z return mod(**inputs) 2025-08-14T21:59:15.7800904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7800977Z outputs = self.model( 2025-08-14T21:59:15.7801245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7801323Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7801562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7801654Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7801922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7802037Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7802299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7802387Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7802390Z 2025-08-14T21:59:15.7802506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7802720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7802791Z return mod(**inputs) 2025-08-14T21:59:15.7803063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7803140Z outputs = self.model( 2025-08-14T21:59:15.7803413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7803492Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7803740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7803829Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7804094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7804204Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7804465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7804610Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7804613Z 2025-08-14T21:59:15.7804726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7804975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7805053Z return mod(**inputs) 2025-08-14T21:59:15.7805325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7805393Z outputs = self.model( 2025-08-14T21:59:15.7805641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7805714Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7805928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7806012Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7806294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7806391Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7806655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7806791Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7806795Z 2025-08-14T21:59:15.7806899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7807100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7807165Z return mod(**inputs) 2025-08-14T21:59:15.7807418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7807485Z outputs = self.model( 2025-08-14T21:59:15.7807738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7807809Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7808029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7808124Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7808360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7808454Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7808844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7808934Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7808938Z 2025-08-14T21:59:15.7809043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7809234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7809301Z return mod(**inputs) 2025-08-14T21:59:15.7809545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7809612Z outputs = self.model( 2025-08-14T21:59:15.7809856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7809929Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7810141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7810226Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7810464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7810559Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7810803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7810938Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7810942Z 2025-08-14T21:59:15.7811045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7811234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7811324Z return mod(**inputs) 2025-08-14T21:59:15.7811568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7811631Z outputs = self.model( 2025-08-14T21:59:15.7811875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7811952Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7812198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7812287Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7812551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7812648Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7812901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7813027Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7813031Z 2025-08-14T21:59:15.7813137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7813333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7813398Z return mod(**inputs) 2025-08-14T21:59:15.7813654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7813727Z outputs = self.model( 2025-08-14T21:59:15.7813984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7814068Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7814296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7814386Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7814642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7814742Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7815006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7815089Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7815093Z 2025-08-14T21:59:15.7815209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7815414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7815482Z return mod(**inputs) 2025-08-14T21:59:15.7815746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7815819Z outputs = self.model( 2025-08-14T21:59:15.7816074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7816168Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7816384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7816468Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7816710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7816866Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7816869Z 2025-08-14T21:59:15.7816974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7817165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7817256Z return mod(**inputs) 2025-08-14T21:59:15.7817498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7817564Z outputs = self.model( 2025-08-14T21:59:15.7817815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7817886Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7818113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7818196Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7818452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7818592Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7818798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7818867Z return self.act(input) 2025-08-14T21:59:15.7818871Z 2025-08-14T21:59:15.7818978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7819169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7819232Z return mod(**inputs) 2025-08-14T21:59:15.7819473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7819538Z outputs = self.model( 2025-08-14T21:59:15.7819781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7819852Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7820063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7820146Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7820383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7820467Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7820471Z 2025-08-14T21:59:15.7820571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7820765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7820838Z return mod(**inputs) 2025-08-14T21:59:15.7821081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7821151Z outputs = self.model( 2025-08-14T21:59:15.7821399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7821469Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7821693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7821771Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7822012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7822099Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7822102Z 2025-08-14T21:59:15.7822204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7822398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7822470Z return mod(**inputs) 2025-08-14T21:59:15.7822727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7822802Z outputs = self.model( 2025-08-14T21:59:15.7823042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7823131Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7823357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7823433Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7823683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7823779Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7824018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7824152Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7824156Z 2025-08-14T21:59:15.7824256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7824465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7824541Z return mod(**inputs) 2025-08-14T21:59:15.7824781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7824853Z outputs = self.model( 2025-08-14T21:59:15.7825093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7825164Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7825384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7825460Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7825703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7825805Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7826043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7826130Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7826133Z 2025-08-14T21:59:15.7826231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7826425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7826496Z return mod(**inputs) 2025-08-14T21:59:15.7826737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7826808Z outputs = self.model( 2025-08-14T21:59:15.7827048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7827120Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7827340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7827417Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7827657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7827760Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7828005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7828119Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7828123Z 2025-08-14T21:59:15.7828223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7828422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7828523Z return mod(**inputs) 2025-08-14T21:59:15.7828766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7828840Z outputs = self.model( 2025-08-14T21:59:15.7829127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7829199Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7829422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7829500Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7829742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7829843Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7830101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7830242Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7830260Z 2025-08-14T21:59:15.7830363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7830568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7830644Z return mod(**inputs) 2025-08-14T21:59:15.7830899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7830976Z outputs = self.model( 2025-08-14T21:59:15.7831228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7831303Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7831540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7831624Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7831880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7831989Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7832243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7832345Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7832348Z 2025-08-14T21:59:15.7832454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7832657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7832733Z return mod(**inputs) 2025-08-14T21:59:15.7832987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7833060Z outputs = self.model( 2025-08-14T21:59:15.7833321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7833398Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7833632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7833716Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7833971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7834080Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7834338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7834446Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7834470Z 2025-08-14T21:59:15.7834581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7834796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7834882Z return mod(**inputs) 2025-08-14T21:59:15.7835144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7835236Z outputs = self.model( 2025-08-14T21:59:15.7835503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7835580Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7835896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7835986Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7836273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7836389Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7836676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7836830Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7836836Z 2025-08-14T21:59:15.7836942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7837151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7837230Z return mod(**inputs) 2025-08-14T21:59:15.7837481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7837553Z outputs = self.model( 2025-08-14T21:59:15.7837820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7837893Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7838117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7838196Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7838437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7838541Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7838781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7838861Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7838872Z 2025-08-14T21:59:15.7838972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7839166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7839239Z return mod(**inputs) 2025-08-14T21:59:15.7839478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7839545Z outputs = self.model( 2025-08-14T21:59:15.7839791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7839861Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7840092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7840167Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7840400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7840521Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7840524Z 2025-08-14T21:59:15.7840620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7840832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7840902Z return mod(**inputs) 2025-08-14T21:59:15.7841138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7841227Z outputs = self.model( 2025-08-14T21:59:15.7841469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7841538Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7841755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7841828Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7842077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7842208Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7857531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7857784Z return self.act(input) 2025-08-14T21:59:15.7857896Z 2025-08-14T21:59:15.7858053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7858291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7858375Z return mod(**inputs) 2025-08-14T21:59:15.7858672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7858747Z outputs = self.model( 2025-08-14T21:59:15.7859004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7859090Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7859319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7859417Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7859666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7859750Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7859757Z 2025-08-14T21:59:15.7859879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7860086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7860161Z return mod(**inputs) 2025-08-14T21:59:15.7860409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7860480Z outputs = self.model( 2025-08-14T21:59:15.7860731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7860809Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7861029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7861122Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7861367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7861482Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7861727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7861841Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7861846Z 2025-08-14T21:59:15.7861962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7862163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7862271Z return mod(**inputs) 2025-08-14T21:59:15.7862517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7862587Z outputs = self.model( 2025-08-14T21:59:15.7862839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7862946Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7863171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7863259Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7863505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7863615Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7863892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7863976Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7863979Z 2025-08-14T21:59:15.7864090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7864305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7864374Z return mod(**inputs) 2025-08-14T21:59:15.7864623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7864689Z outputs = self.model( 2025-08-14T21:59:15.7864938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7865011Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7865227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7865317Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7865562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7865670Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7865913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7866026Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7866030Z 2025-08-14T21:59:15.7866141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7866339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7866404Z return mod(**inputs) 2025-08-14T21:59:15.7866654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7866721Z outputs = self.model( 2025-08-14T21:59:15.7866981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7867056Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7867297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7867389Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7867655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7867764Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7868019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7868169Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7868173Z 2025-08-14T21:59:15.7868284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7868502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7868569Z return mod(**inputs) 2025-08-14T21:59:15.7868825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7868892Z outputs = self.model( 2025-08-14T21:59:15.7869165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7869237Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7869456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7869548Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7869815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7869935Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7870204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7870312Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7870316Z 2025-08-14T21:59:15.7870433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7870643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7870714Z return mod(**inputs) 2025-08-14T21:59:15.7870984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7871055Z outputs = self.model( 2025-08-14T21:59:15.7871319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7871394Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7871632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7871724Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7871988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7872089Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7872372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7872473Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7872477Z 2025-08-14T21:59:15.7872591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7872797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7872866Z return mod(**inputs) 2025-08-14T21:59:15.7873128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7873200Z outputs = self.model( 2025-08-14T21:59:15.7873469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7873544Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7873772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7873860Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7874120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7874221Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7874490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7874626Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7874651Z 2025-08-14T21:59:15.7874768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7874977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7875047Z return mod(**inputs) 2025-08-14T21:59:15.7875310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7875417Z outputs = self.model( 2025-08-14T21:59:15.7875694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7875869Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7876119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7876213Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7876497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7876607Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7876902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7876993Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7877000Z 2025-08-14T21:59:15.7877127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7877338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7877408Z return mod(**inputs) 2025-08-14T21:59:15.7877673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7877745Z outputs = self.model( 2025-08-14T21:59:15.7878005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7878094Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7878324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7878416Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7878672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7878803Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7878808Z 2025-08-14T21:59:15.7878923Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7879130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7879208Z return mod(**inputs) 2025-08-14T21:59:15.7879466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7879541Z outputs = self.model( 2025-08-14T21:59:15.7879806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7879882Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7880113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7880204Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7880460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7880591Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7880814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7880887Z return self.act(input) 2025-08-14T21:59:15.7880891Z 2025-08-14T21:59:15.7881003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7881228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7881296Z return mod(**inputs) 2025-08-14T21:59:15.7881557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7881628Z outputs = self.model( 2025-08-14T21:59:15.7881907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7881982Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7882209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7882298Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7882550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7882645Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7882673Z 2025-08-14T21:59:15.7882783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7883007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7883092Z return mod(**inputs) 2025-08-14T21:59:15.7883327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7883394Z outputs = self.model( 2025-08-14T21:59:15.7883635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7883704Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7883925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7884000Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7884235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7884323Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7884326Z 2025-08-14T21:59:15.7884425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7884622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7884687Z return mod(**inputs) 2025-08-14T21:59:15.7884918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7884991Z outputs = self.model( 2025-08-14T21:59:15.7885223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7885293Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7885511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7885590Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7885839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7885937Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7886177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7886299Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7886303Z 2025-08-14T21:59:15.7886404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7886599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7886670Z return mod(**inputs) 2025-08-14T21:59:15.7886909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7886983Z outputs = self.model( 2025-08-14T21:59:15.7887241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7887313Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7887545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7887640Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7887880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7887974Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7888204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7888289Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7888293Z 2025-08-14T21:59:15.7888389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7888593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7888666Z return mod(**inputs) 2025-08-14T21:59:15.7888924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7889000Z outputs = self.model( 2025-08-14T21:59:15.7889233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7889304Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7889521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7889596Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7889830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7889934Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7890166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7890283Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7890286Z 2025-08-14T21:59:15.7890384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7890576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7890649Z return mod(**inputs) 2025-08-14T21:59:15.7890881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7890952Z outputs = self.model( 2025-08-14T21:59:15.7891185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7891257Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7891476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7891552Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7891785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7891886Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7892120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7892260Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7892263Z 2025-08-14T21:59:15.7892361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7892549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7892621Z return mod(**inputs) 2025-08-14T21:59:15.7892857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7892949Z outputs = self.model( 2025-08-14T21:59:15.7893186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7893257Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7893494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7893571Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7893808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7893912Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7894146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7894253Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7894259Z 2025-08-14T21:59:15.7894357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7894560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7894631Z return mod(**inputs) 2025-08-14T21:59:15.7894873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7894946Z outputs = self.model( 2025-08-14T21:59:15.7895185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7895256Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7895474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7895556Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7895805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7895904Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7896153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7896249Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7896253Z 2025-08-14T21:59:15.7896354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7896557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7896623Z return mod(**inputs) 2025-08-14T21:59:15.7896881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7896945Z outputs = self.model( 2025-08-14T21:59:15.7897180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7897257Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7897467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7897542Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7897783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7897876Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7898116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7898239Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7898243Z 2025-08-14T21:59:15.7898339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7898540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7898630Z return mod(**inputs) 2025-08-14T21:59:15.7898864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7898936Z outputs = self.model( 2025-08-14T21:59:15.7899169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7899264Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7899475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7899551Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7899790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7899882Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7900137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7900220Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7900224Z 2025-08-14T21:59:15.7900337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7900534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7900600Z return mod(**inputs) 2025-08-14T21:59:15.7900835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7900906Z outputs = self.model( 2025-08-14T21:59:15.7901140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7901216Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7901425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7901505Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7901747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7901864Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7901867Z 2025-08-14T21:59:15.7902000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7902188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7902250Z return mod(**inputs) 2025-08-14T21:59:15.7902488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7902561Z outputs = self.model( 2025-08-14T21:59:15.7902787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7902864Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7903070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7903151Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7903396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7903512Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7903728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7903796Z return self.act(input) 2025-08-14T21:59:15.7903800Z 2025-08-14T21:59:15.7903899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7904097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7904161Z return mod(**inputs) 2025-08-14T21:59:15.7904409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7904492Z outputs = self.model( 2025-08-14T21:59:15.7904735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7904812Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7905047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7905144Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7905371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7905446Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7905450Z 2025-08-14T21:59:15.7905551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7905751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7905815Z return mod(**inputs) 2025-08-14T21:59:15.7906051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7906129Z outputs = self.model( 2025-08-14T21:59:15.7906365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7906435Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7906635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7906716Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7906943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7907034Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7907270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7907376Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7907379Z 2025-08-14T21:59:15.7907483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7907666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7907729Z return mod(**inputs) 2025-08-14T21:59:15.7907967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7908028Z outputs = self.model( 2025-08-14T21:59:15.7908266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7908333Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7908541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7908623Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7909009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7909110Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7909355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7909432Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7909436Z 2025-08-14T21:59:15.7909544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7909732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7909797Z return mod(**inputs) 2025-08-14T21:59:15.7910038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7910169Z outputs = self.model( 2025-08-14T21:59:15.7910409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7910482Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7910696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7910806Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7911092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7911189Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7911440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7911552Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7911556Z 2025-08-14T21:59:15.7911690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7911889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7911956Z return mod(**inputs) 2025-08-14T21:59:15.7912231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7912301Z outputs = self.model( 2025-08-14T21:59:15.7912547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7912625Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7912844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7912930Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7913176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7913274Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7913531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7913673Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7913676Z 2025-08-14T21:59:15.7913792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7914000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7914067Z return mod(**inputs) 2025-08-14T21:59:15.7914338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7914408Z outputs = self.model( 2025-08-14T21:59:15.7914674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7914758Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7914988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7915077Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7915345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7915449Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7915712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7915866Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7915872Z 2025-08-14T21:59:15.7915994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7916204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7916281Z return mod(**inputs) 2025-08-14T21:59:15.7916585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7916658Z outputs = self.model( 2025-08-14T21:59:15.7916940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7917046Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7917268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7917356Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7917600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7917697Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7917949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7918066Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7918070Z 2025-08-14T21:59:15.7918179Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7918397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7918465Z return mod(**inputs) 2025-08-14T21:59:15.7918716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7918782Z outputs = self.model( 2025-08-14T21:59:15.7919037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7919112Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7919341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7919429Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7919688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7919789Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7920052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7920193Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7920196Z 2025-08-14T21:59:15.7920303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7920498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7920564Z return mod(**inputs) 2025-08-14T21:59:15.7920810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7920876Z outputs = self.model( 2025-08-14T21:59:15.7921120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7921200Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7921416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7921499Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7921740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7921833Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7922078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7922157Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7922161Z 2025-08-14T21:59:15.7922268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7922465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7922548Z return mod(**inputs) 2025-08-14T21:59:15.7922797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7922862Z outputs = self.model( 2025-08-14T21:59:15.7923105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7923200Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7923414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7923495Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7923733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7923849Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7923854Z 2025-08-14T21:59:15.7923977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7924174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7924260Z return mod(**inputs) 2025-08-14T21:59:15.7924501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7924570Z outputs = self.model( 2025-08-14T21:59:15.7924814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7924884Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7925109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7925198Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7925454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7925585Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7925805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7925878Z return self.act(input) 2025-08-14T21:59:15.7925882Z 2025-08-14T21:59:15.7925999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7926204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7926274Z return mod(**inputs) 2025-08-14T21:59:15.7926532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7926600Z outputs = self.model( 2025-08-14T21:59:15.7926861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7926938Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7927171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7927264Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7927521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7927616Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7927621Z 2025-08-14T21:59:15.7927726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7927932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7928010Z return mod(**inputs) 2025-08-14T21:59:15.7928268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7928338Z outputs = self.model( 2025-08-14T21:59:15.7928603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7928708Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7928930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7929009Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7929268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:59:15.7929354Z hidden_states = residual + hidden_states 2025-08-14T21:59:15.7929358Z 2025-08-14T21:59:15.7929458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7929658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7929723Z return mod(**inputs) 2025-08-14T21:59:15.7929963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7930054Z outputs = self.model( 2025-08-14T21:59:15.7930295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7930379Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7930609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7930687Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7930935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7931032Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7931270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7931385Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7931389Z 2025-08-14T21:59:15.7931492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7931684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7931757Z return mod(**inputs) 2025-08-14T21:59:15.7931996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7932069Z outputs = self.model( 2025-08-14T21:59:15.7932309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7932379Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7932602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7932678Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7932926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7933025Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7933266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:59:15.7933353Z key_states = self.k_proj(current_states) 2025-08-14T21:59:15.7933358Z 2025-08-14T21:59:15.7933458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7933649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7933725Z return mod(**inputs) 2025-08-14T21:59:15.7933978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7934053Z outputs = self.model( 2025-08-14T21:59:15.7934308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7934385Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7934642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7934725Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7934985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7935105Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7935342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:59:15.7935459Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:59:15.7935462Z 2025-08-14T21:59:15.7935567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7935770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7935847Z return mod(**inputs) 2025-08-14T21:59:15.7936119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7936195Z outputs = self.model( 2025-08-14T21:59:15.7936490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7936570Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7936804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7936883Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7937138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7937245Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7937506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:59:15.7937648Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:59:15.7937651Z 2025-08-14T21:59:15.7937751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7937945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7938017Z return mod(**inputs) 2025-08-14T21:59:15.7938256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7938325Z outputs = self.model( 2025-08-14T21:59:15.7938562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7938633Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7938854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7938932Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7939174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7939279Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7939526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:59:15.7939620Z value_states = self.v_proj(current_states) 2025-08-14T21:59:15.7939623Z 2025-08-14T21:59:15.7939723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7939914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7939987Z return mod(**inputs) 2025-08-14T21:59:15.7940224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7940291Z outputs = self.model( 2025-08-14T21:59:15.7940546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7940631Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7940850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7940925Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7941178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7941279Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7941513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:59:15.7941611Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:59:15.7941615Z 2025-08-14T21:59:15.7941712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7941957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7942032Z return mod(**inputs) 2025-08-14T21:59:15.7942285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7942351Z outputs = self.model( 2025-08-14T21:59:15.7942594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7942664Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7942882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7942955Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7943189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7943291Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7943527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:59:15.7943656Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:59:15.7943661Z 2025-08-14T21:59:15.7943761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7943950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7944021Z return mod(**inputs) 2025-08-14T21:59:15.7944256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7944320Z outputs = self.model( 2025-08-14T21:59:15.7944560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7944630Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7944852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7944937Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7945192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:59:15.7945301Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:59:15.7945558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:59:15.7945648Z attn_output = self.out_proj(attn_output) 2025-08-14T21:59:15.7945653Z 2025-08-14T21:59:15.7945757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7945961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7946036Z return mod(**inputs) 2025-08-14T21:59:15.7946291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7946380Z outputs = self.model( 2025-08-14T21:59:15.7946644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7946721Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7946957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7947066Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7947298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7947417Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7947421Z 2025-08-14T21:59:15.7947518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7947709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7947796Z return mod(**inputs) 2025-08-14T21:59:15.7948033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7948105Z outputs = self.model( 2025-08-14T21:59:15.7948367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7948444Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7948681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7948765Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7949028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:59:15.7949151Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:59:15.7949373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:59:15.7949455Z return self.act(input) 2025-08-14T21:59:15.7949459Z 2025-08-14T21:59:15.7949566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7949774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7949850Z return mod(**inputs) 2025-08-14T21:59:15.7950105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:59:15.7950181Z outputs = self.model( 2025-08-14T21:59:15.7950440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:59:15.7950514Z layer_outputs = decoder_layer( 2025-08-14T21:59:15.7950748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:59:15.7950829Z return super().__call__(*args, **kwargs) 2025-08-14T21:59:15.7951089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:59:15.7951180Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:59:15.7951186Z 2025-08-14T21:59:15.7951292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7951507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7951573Z return mod(**inputs) 2025-08-14T21:59:15.7951836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 681, in forward 2025-08-14T21:59:15.7951926Z logits = self.lm_head(outputs[0]) 2025-08-14T21:59:15.7951930Z 2025-08-14T21:59:15.7952036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:59:15.7952278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:59:15.7952345Z return mod(**inputs) 2025-08-14T21:59:15.7952623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 685, in forward 2025-08-14T21:59:15.7952708Z loss = self.loss_function( 2025-08-14T21:59:15.7952970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:59:15.7953178Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:59:15.7953451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:59:15.7953661Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:59:15.7953665Z 2025-08-14T21:59:27.8075549Z Compilation time (from dynamo_timed): 25.251679734 2025-08-14T21:59:27.8182855Z pass 2025-08-14T21:59:27.8183806Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:59:27.8184742Z TIMING: _recursive_pre_grad_passes:0.01353 _recursive_joint_graph_passes:0.79888 _recursive_post_grad_passes:0.28799 async_compile.wait:0.83565 code_gen:11.24016 inductor_compile:14.40362 backend_compile:20.49714 gc:0.00073 entire_frame_compile:25.25168 total_wall_time:25.25168 2025-08-14T21:59:27.8185698Z STATS: call_* op count: 921 | FakeTensorMode.__torch_dispatch__:29112 | FakeTensor.__torch_dispatch__:10687 | ProxyTorchDispatchMode.__torch_dispatch__:10816 2025-08-14T21:59:27.8186249Z Dynamo produced 1 graphs covering 921 ops with 0 graph breaks (0 unique) 2025-08-14T21:59:33.5826619Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:59:33.5827538Z from pkg_resources import resource_filename 2025-08-14T21:59:34.1699208Z 2025-08-14T21:59:37.5175716Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:59:37.5178872Z loading model: 0it [00:03, ?it/s] 2025-08-14T21:59:37.5201173Z cpu eval XLNetLMHeadModel 2025-08-14T21:59:40.2060300Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:59:41.1564954Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:59:42.1206034Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T22:00:03.1853609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1856875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1860562Z return mod(**inputs) 2025-08-14T22:00:03.1861413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1861933Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1862394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1307, in forward 2025-08-14T22:00:03.1862832Z word_emb_k = self.word_embedding(input_ids) 2025-08-14T22:00:03.1862997Z 2025-08-14T22:00:03.1863116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1863491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1863829Z return mod(**inputs) 2025-08-14T22:00:03.1864200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1864642Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1865134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-08-14T22:00:03.1865908Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-08-14T22:00:03.1866436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-08-14T22:00:03.1866978Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-08-14T22:00:03.1867550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-08-14T22:00:03.1868076Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-08-14T22:00:03.1868347Z 2025-08-14T22:00:03.1868472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1868858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1869206Z return mod(**inputs) 2025-08-14T22:00:03.1869701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1870131Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1870598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-08-14T22:00:03.1871066Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-08-14T22:00:03.1871577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-08-14T22:00:03.1872097Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-08-14T22:00:03.1872600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-08-14T22:00:03.1873166Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-08-14T22:00:03.1873391Z 2025-08-14T22:00:03.1873509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1873888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1874229Z return mod(**inputs) 2025-08-14T22:00:03.1874642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1875072Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1875496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1876119Z outputs = layer_module( 2025-08-14T22:00:03.1876512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1876930Z outputs = self.rel_attn( 2025-08-14T22:00:03.1877342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1877762Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1878208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1878680Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1878870Z 2025-08-14T22:00:03.1878990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1879381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1879746Z return mod(**inputs) 2025-08-14T22:00:03.1880148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1880588Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1881019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1881486Z outputs = layer_module( 2025-08-14T22:00:03.1882066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1882491Z outputs = self.rel_attn( 2025-08-14T22:00:03.1882900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1883360Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1883813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1884294Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1884485Z 2025-08-14T22:00:03.1884599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1884995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1885366Z return mod(**inputs) 2025-08-14T22:00:03.1885758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1886264Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1886695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1887117Z outputs = layer_module( 2025-08-14T22:00:03.1887517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1887943Z outputs = self.rel_attn( 2025-08-14T22:00:03.1888344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1888780Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1889231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1889716Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1889898Z 2025-08-14T22:00:03.1890024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1890413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1890768Z return mod(**inputs) 2025-08-14T22:00:03.1891163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1891587Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1892009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1892433Z outputs = layer_module( 2025-08-14T22:00:03.1892833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1893242Z outputs = self.rel_attn( 2025-08-14T22:00:03.1893640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1894073Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1894518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1894997Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1895181Z 2025-08-14T22:00:03.1895297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1895684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1896031Z return mod(**inputs) 2025-08-14T22:00:03.1896424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1896888Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1897325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1897745Z outputs = layer_module( 2025-08-14T22:00:03.1898144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1898600Z outputs = self.rel_attn( 2025-08-14T22:00:03.1898997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1899435Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1899888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1900375Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1900556Z 2025-08-14T22:00:03.1900690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1901085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1901449Z return mod(**inputs) 2025-08-14T22:00:03.1901869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1902306Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1902722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1903856Z outputs = layer_module( 2025-08-14T22:00:03.1904257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1904737Z outputs = self.rel_attn( 2025-08-14T22:00:03.1905154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1905595Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1906051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1907063Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1907255Z 2025-08-14T22:00:03.1907382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1907779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1908126Z return mod(**inputs) 2025-08-14T22:00:03.1908525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1909207Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1909630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1910059Z outputs = layer_module( 2025-08-14T22:00:03.1910460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1910875Z outputs = self.rel_attn( 2025-08-14T22:00:03.1911271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1911719Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1912178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1912647Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1912854Z 2025-08-14T22:00:03.1912966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1913370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1913800Z return mod(**inputs) 2025-08-14T22:00:03.1914199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1914647Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1915093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1915554Z outputs = layer_module( 2025-08-14T22:00:03.1916036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1916457Z outputs = self.rel_attn( 2025-08-14T22:00:03.1916860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1917300Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1917809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1918292Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1918487Z 2025-08-14T22:00:03.1918636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1919021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1919387Z return mod(**inputs) 2025-08-14T22:00:03.1919799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1920247Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1920682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1921100Z outputs = layer_module( 2025-08-14T22:00:03.1921500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1921908Z outputs = self.rel_attn( 2025-08-14T22:00:03.1922387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1922839Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1923288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1923766Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1923957Z 2025-08-14T22:00:03.1924069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1924459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1924806Z return mod(**inputs) 2025-08-14T22:00:03.1925189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1925630Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1926052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1926470Z outputs = layer_module( 2025-08-14T22:00:03.1926871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1927284Z outputs = self.rel_attn( 2025-08-14T22:00:03.1927681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1928115Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1928569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1929042Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1929220Z 2025-08-14T22:00:03.1929358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1929744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1930103Z return mod(**inputs) 2025-08-14T22:00:03.1930484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1930918Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1931332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1931759Z outputs = layer_module( 2025-08-14T22:00:03.1932141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1932541Z outputs = self.rel_attn( 2025-08-14T22:00:03.1932963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1933395Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1933857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1934345Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1934526Z 2025-08-14T22:00:03.1934636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1935010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1935342Z return mod(**inputs) 2025-08-14T22:00:03.1935832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1936258Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1936681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1937076Z outputs = layer_module( 2025-08-14T22:00:03.1937466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1937876Z outputs = self.rel_attn( 2025-08-14T22:00:03.1938255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1938680Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1939119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1939585Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1939762Z 2025-08-14T22:00:03.1939873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1940257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1940622Z return mod(**inputs) 2025-08-14T22:00:03.1940998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1941419Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1941842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1942243Z outputs = layer_module( 2025-08-14T22:00:03.1942630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1943041Z outputs = self.rel_attn( 2025-08-14T22:00:03.1943438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1943879Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1944308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1944801Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1944974Z 2025-08-14T22:00:03.1945094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1945471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1945837Z return mod(**inputs) 2025-08-14T22:00:03.1946232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1946657Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1947070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1947479Z outputs = layer_module( 2025-08-14T22:00:03.1947881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1948280Z outputs = self.rel_attn( 2025-08-14T22:00:03.1948655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1949094Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1949544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1950014Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1950199Z 2025-08-14T22:00:03.1950311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1950700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1951058Z return mod(**inputs) 2025-08-14T22:00:03.1951440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1951870Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1952302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1952717Z outputs = layer_module( 2025-08-14T22:00:03.1953105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1953522Z outputs = self.rel_attn( 2025-08-14T22:00:03.1953919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1954345Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1954798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1955274Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1955452Z 2025-08-14T22:00:03.1955577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1956048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1956424Z return mod(**inputs) 2025-08-14T22:00:03.1956824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1957254Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1957683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1958103Z outputs = layer_module( 2025-08-14T22:00:03.1958500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1958909Z outputs = self.rel_attn( 2025-08-14T22:00:03.1959316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1960305Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1960762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1961237Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1961456Z 2025-08-14T22:00:03.1961571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1961959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1962301Z return mod(**inputs) 2025-08-14T22:00:03.1962693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1963118Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1963566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1963975Z outputs = layer_module( 2025-08-14T22:00:03.1964370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1964804Z outputs = self.rel_attn( 2025-08-14T22:00:03.1965197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1965626Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1966075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1966549Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1966727Z 2025-08-14T22:00:03.1966840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1967228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1967586Z return mod(**inputs) 2025-08-14T22:00:03.1967980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1968418Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1968846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1969265Z outputs = layer_module( 2025-08-14T22:00:03.1969640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1970043Z outputs = self.rel_attn( 2025-08-14T22:00:03.1970427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1970852Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1971283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1971751Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1971923Z 2025-08-14T22:00:03.1972040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1972416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1972752Z return mod(**inputs) 2025-08-14T22:00:03.1973133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1973555Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1973966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1974365Z outputs = layer_module( 2025-08-14T22:00:03.1974747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1975172Z outputs = self.rel_attn( 2025-08-14T22:00:03.1975559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1975985Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1976434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1976931Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1977102Z 2025-08-14T22:00:03.1977212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1977584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1977929Z return mod(**inputs) 2025-08-14T22:00:03.1978304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1978741Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1979162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1979582Z outputs = layer_module( 2025-08-14T22:00:03.1979973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1980389Z outputs = self.rel_attn( 2025-08-14T22:00:03.1980786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1981252Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1981757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1982231Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1982430Z 2025-08-14T22:00:03.1982556Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1982959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1983331Z return mod(**inputs) 2025-08-14T22:00:03.1983748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1984206Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1984644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1985071Z outputs = layer_module( 2025-08-14T22:00:03.1985476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1985884Z outputs = self.rel_attn( 2025-08-14T22:00:03.1986292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1986736Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1987195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1987673Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1987873Z 2025-08-14T22:00:03.1987992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1988395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1988768Z return mod(**inputs) 2025-08-14T22:00:03.1989175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1989621Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1990059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1990492Z outputs = layer_module( 2025-08-14T22:00:03.1990889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1991306Z outputs = self.rel_attn( 2025-08-14T22:00:03.1991705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1992153Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1992607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1993088Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1993269Z 2025-08-14T22:00:03.1993382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1993773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1994141Z return mod(**inputs) 2025-08-14T22:00:03.1994537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.1994995Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.1995454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.1995977Z outputs = layer_module( 2025-08-14T22:00:03.1996391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.1996807Z outputs = self.rel_attn( 2025-08-14T22:00:03.1997215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.1997675Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.1998132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.1998619Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.1998810Z 2025-08-14T22:00:03.1998924Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.1999317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.1999671Z return mod(**inputs) 2025-08-14T22:00:03.2000065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2000497Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2000920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2001343Z outputs = layer_module( 2025-08-14T22:00:03.2001733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2002134Z outputs = self.rel_attn( 2025-08-14T22:00:03.2002520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2002947Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2003389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2003852Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2004030Z 2025-08-14T22:00:03.2004134Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2004495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2004838Z return mod(**inputs) 2025-08-14T22:00:03.2005213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2005662Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2006074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2006474Z outputs = layer_module( 2025-08-14T22:00:03.2006849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2007271Z outputs = self.rel_attn( 2025-08-14T22:00:03.2007655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2008081Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2008252Z 2025-08-14T22:00:03.2008360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2008943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2009286Z return mod(**inputs) 2025-08-14T22:00:03.2009700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2010093Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2010515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2010902Z outputs = layer_module( 2025-08-14T22:00:03.2011261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2011646Z outputs = self.rel_attn( 2025-08-14T22:00:03.2012019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2012435Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2012600Z 2025-08-14T22:00:03.2012710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2013070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2013393Z return mod(**inputs) 2025-08-14T22:00:03.2013753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2014155Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2014550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2014930Z outputs = layer_module( 2025-08-14T22:00:03.2015298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2015705Z outputs = self.rel_attn( 2025-08-14T22:00:03.2016094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2016504Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2016932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2017430Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2017629Z 2025-08-14T22:00:03.2017746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2018126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2018467Z return mod(**inputs) 2025-08-14T22:00:03.2018855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2019272Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2019692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-08-14T22:00:03.2020166Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-08-14T22:00:03.2020719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-08-14T22:00:03.2021238Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-08-14T22:00:03.2021735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-08-14T22:00:03.2022296Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-08-14T22:00:03.2022514Z 2025-08-14T22:00:03.2022630Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2022999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2023342Z return mod(**inputs) 2025-08-14T22:00:03.2023742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2024171Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2024597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2025009Z outputs = layer_module( 2025-08-14T22:00:03.2025393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2025793Z outputs = self.rel_attn( 2025-08-14T22:00:03.2026185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2026660Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2026861Z 2025-08-14T22:00:03.2026976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2027349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2027695Z return mod(**inputs) 2025-08-14T22:00:03.2028081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2028500Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2028916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2029327Z outputs = layer_module( 2025-08-14T22:00:03.2029714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2030114Z outputs = self.rel_attn( 2025-08-14T22:00:03.2030504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2030953Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2031370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2031862Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2032061Z 2025-08-14T22:00:03.2032171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2032549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2032884Z return mod(**inputs) 2025-08-14T22:00:03.2033271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2033697Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2034115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2034510Z outputs = layer_module( 2025-08-14T22:00:03.2034905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2035356Z outputs = self.rel_attn( 2025-08-14T22:00:03.2035745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2036300Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2036510Z 2025-08-14T22:00:03.2036628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2037031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2037383Z return mod(**inputs) 2025-08-14T22:00:03.2037783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2038291Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2038728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2039168Z outputs = layer_module( 2025-08-14T22:00:03.2039573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2040005Z outputs = self.rel_attn( 2025-08-14T22:00:03.2040394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2040809Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2041240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2041727Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2041916Z 2025-08-14T22:00:03.2042028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2042413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2042761Z return mod(**inputs) 2025-08-14T22:00:03.2043146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2043576Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2044002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2044416Z outputs = layer_module( 2025-08-14T22:00:03.2044802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2045213Z outputs = self.rel_attn( 2025-08-14T22:00:03.2045611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2046037Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2046472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2046939Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2047115Z 2025-08-14T22:00:03.2047231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2047599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2047944Z return mod(**inputs) 2025-08-14T22:00:03.2048322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2048738Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2049144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2049545Z outputs = layer_module( 2025-08-14T22:00:03.2049929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2050372Z outputs = self.rel_attn( 2025-08-14T22:00:03.2050754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2051185Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2051628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2052116Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2052300Z 2025-08-14T22:00:03.2052409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2052787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2053127Z return mod(**inputs) 2025-08-14T22:00:03.2053501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2053944Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2054361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2054752Z outputs = layer_module( 2025-08-14T22:00:03.2055166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2055724Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2056286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2056706Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2057123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2057537Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2057935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2058339Z output = self.layer_1(output) 2025-08-14T22:00:03.2058485Z 2025-08-14T22:00:03.2058597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2058978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2059321Z return mod(**inputs) 2025-08-14T22:00:03.2059700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2060128Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2060516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2060887Z outputs = layer_module( 2025-08-14T22:00:03.2061249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2061763Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2062285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2062675Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2063058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2063446Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2063812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2064228Z output = self.activation_function(output) 2025-08-14T22:00:03.2064606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2065000Z return self.act(input) 2025-08-14T22:00:03.2065122Z 2025-08-14T22:00:03.2065233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2065614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2065958Z return mod(**inputs) 2025-08-14T22:00:03.2066330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2066772Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2067184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2067589Z outputs = layer_module( 2025-08-14T22:00:03.2067969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2068536Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2069093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2069530Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2069930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2070340Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2070731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2071149Z output = self.layer_2(output) 2025-08-14T22:00:03.2071283Z 2025-08-14T22:00:03.2071393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2071772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2072117Z return mod(**inputs) 2025-08-14T22:00:03.2072493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2072919Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2073348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2073768Z outputs = layer_module( 2025-08-14T22:00:03.2074154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2074573Z outputs = self.rel_attn( 2025-08-14T22:00:03.2074968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2075404Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2075575Z 2025-08-14T22:00:03.2075686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2076171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2076538Z return mod(**inputs) 2025-08-14T22:00:03.2076955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2077403Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2077828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2078230Z outputs = layer_module( 2025-08-14T22:00:03.2078619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2079028Z outputs = self.rel_attn( 2025-08-14T22:00:03.2079427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2079870Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2080073Z 2025-08-14T22:00:03.2080184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2080567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2080908Z return mod(**inputs) 2025-08-14T22:00:03.2081278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2081730Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2082154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2082552Z outputs = layer_module( 2025-08-14T22:00:03.2082937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2083342Z outputs = self.rel_attn( 2025-08-14T22:00:03.2083760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2084165Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2084617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2085109Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2085315Z 2025-08-14T22:00:03.2085426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2085778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2086098Z return mod(**inputs) 2025-08-14T22:00:03.2086460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2086852Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2087247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2087629Z outputs = layer_module( 2025-08-14T22:00:03.2087990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2088360Z outputs = self.rel_attn( 2025-08-14T22:00:03.2088730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2089231Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2089427Z 2025-08-14T22:00:03.2089535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2089907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2090233Z return mod(**inputs) 2025-08-14T22:00:03.2090595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2091056Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2091453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2091837Z outputs = layer_module( 2025-08-14T22:00:03.2092203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2092575Z outputs = self.rel_attn( 2025-08-14T22:00:03.2092947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2093331Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2093717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2094175Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2094398Z 2025-08-14T22:00:03.2094503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2094859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2095182Z return mod(**inputs) 2025-08-14T22:00:03.2095549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2095967Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2096354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2096734Z outputs = layer_module( 2025-08-14T22:00:03.2097097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2097477Z outputs = self.rel_attn( 2025-08-14T22:00:03.2097852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2098272Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2098426Z 2025-08-14T22:00:03.2098556Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2098906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2099221Z return mod(**inputs) 2025-08-14T22:00:03.2099582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2099965Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2100338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2100712Z outputs = layer_module( 2025-08-14T22:00:03.2101065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2101435Z outputs = self.rel_attn( 2025-08-14T22:00:03.2101782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2102154Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2102539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2102967Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2103150Z 2025-08-14T22:00:03.2103252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2103608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2103929Z return mod(**inputs) 2025-08-14T22:00:03.2104277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2104679Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2105075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2105443Z outputs = layer_module( 2025-08-14T22:00:03.2105792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2106168Z outputs = self.rel_attn( 2025-08-14T22:00:03.2106530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2106916Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2107324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2107768Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2107934Z 2025-08-14T22:00:03.2108044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2108431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2108944Z return mod(**inputs) 2025-08-14T22:00:03.2109335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2109807Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2110217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2110618Z outputs = layer_module( 2025-08-14T22:00:03.2111000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2111395Z outputs = self.rel_attn( 2025-08-14T22:00:03.2111782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2112233Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2112703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2113166Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2113351Z 2025-08-14T22:00:03.2113462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2113843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2114178Z return mod(**inputs) 2025-08-14T22:00:03.2114563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2114985Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2115399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2115853Z outputs = layer_module( 2025-08-14T22:00:03.2116256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2116823Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2117398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2117812Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2118220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2118629Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2119024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2119432Z output = self.layer_1(output) 2025-08-14T22:00:03.2119575Z 2025-08-14T22:00:03.2119685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2120062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2120396Z return mod(**inputs) 2025-08-14T22:00:03.2120774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2121195Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2121611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2122005Z outputs = layer_module( 2025-08-14T22:00:03.2122389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2122933Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2123529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2123947Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2124356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2124790Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2125181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2125607Z output = self.activation_function(output) 2025-08-14T22:00:03.2125985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2126351Z return self.act(input) 2025-08-14T22:00:03.2126469Z 2025-08-14T22:00:03.2126580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2126982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2127329Z return mod(**inputs) 2025-08-14T22:00:03.2127725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2128152Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2128535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2128907Z outputs = layer_module( 2025-08-14T22:00:03.2129254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2129769Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2130299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2130704Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2131126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2131506Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2131874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2132245Z output = self.layer_2(output) 2025-08-14T22:00:03.2132377Z 2025-08-14T22:00:03.2132481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2132838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2133162Z return mod(**inputs) 2025-08-14T22:00:03.2133515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2133919Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2134314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2134687Z outputs = layer_module( 2025-08-14T22:00:03.2135050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2135438Z outputs = self.rel_attn( 2025-08-14T22:00:03.2135826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2136251Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2136419Z 2025-08-14T22:00:03.2136527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2136899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2137240Z return mod(**inputs) 2025-08-14T22:00:03.2137621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2138046Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2138442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2138822Z outputs = layer_module( 2025-08-14T22:00:03.2139176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2139564Z outputs = self.rel_attn( 2025-08-14T22:00:03.2139951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2140370Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2140532Z 2025-08-14T22:00:03.2140633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2141021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2141342Z return mod(**inputs) 2025-08-14T22:00:03.2141723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2142127Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2142507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2142870Z outputs = layer_module( 2025-08-14T22:00:03.2143227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2143621Z outputs = self.rel_attn( 2025-08-14T22:00:03.2143996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2144403Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2144823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2145314Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2145509Z 2025-08-14T22:00:03.2145620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2146001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2146342Z return mod(**inputs) 2025-08-14T22:00:03.2146730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2147146Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2147583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2147987Z outputs = layer_module( 2025-08-14T22:00:03.2148362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2148762Z outputs = self.rel_attn( 2025-08-14T22:00:03.2149153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2149620Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2149818Z 2025-08-14T22:00:03.2149927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2150303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2150684Z return mod(**inputs) 2025-08-14T22:00:03.2151062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2151472Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2151886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2152317Z outputs = layer_module( 2025-08-14T22:00:03.2152696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2153098Z outputs = self.rel_attn( 2025-08-14T22:00:03.2153517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2153950Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2154375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2154874Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2155073Z 2025-08-14T22:00:03.2155193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2155601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2156051Z return mod(**inputs) 2025-08-14T22:00:03.2156449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2156928Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2157353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2157775Z outputs = layer_module( 2025-08-14T22:00:03.2158160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2158561Z outputs = self.rel_attn( 2025-08-14T22:00:03.2158942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2159380Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2159545Z 2025-08-14T22:00:03.2159664Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2160033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2160373Z return mod(**inputs) 2025-08-14T22:00:03.2160735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2161135Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2161536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2161939Z outputs = layer_module( 2025-08-14T22:00:03.2162319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2162711Z outputs = self.rel_attn( 2025-08-14T22:00:03.2163107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2163514Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2163933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2164401Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2164594Z 2025-08-14T22:00:03.2164704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2165082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2165431Z return mod(**inputs) 2025-08-14T22:00:03.2165820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2166217Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2166614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2167014Z outputs = layer_module( 2025-08-14T22:00:03.2167400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2167800Z outputs = self.rel_attn( 2025-08-14T22:00:03.2168189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2168626Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2169069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2169521Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2169696Z 2025-08-14T22:00:03.2169810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2170184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2170545Z return mod(**inputs) 2025-08-14T22:00:03.2170936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2171371Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2171778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2172160Z outputs = layer_module( 2025-08-14T22:00:03.2172534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2172932Z outputs = self.rel_attn( 2025-08-14T22:00:03.2173328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2173769Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2174213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2174689Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2174877Z 2025-08-14T22:00:03.2174992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2175381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2175729Z return mod(**inputs) 2025-08-14T22:00:03.2176118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2176529Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2176931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2177315Z outputs = layer_module( 2025-08-14T22:00:03.2177689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2178225Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2178785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2179212Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2179634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2180052Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2180459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2180882Z output = self.layer_1(output) 2025-08-14T22:00:03.2181020Z 2025-08-14T22:00:03.2181141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2181529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2181898Z return mod(**inputs) 2025-08-14T22:00:03.2182278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2182697Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2183103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2183523Z outputs = layer_module( 2025-08-14T22:00:03.2183906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2184450Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2184992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2185427Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2185835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2186262Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2186654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2187081Z output = self.activation_function(output) 2025-08-14T22:00:03.2187459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2187815Z return self.act(input) 2025-08-14T22:00:03.2187941Z 2025-08-14T22:00:03.2188049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2188431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2188775Z return mod(**inputs) 2025-08-14T22:00:03.2189153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2189574Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2189994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2190394Z outputs = layer_module( 2025-08-14T22:00:03.2190779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2191331Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2191881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2192296Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2192706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2193118Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2193521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2193922Z output = self.layer_2(output) 2025-08-14T22:00:03.2194062Z 2025-08-14T22:00:03.2194172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2194551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2194889Z return mod(**inputs) 2025-08-14T22:00:03.2195277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2195699Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2196208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2196661Z outputs = layer_module( 2025-08-14T22:00:03.2197069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2197477Z outputs = self.rel_attn( 2025-08-14T22:00:03.2197861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2198332Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2198501Z 2025-08-14T22:00:03.2198609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2198982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2199321Z return mod(**inputs) 2025-08-14T22:00:03.2199699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2200117Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2200549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2200945Z outputs = layer_module( 2025-08-14T22:00:03.2201350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2201733Z outputs = self.rel_attn( 2025-08-14T22:00:03.2202104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2202540Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2202712Z 2025-08-14T22:00:03.2202822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2203197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2203529Z return mod(**inputs) 2025-08-14T22:00:03.2203911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2204336Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2204751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2205144Z outputs = layer_module( 2025-08-14T22:00:03.2205533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2205931Z outputs = self.rel_attn( 2025-08-14T22:00:03.2206312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2206721Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2207144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2207637Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2207833Z 2025-08-14T22:00:03.2207941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2208320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2208846Z return mod(**inputs) 2025-08-14T22:00:03.2209235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2209659Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2210077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2210487Z outputs = layer_module( 2025-08-14T22:00:03.2210845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2211222Z outputs = self.rel_attn( 2025-08-14T22:00:03.2211593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2212088Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2212273Z 2025-08-14T22:00:03.2212377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2212762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2213086Z return mod(**inputs) 2025-08-14T22:00:03.2213450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2213871Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2214281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2214659Z outputs = layer_module( 2025-08-14T22:00:03.2215037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2215423Z outputs = self.rel_attn( 2025-08-14T22:00:03.2215811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2216193Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2216596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2217057Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2217239Z 2025-08-14T22:00:03.2217348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2217698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2218027Z return mod(**inputs) 2025-08-14T22:00:03.2218411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2218833Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2219229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2219608Z outputs = layer_module( 2025-08-14T22:00:03.2219974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2220343Z outputs = self.rel_attn( 2025-08-14T22:00:03.2220711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2221129Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2221285Z 2025-08-14T22:00:03.2221396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2221747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2222075Z return mod(**inputs) 2025-08-14T22:00:03.2222435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2222838Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2223224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2223608Z outputs = layer_module( 2025-08-14T22:00:03.2223971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2224341Z outputs = self.rel_attn( 2025-08-14T22:00:03.2224725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2225178Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2225571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2226042Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2226224Z 2025-08-14T22:00:03.2226328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2226683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2227019Z return mod(**inputs) 2025-08-14T22:00:03.2227378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2227779Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2228171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2228545Z outputs = layer_module( 2025-08-14T22:00:03.2228925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2229309Z outputs = self.rel_attn( 2025-08-14T22:00:03.2229689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2230099Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2230548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2231010Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2231177Z 2025-08-14T22:00:03.2231282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2231640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2231971Z return mod(**inputs) 2025-08-14T22:00:03.2232356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2232771Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2233194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2233600Z outputs = layer_module( 2025-08-14T22:00:03.2233978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2234381Z outputs = self.rel_attn( 2025-08-14T22:00:03.2234768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2235206Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2235655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2236213Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2236398Z 2025-08-14T22:00:03.2236519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2236921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2237259Z return mod(**inputs) 2025-08-14T22:00:03.2237647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2238069Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2238463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2238851Z outputs = layer_module( 2025-08-14T22:00:03.2239219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2239768Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2240338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2240761Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2241171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2241591Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2241961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2242345Z output = self.layer_1(output) 2025-08-14T22:00:03.2242469Z 2025-08-14T22:00:03.2242583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2242953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2243292Z return mod(**inputs) 2025-08-14T22:00:03.2243690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2244114Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2244546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2244925Z outputs = layer_module( 2025-08-14T22:00:03.2245292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2245807Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2246320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2246716Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2247107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2247492Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2247871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2248275Z output = self.activation_function(output) 2025-08-14T22:00:03.2248641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2248979Z return self.act(input) 2025-08-14T22:00:03.2249101Z 2025-08-14T22:00:03.2249204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2249567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2249886Z return mod(**inputs) 2025-08-14T22:00:03.2250252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2250650Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2251051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2251429Z outputs = layer_module( 2025-08-14T22:00:03.2251795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2252316Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2252842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2253230Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2253615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2254001Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2254371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2254801Z output = self.layer_2(output) 2025-08-14T22:00:03.2254935Z 2025-08-14T22:00:03.2255049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2255429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2255783Z return mod(**inputs) 2025-08-14T22:00:03.2256162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2256580Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2256996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2257388Z outputs = layer_module( 2025-08-14T22:00:03.2257775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2258195Z outputs = self.rel_attn( 2025-08-14T22:00:03.2258580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2259028Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2259197Z 2025-08-14T22:00:03.2259862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2260242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2260580Z return mod(**inputs) 2025-08-14T22:00:03.2260966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2261386Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2261789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2262194Z outputs = layer_module( 2025-08-14T22:00:03.2262580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2262976Z outputs = self.rel_attn( 2025-08-14T22:00:03.2263353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2263788Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2263956Z 2025-08-14T22:00:03.2264063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2264437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2264768Z return mod(**inputs) 2025-08-14T22:00:03.2265142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2265559Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2265974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2266375Z outputs = layer_module( 2025-08-14T22:00:03.2266764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2267164Z outputs = self.rel_attn( 2025-08-14T22:00:03.2267544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2267952Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2268374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2268856Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2269050Z 2025-08-14T22:00:03.2269159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2269564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2269907Z return mod(**inputs) 2025-08-14T22:00:03.2270285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2270710Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2271150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2271551Z outputs = layer_module( 2025-08-14T22:00:03.2271927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2272327Z outputs = self.rel_attn( 2025-08-14T22:00:03.2272710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2273239Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2273457Z 2025-08-14T22:00:03.2273567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2273983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2274346Z return mod(**inputs) 2025-08-14T22:00:03.2274636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2274726Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2275016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2275087Z outputs = layer_module( 2025-08-14T22:00:03.2275376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2275451Z outputs = self.rel_attn( 2025-08-14T22:00:03.2275734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2275909Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2276293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2276436Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2276449Z 2025-08-14T22:00:03.2276561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2276773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2276851Z return mod(**inputs) 2025-08-14T22:00:03.2277119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2277207Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2277484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2277558Z outputs = layer_module( 2025-08-14T22:00:03.2277829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2277902Z outputs = self.rel_attn( 2025-08-14T22:00:03.2278168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2278286Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2278289Z 2025-08-14T22:00:03.2278399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2278612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2278691Z return mod(**inputs) 2025-08-14T22:00:03.2278958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2279085Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2279346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2279416Z outputs = layer_module( 2025-08-14T22:00:03.2279685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2279778Z outputs = self.rel_attn( 2025-08-14T22:00:03.2280046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2280122Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2280401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2280536Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2280560Z 2025-08-14T22:00:03.2280669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2280873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2280964Z return mod(**inputs) 2025-08-14T22:00:03.2281232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2281328Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2281592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2281663Z outputs = layer_module( 2025-08-14T22:00:03.2281938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2282009Z outputs = self.rel_attn( 2025-08-14T22:00:03.2282277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2282381Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2282670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2282798Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2282804Z 2025-08-14T22:00:03.2282913Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2283123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2283203Z return mod(**inputs) 2025-08-14T22:00:03.2283480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2283577Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2283854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2283927Z outputs = layer_module( 2025-08-14T22:00:03.2284213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2284288Z outputs = self.rel_attn( 2025-08-14T22:00:03.2284570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2284674Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2284963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2285087Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2285091Z 2025-08-14T22:00:03.2285198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2285411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2285507Z return mod(**inputs) 2025-08-14T22:00:03.2285776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2285873Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2286140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2286228Z outputs = layer_module( 2025-08-14T22:00:03.2286498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2286716Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2286997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2287101Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2287370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2287455Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2287752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2287835Z output = self.layer_1(output) 2025-08-14T22:00:03.2287839Z 2025-08-14T22:00:03.2287956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2288165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2288241Z return mod(**inputs) 2025-08-14T22:00:03.2288512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2288599Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2288879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2288956Z outputs = layer_module( 2025-08-14T22:00:03.2289222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2289446Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2289723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2289811Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2290080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2290156Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2290433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2290532Z output = self.activation_function(output) 2025-08-14T22:00:03.2290769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2290843Z return self.act(input) 2025-08-14T22:00:03.2290847Z 2025-08-14T22:00:03.2290957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2291172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2291240Z return mod(**inputs) 2025-08-14T22:00:03.2291509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2291603Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2291872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2291952Z outputs = layer_module( 2025-08-14T22:00:03.2292238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2292454Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2292729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2292829Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2293099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2293174Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2293441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2293526Z output = self.layer_2(output) 2025-08-14T22:00:03.2293530Z 2025-08-14T22:00:03.2293660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2293869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2293961Z return mod(**inputs) 2025-08-14T22:00:03.2294224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2294319Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2294579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2294648Z outputs = layer_module( 2025-08-14T22:00:03.2294915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2294987Z outputs = self.rel_attn( 2025-08-14T22:00:03.2295256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2295360Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2295364Z 2025-08-14T22:00:03.2295473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2295687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2295754Z return mod(**inputs) 2025-08-14T22:00:03.2296015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2296108Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2296370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2296446Z outputs = layer_module( 2025-08-14T22:00:03.2296706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2296781Z outputs = self.rel_attn( 2025-08-14T22:00:03.2297048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2297157Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2297160Z 2025-08-14T22:00:03.2297272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2297485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2297553Z return mod(**inputs) 2025-08-14T22:00:03.2297823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2297908Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2298168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2298244Z outputs = layer_module( 2025-08-14T22:00:03.2298528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2298605Z outputs = self.rel_attn( 2025-08-14T22:00:03.2298871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2298974Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2299265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2299405Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2299408Z 2025-08-14T22:00:03.2299524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2299742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2299806Z return mod(**inputs) 2025-08-14T22:00:03.2300081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2300165Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2300434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2300513Z outputs = layer_module( 2025-08-14T22:00:03.2300764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2300838Z outputs = self.rel_attn( 2025-08-14T22:00:03.2301090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2301224Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2301227Z 2025-08-14T22:00:03.2301335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2301533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2301602Z return mod(**inputs) 2025-08-14T22:00:03.2301880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2301966Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2302243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2302343Z outputs = layer_module( 2025-08-14T22:00:03.2302613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2302692Z outputs = self.rel_attn( 2025-08-14T22:00:03.2302955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2303038Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2303323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2303462Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2303465Z 2025-08-14T22:00:03.2303580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2303789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2303859Z return mod(**inputs) 2025-08-14T22:00:03.2304132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2304218Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2304488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2304558Z outputs = layer_module( 2025-08-14T22:00:03.2304845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2304923Z outputs = self.rel_attn( 2025-08-14T22:00:03.2305189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2305321Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2305325Z 2025-08-14T22:00:03.2305429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2305636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2305713Z return mod(**inputs) 2025-08-14T22:00:03.2305975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2306063Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2306352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2306426Z outputs = layer_module( 2025-08-14T22:00:03.2306710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2306779Z outputs = self.rel_attn( 2025-08-14T22:00:03.2307026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2307107Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2307371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2307504Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2307508Z 2025-08-14T22:00:03.2307610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2307808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2307883Z return mod(**inputs) 2025-08-14T22:00:03.2308135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2308219Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2308476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2308542Z outputs = layer_module( 2025-08-14T22:00:03.2308968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2309042Z outputs = self.rel_attn( 2025-08-14T22:00:03.2309296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2309393Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2309680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2309802Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2309813Z 2025-08-14T22:00:03.2309921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2310131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2310209Z return mod(**inputs) 2025-08-14T22:00:03.2310480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2310567Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2310846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2310915Z outputs = layer_module( 2025-08-14T22:00:03.2311194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2311336Z outputs = self.rel_attn( 2025-08-14T22:00:03.2311601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2311701Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2312015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2312133Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2312144Z 2025-08-14T22:00:03.2312250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2312458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2312531Z return mod(**inputs) 2025-08-14T22:00:03.2312830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2312921Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2313218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2313289Z outputs = layer_module( 2025-08-14T22:00:03.2313561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2313776Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2314048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2314136Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2314402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2314481Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2314753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2314833Z output = self.layer_1(output) 2025-08-14T22:00:03.2314836Z 2025-08-14T22:00:03.2314951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2315162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2315229Z return mod(**inputs) 2025-08-14T22:00:03.2315501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2315588Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2315911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2315990Z outputs = layer_module( 2025-08-14T22:00:03.2316287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2316518Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2316803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2316889Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2317184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2317262Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2317552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2317645Z output = self.activation_function(output) 2025-08-14T22:00:03.2317867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2317971Z return self.act(input) 2025-08-14T22:00:03.2317975Z 2025-08-14T22:00:03.2318085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2318303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2318397Z return mod(**inputs) 2025-08-14T22:00:03.2318668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2318763Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2319035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2319104Z outputs = layer_module( 2025-08-14T22:00:03.2319379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2319621Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2319916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2319999Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2320268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2320350Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2320612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2320696Z output = self.layer_2(output) 2025-08-14T22:00:03.2320699Z 2025-08-14T22:00:03.2320810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2321019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2321099Z return mod(**inputs) 2025-08-14T22:00:03.2321367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2321453Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2321725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2321797Z outputs = layer_module( 2025-08-14T22:00:03.2322070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2322142Z outputs = self.rel_attn( 2025-08-14T22:00:03.2322408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2322521Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2322524Z 2025-08-14T22:00:03.2322634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2322846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2322913Z return mod(**inputs) 2025-08-14T22:00:03.2323179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2323277Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2323541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2323612Z outputs = layer_module( 2025-08-14T22:00:03.2323881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2323953Z outputs = self.rel_attn( 2025-08-14T22:00:03.2324220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2324347Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2324351Z 2025-08-14T22:00:03.2324457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2324671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2324758Z return mod(**inputs) 2025-08-14T22:00:03.2325028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2325113Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2325375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2325451Z outputs = layer_module( 2025-08-14T22:00:03.2325711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2325804Z outputs = self.rel_attn( 2025-08-14T22:00:03.2326075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2326151Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2326455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2326597Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2326601Z 2025-08-14T22:00:03.2326707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2326918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2326986Z return mod(**inputs) 2025-08-14T22:00:03.2327256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2327343Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2327595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2327667Z outputs = layer_module( 2025-08-14T22:00:03.2327914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2327981Z outputs = self.rel_attn( 2025-08-14T22:00:03.2328239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2328368Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2328371Z 2025-08-14T22:00:03.2328477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2328670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2328733Z return mod(**inputs) 2025-08-14T22:00:03.2328990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2329073Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2329334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2329410Z outputs = layer_module( 2025-08-14T22:00:03.2329668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2329745Z outputs = self.rel_attn( 2025-08-14T22:00:03.2330003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2330077Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2330363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2330523Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2330527Z 2025-08-14T22:00:03.2330639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2330846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2330914Z return mod(**inputs) 2025-08-14T22:00:03.2331210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2331298Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2331563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2331640Z outputs = layer_module( 2025-08-14T22:00:03.2331904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2331978Z outputs = self.rel_attn( 2025-08-14T22:00:03.2332255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2332361Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2332423Z 2025-08-14T22:00:03.2332540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2332749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2332826Z return mod(**inputs) 2025-08-14T22:00:03.2333090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2333176Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2333445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2333515Z outputs = layer_module( 2025-08-14T22:00:03.2333777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2333857Z outputs = self.rel_attn( 2025-08-14T22:00:03.2334122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2334205Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2334485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2334608Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2334611Z 2025-08-14T22:00:03.2334719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2334912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2334985Z return mod(**inputs) 2025-08-14T22:00:03.2335239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2335323Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2335582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2335647Z outputs = layer_module( 2025-08-14T22:00:03.2335896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2335972Z outputs = self.rel_attn( 2025-08-14T22:00:03.2336249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2336352Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2336634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2336752Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2336777Z 2025-08-14T22:00:03.2336891Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2337099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2337166Z return mod(**inputs) 2025-08-14T22:00:03.2337437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2337559Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2337831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2337899Z outputs = layer_module( 2025-08-14T22:00:03.2338160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2338239Z outputs = self.rel_attn( 2025-08-14T22:00:03.2338523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2338629Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2338927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2339047Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2339051Z 2025-08-14T22:00:03.2339163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2339368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2339435Z return mod(**inputs) 2025-08-14T22:00:03.2339706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2339792Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2340069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2340141Z outputs = layer_module( 2025-08-14T22:00:03.2340409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2340632Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2340908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2340994Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2341262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2341338Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2341610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2341689Z output = self.layer_1(output) 2025-08-14T22:00:03.2341693Z 2025-08-14T22:00:03.2341810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2342012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2342075Z return mod(**inputs) 2025-08-14T22:00:03.2342333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2342416Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2342665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2342739Z outputs = layer_module( 2025-08-14T22:00:03.2342988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2343198Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2343477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2343553Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2343820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2343920Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2344181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2344280Z output = self.activation_function(output) 2025-08-14T22:00:03.2344503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2344577Z return self.act(input) 2025-08-14T22:00:03.2344581Z 2025-08-14T22:00:03.2344704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2344903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2344976Z return mod(**inputs) 2025-08-14T22:00:03.2345262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2345358Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2345623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2345693Z outputs = layer_module( 2025-08-14T22:00:03.2345962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2346174Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2346447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2346534Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2346800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2346884Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2347145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2347221Z output = self.layer_2(output) 2025-08-14T22:00:03.2347224Z 2025-08-14T22:00:03.2347339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2347545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2347622Z return mod(**inputs) 2025-08-14T22:00:03.2347886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2347974Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2348244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2348313Z outputs = layer_module( 2025-08-14T22:00:03.2348576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2348656Z outputs = self.rel_attn( 2025-08-14T22:00:03.2348919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2349030Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2349033Z 2025-08-14T22:00:03.2349140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2349347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2349447Z return mod(**inputs) 2025-08-14T22:00:03.2349711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2349804Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2350066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2350159Z outputs = layer_module( 2025-08-14T22:00:03.2350427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2350498Z outputs = self.rel_attn( 2025-08-14T22:00:03.2350756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2350866Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2350870Z 2025-08-14T22:00:03.2351000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2351214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2351282Z return mod(**inputs) 2025-08-14T22:00:03.2351565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2351664Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2351935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2352005Z outputs = layer_module( 2025-08-14T22:00:03.2352277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2352348Z outputs = self.rel_attn( 2025-08-14T22:00:03.2352628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2352709Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2352997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2353146Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2353150Z 2025-08-14T22:00:03.2353259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2353473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2353541Z return mod(**inputs) 2025-08-14T22:00:03.2353810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2353903Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2354170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2354243Z outputs = layer_module( 2025-08-14T22:00:03.2354519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2354593Z outputs = self.rel_attn( 2025-08-14T22:00:03.2354866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2355007Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2355010Z 2025-08-14T22:00:03.2355120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2355338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2355406Z return mod(**inputs) 2025-08-14T22:00:03.2355683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2355772Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2356167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2356246Z outputs = layer_module( 2025-08-14T22:00:03.2356516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2356617Z outputs = self.rel_attn( 2025-08-14T22:00:03.2356904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2356982Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2357277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2357417Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2357421Z 2025-08-14T22:00:03.2357542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2357779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2357850Z return mod(**inputs) 2025-08-14T22:00:03.2358199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2358290Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2358550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2358627Z outputs = layer_module( 2025-08-14T22:00:03.2358887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2358956Z outputs = self.rel_attn( 2025-08-14T22:00:03.2359276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2359388Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2359392Z 2025-08-14T22:00:03.2359506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2359713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2359780Z return mod(**inputs) 2025-08-14T22:00:03.2360050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2360137Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2360406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2360475Z outputs = layer_module( 2025-08-14T22:00:03.2360735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2360811Z outputs = self.rel_attn( 2025-08-14T22:00:03.2361074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2361159Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2361433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2361556Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2361560Z 2025-08-14T22:00:03.2361668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2361861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2361925Z return mod(**inputs) 2025-08-14T22:00:03.2362180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2362261Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2362510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2362613Z outputs = layer_module( 2025-08-14T22:00:03.2362860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2362933Z outputs = self.rel_attn( 2025-08-14T22:00:03.2363201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2363291Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2363570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2363680Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2363684Z 2025-08-14T22:00:03.2363791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2364017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2364084Z return mod(**inputs) 2025-08-14T22:00:03.2364366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2364452Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2364701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2364777Z outputs = layer_module( 2025-08-14T22:00:03.2365023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2365100Z outputs = self.rel_attn( 2025-08-14T22:00:03.2365346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2365434Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2365712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2365825Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2365828Z 2025-08-14T22:00:03.2365935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2366131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2366196Z return mod(**inputs) 2025-08-14T22:00:03.2366452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2366535Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2366781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2366854Z outputs = layer_module( 2025-08-14T22:00:03.2367104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2367318Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2367570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2367650Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2367905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2367977Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2368228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2368300Z output = self.layer_1(output) 2025-08-14T22:00:03.2368304Z 2025-08-14T22:00:03.2368403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2368628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2368693Z return mod(**inputs) 2025-08-14T22:00:03.2368942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2369081Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2369328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2369403Z outputs = layer_module( 2025-08-14T22:00:03.2369647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2369848Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2370126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2370206Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2370479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2370551Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2370803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2370900Z output = self.activation_function(output) 2025-08-14T22:00:03.2371108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2371176Z return self.act(input) 2025-08-14T22:00:03.2371188Z 2025-08-14T22:00:03.2371290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2371484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2371559Z return mod(**inputs) 2025-08-14T22:00:03.2371809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2371893Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2372150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2372219Z outputs = layer_module( 2025-08-14T22:00:03.2372478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2372680Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2372935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2373022Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2373278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2373354Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2373628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2373704Z output = self.layer_2(output) 2025-08-14T22:00:03.2373708Z 2025-08-14T22:00:03.2373824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2374031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2374098Z return mod(**inputs) 2025-08-14T22:00:03.2374366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2374451Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2374724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2374822Z outputs = layer_module( 2025-08-14T22:00:03.2375133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2375223Z outputs = self.rel_attn( 2025-08-14T22:00:03.2375490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2375589Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2375593Z 2025-08-14T22:00:03.2375701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2375899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2375970Z return mod(**inputs) 2025-08-14T22:00:03.2376219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2376317Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2376587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2376674Z outputs = layer_module( 2025-08-14T22:00:03.2376944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2377017Z outputs = self.rel_attn( 2025-08-14T22:00:03.2377278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2377389Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2377393Z 2025-08-14T22:00:03.2377498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2377704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2377784Z return mod(**inputs) 2025-08-14T22:00:03.2378048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2378145Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2378409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2378481Z outputs = layer_module( 2025-08-14T22:00:03.2378751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2378822Z outputs = self.rel_attn( 2025-08-14T22:00:03.2379086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2379168Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2379447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2379595Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2379598Z 2025-08-14T22:00:03.2379706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2379913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2379992Z return mod(**inputs) 2025-08-14T22:00:03.2380267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2380360Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2380623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2380692Z outputs = layer_module( 2025-08-14T22:00:03.2380962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2381056Z outputs = self.rel_attn( 2025-08-14T22:00:03.2381318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2381467Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2381471Z 2025-08-14T22:00:03.2381596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2381809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2381878Z return mod(**inputs) 2025-08-14T22:00:03.2382141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2382253Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2382518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2382612Z outputs = layer_module( 2025-08-14T22:00:03.2382882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2382954Z outputs = self.rel_attn( 2025-08-14T22:00:03.2383255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2383333Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2383608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2383750Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2383754Z 2025-08-14T22:00:03.2383861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2384073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2384141Z return mod(**inputs) 2025-08-14T22:00:03.2384410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2384504Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2384767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2384845Z outputs = layer_module( 2025-08-14T22:00:03.2385104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2385175Z outputs = self.rel_attn( 2025-08-14T22:00:03.2385443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2385549Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2385553Z 2025-08-14T22:00:03.2385657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2385874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2385943Z return mod(**inputs) 2025-08-14T22:00:03.2386213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2386298Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2386564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2386642Z outputs = layer_module( 2025-08-14T22:00:03.2386906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2386977Z outputs = self.rel_attn( 2025-08-14T22:00:03.2387244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2387322Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2387631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2387762Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2387766Z 2025-08-14T22:00:03.2387873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2388104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2388175Z return mod(**inputs) 2025-08-14T22:00:03.2388445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2388531Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2388791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2388869Z outputs = layer_module( 2025-08-14T22:00:03.2389150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2389222Z outputs = self.rel_attn( 2025-08-14T22:00:03.2392881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2393030Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2393340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2393462Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2393468Z 2025-08-14T22:00:03.2393579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2393803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2393873Z return mod(**inputs) 2025-08-14T22:00:03.2394147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2394249Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2394546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2394651Z outputs = layer_module( 2025-08-14T22:00:03.2394944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2395018Z outputs = self.rel_attn( 2025-08-14T22:00:03.2395306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2395401Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2395698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2395891Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2395898Z 2025-08-14T22:00:03.2396013Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2396244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2396317Z return mod(**inputs) 2025-08-14T22:00:03.2396608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2396706Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2396988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2397059Z outputs = layer_module( 2025-08-14T22:00:03.2397348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2397570Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2397909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2397999Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2398294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2398406Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2398692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2398776Z output = self.layer_1(output) 2025-08-14T22:00:03.2398780Z 2025-08-14T22:00:03.2398890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2399104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2399181Z return mod(**inputs) 2025-08-14T22:00:03.2399523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2399613Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2399985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2400061Z outputs = layer_module( 2025-08-14T22:00:03.2400350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2400573Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2400855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2400946Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2401232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2401321Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2401611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2401708Z output = self.activation_function(output) 2025-08-14T22:00:03.2401948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2402024Z return self.act(input) 2025-08-14T22:00:03.2402028Z 2025-08-14T22:00:03.2402148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2402363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2402432Z return mod(**inputs) 2025-08-14T22:00:03.2402720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2402812Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2403097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2403179Z outputs = layer_module( 2025-08-14T22:00:03.2403467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2403701Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2403983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2404064Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2404351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2404427Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2404733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2404817Z output = self.layer_2(output) 2025-08-14T22:00:03.2404822Z 2025-08-14T22:00:03.2404934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2405176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2405245Z return mod(**inputs) 2025-08-14T22:00:03.2405514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2405612Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2405882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2405961Z outputs = layer_module( 2025-08-14T22:00:03.2406231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2406307Z outputs = self.rel_attn( 2025-08-14T22:00:03.2406610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2406755Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2406760Z 2025-08-14T22:00:03.2406872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2407098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2407168Z return mod(**inputs) 2025-08-14T22:00:03.2407458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2407544Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2407806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2407885Z outputs = layer_module( 2025-08-14T22:00:03.2408149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2408229Z outputs = self.rel_attn( 2025-08-14T22:00:03.2408493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2408601Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2408605Z 2025-08-14T22:00:03.2408877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2409091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2409159Z return mod(**inputs) 2025-08-14T22:00:03.2409431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2409521Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2409792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2409865Z outputs = layer_module( 2025-08-14T22:00:03.2410129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2410211Z outputs = self.rel_attn( 2025-08-14T22:00:03.2410471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2410548Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2410838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2410976Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2410980Z 2025-08-14T22:00:03.2411141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2411346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2411419Z return mod(**inputs) 2025-08-14T22:00:03.2411692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2411824Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2412094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2412163Z outputs = layer_module( 2025-08-14T22:00:03.2412424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2412501Z outputs = self.rel_attn( 2025-08-14T22:00:03.2412766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2412905Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2412917Z 2025-08-14T22:00:03.2413052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2413289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2413367Z return mod(**inputs) 2025-08-14T22:00:03.2413629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2413716Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2413992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2414063Z outputs = layer_module( 2025-08-14T22:00:03.2414331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2414404Z outputs = self.rel_attn( 2025-08-14T22:00:03.2414663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2414749Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2415031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2415168Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2415179Z 2025-08-14T22:00:03.2415286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2415493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2415568Z return mod(**inputs) 2025-08-14T22:00:03.2415832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2415920Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2416191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2416263Z outputs = layer_module( 2025-08-14T22:00:03.2416532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2416604Z outputs = self.rel_attn( 2025-08-14T22:00:03.2416864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2416976Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2416979Z 2025-08-14T22:00:03.2417085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2417295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2417369Z return mod(**inputs) 2025-08-14T22:00:03.2417656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2417751Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2418020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2418112Z outputs = layer_module( 2025-08-14T22:00:03.2418385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2418456Z outputs = self.rel_attn( 2025-08-14T22:00:03.2418730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2418805Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2419087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2419225Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2419229Z 2025-08-14T22:00:03.2419336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2419560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2419671Z return mod(**inputs) 2025-08-14T22:00:03.2419937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2420031Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2420293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2420362Z outputs = layer_module( 2025-08-14T22:00:03.2420633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2420703Z outputs = self.rel_attn( 2025-08-14T22:00:03.2420967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2421070Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2421357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2421482Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2421486Z 2025-08-14T22:00:03.2421591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2421797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2421872Z return mod(**inputs) 2025-08-14T22:00:03.2422135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2422228Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2422494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2422562Z outputs = layer_module( 2025-08-14T22:00:03.2422836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2422907Z outputs = self.rel_attn( 2025-08-14T22:00:03.2423167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2423269Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2423553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2423685Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2423688Z 2025-08-14T22:00:03.2423787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2424004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2424074Z return mod(**inputs) 2025-08-14T22:00:03.2424321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2424426Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2424672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2424738Z outputs = layer_module( 2025-08-14T22:00:03.2425007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2425222Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2425501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2425585Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2425868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2425952Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2426232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2426311Z output = self.layer_1(output) 2025-08-14T22:00:03.2426315Z 2025-08-14T22:00:03.2426430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2426636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2426720Z return mod(**inputs) 2025-08-14T22:00:03.2426972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2427055Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2427311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2427379Z outputs = layer_module( 2025-08-14T22:00:03.2427628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2427837Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2428110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2428201Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2428467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2428542Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2428817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2428911Z output = self.activation_function(output) 2025-08-14T22:00:03.2429143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2429220Z return self.act(input) 2025-08-14T22:00:03.2429223Z 2025-08-14T22:00:03.2429329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2429546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2429616Z return mod(**inputs) 2025-08-14T22:00:03.2429884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2429975Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2430239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2430337Z outputs = layer_module( 2025-08-14T22:00:03.2430606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2430819Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2431122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2431204Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2431480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2431556Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2431821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2431908Z output = self.layer_2(output) 2025-08-14T22:00:03.2431912Z 2025-08-14T22:00:03.2432019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2432246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2432323Z return mod(**inputs) 2025-08-14T22:00:03.2432606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2432702Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2432968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2433036Z outputs = layer_module( 2025-08-14T22:00:03.2433304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2433377Z outputs = self.rel_attn( 2025-08-14T22:00:03.2433645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2433749Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2433755Z 2025-08-14T22:00:03.2433861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2434078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2434148Z return mod(**inputs) 2025-08-14T22:00:03.2434410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2434503Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2434764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2434840Z outputs = layer_module( 2025-08-14T22:00:03.2435100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2435173Z outputs = self.rel_attn( 2025-08-14T22:00:03.2435442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2435550Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2435555Z 2025-08-14T22:00:03.2435669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2436180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2436256Z return mod(**inputs) 2025-08-14T22:00:03.2436528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2436616Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2436886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2437016Z outputs = layer_module( 2025-08-14T22:00:03.2437287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2437369Z outputs = self.rel_attn( 2025-08-14T22:00:03.2437644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2437738Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2438016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2438148Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2438151Z 2025-08-14T22:00:03.2438262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2438462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2438527Z return mod(**inputs) 2025-08-14T22:00:03.2438792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2438898Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2439182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2439264Z outputs = layer_module( 2025-08-14T22:00:03.2439528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2439607Z outputs = self.rel_attn( 2025-08-14T22:00:03.2439870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2440012Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2440016Z 2025-08-14T22:00:03.2440132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2440339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2440407Z return mod(**inputs) 2025-08-14T22:00:03.2440689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2440779Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2441051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2441120Z outputs = layer_module( 2025-08-14T22:00:03.2441383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2441461Z outputs = self.rel_attn( 2025-08-14T22:00:03.2441724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2441808Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2442092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2442229Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2442234Z 2025-08-14T22:00:03.2442348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2442555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2442623Z return mod(**inputs) 2025-08-14T22:00:03.2442899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2442987Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2443258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2443350Z outputs = layer_module( 2025-08-14T22:00:03.2443616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2443695Z outputs = self.rel_attn( 2025-08-14T22:00:03.2443962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2444090Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2444094Z 2025-08-14T22:00:03.2444199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2444406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2444483Z return mod(**inputs) 2025-08-14T22:00:03.2444749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2444836Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2445112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2445180Z outputs = layer_module( 2025-08-14T22:00:03.2445481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2445555Z outputs = self.rel_attn( 2025-08-14T22:00:03.2445818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2445901Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2446178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2446315Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2446319Z 2025-08-14T22:00:03.2446425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2446633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2446709Z return mod(**inputs) 2025-08-14T22:00:03.2446975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2447063Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2447336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2447407Z outputs = layer_module( 2025-08-14T22:00:03.2447677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2447750Z outputs = self.rel_attn( 2025-08-14T22:00:03.2448010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2448116Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2448401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2448518Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2448529Z 2025-08-14T22:00:03.2448641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2448850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2448925Z return mod(**inputs) 2025-08-14T22:00:03.2449189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2449276Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2449550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2449647Z outputs = layer_module( 2025-08-14T22:00:03.2449920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2449992Z outputs = self.rel_attn( 2025-08-14T22:00:03.2450267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2450385Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2450666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2450789Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2450800Z 2025-08-14T22:00:03.2450898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2451093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2451165Z return mod(**inputs) 2025-08-14T22:00:03.2451417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2451498Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2451788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2451858Z outputs = layer_module( 2025-08-14T22:00:03.2452111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2452314Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2452572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2452657Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2452913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2452984Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2453232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2453304Z output = self.layer_1(output) 2025-08-14T22:00:03.2453308Z 2025-08-14T22:00:03.2453416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2453607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2453670Z return mod(**inputs) 2025-08-14T22:00:03.2453922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2454002Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2454256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2454324Z outputs = layer_module( 2025-08-14T22:00:03.2454569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2454783Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2455047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2455122Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2455378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2455449Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2455702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2455790Z output = self.activation_function(output) 2025-08-14T22:00:03.2456026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2456102Z return self.act(input) 2025-08-14T22:00:03.2456107Z 2025-08-14T22:00:03.2456213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2456434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2456499Z return mod(**inputs) 2025-08-14T22:00:03.2456744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2456831Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2457079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2457144Z outputs = layer_module( 2025-08-14T22:00:03.2457396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2457598Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2457903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2457983Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2458233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2458313Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2458564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2458642Z output = self.layer_2(output) 2025-08-14T22:00:03.2458646Z 2025-08-14T22:00:03.2458749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2458945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2459016Z return mod(**inputs) 2025-08-14T22:00:03.2459272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2459353Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2459617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2459680Z outputs = layer_module( 2025-08-14T22:00:03.2459932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2459998Z outputs = self.rel_attn( 2025-08-14T22:00:03.2460239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2460342Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2460348Z 2025-08-14T22:00:03.2460444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2460643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2460706Z return mod(**inputs) 2025-08-14T22:00:03.2460949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2461035Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2461277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2461339Z outputs = layer_module( 2025-08-14T22:00:03.2461589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2461655Z outputs = self.rel_attn( 2025-08-14T22:00:03.2461902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2462020Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2462024Z 2025-08-14T22:00:03.2462125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2462329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2462417Z return mod(**inputs) 2025-08-14T22:00:03.2462675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2462757Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2463006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2463080Z outputs = layer_module( 2025-08-14T22:00:03.2463328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2463397Z outputs = self.rel_attn( 2025-08-14T22:00:03.2463727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2463800Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2464087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2464215Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2464218Z 2025-08-14T22:00:03.2464318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2464515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2464580Z return mod(**inputs) 2025-08-14T22:00:03.2464825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2464914Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2465157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2465229Z outputs = layer_module( 2025-08-14T22:00:03.2465472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2465540Z outputs = self.rel_attn( 2025-08-14T22:00:03.2465796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2465927Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2465930Z 2025-08-14T22:00:03.2466037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2466233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2466300Z return mod(**inputs) 2025-08-14T22:00:03.2466550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2466633Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2466881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2466955Z outputs = layer_module( 2025-08-14T22:00:03.2467203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2467278Z outputs = self.rel_attn( 2025-08-14T22:00:03.2467525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2467595Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2467866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2468012Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2468016Z 2025-08-14T22:00:03.2468125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2468324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2468409Z return mod(**inputs) 2025-08-14T22:00:03.2468671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2468752Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2469008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2469079Z outputs = layer_module( 2025-08-14T22:00:03.2469334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2469410Z outputs = self.rel_attn( 2025-08-14T22:00:03.2469679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2469780Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2469833Z 2025-08-14T22:00:03.2469943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2470137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2470207Z return mod(**inputs) 2025-08-14T22:00:03.2470455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2470543Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2470809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2470880Z outputs = layer_module( 2025-08-14T22:00:03.2471138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2471218Z outputs = self.rel_attn( 2025-08-14T22:00:03.2471483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2471566Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2471844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2471973Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2471978Z 2025-08-14T22:00:03.2472091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2472296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2472371Z return mod(**inputs) 2025-08-14T22:00:03.2472639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2472724Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2472997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2473068Z outputs = layer_module( 2025-08-14T22:00:03.2473330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2473408Z outputs = self.rel_attn( 2025-08-14T22:00:03.2473670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2473767Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2474049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2474187Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2474191Z 2025-08-14T22:00:03.2474304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2474514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2474599Z return mod(**inputs) 2025-08-14T22:00:03.2474869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2474956Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2475224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2475292Z outputs = layer_module( 2025-08-14T22:00:03.2475552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2475632Z outputs = self.rel_attn( 2025-08-14T22:00:03.2475978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2476108Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2476411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2476533Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2476537Z 2025-08-14T22:00:03.2476655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2476871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2476943Z return mod(**inputs) 2025-08-14T22:00:03.2477222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2477321Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2477579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2477647Z outputs = layer_module( 2025-08-14T22:00:03.2477897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2478111Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2478369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2478453Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2478705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2478776Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2479034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2479107Z output = self.layer_1(output) 2025-08-14T22:00:03.2479110Z 2025-08-14T22:00:03.2479212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2479419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2479484Z return mod(**inputs) 2025-08-14T22:00:03.2479743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2479825Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2480076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2480148Z outputs = layer_module( 2025-08-14T22:00:03.2480396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2480627Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2480886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2480982Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2481243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2481317Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2481568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2481664Z output = self.activation_function(output) 2025-08-14T22:00:03.2481875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2481951Z return self.act(input) 2025-08-14T22:00:03.2481954Z 2025-08-14T22:00:03.2482056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2482287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2482365Z return mod(**inputs) 2025-08-14T22:00:03.2482636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2482729Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2482977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2483042Z outputs = layer_module( 2025-08-14T22:00:03.2483297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2483499Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2483757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2483841Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2484094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2484176Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2484423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2484495Z output = self.layer_2(output) 2025-08-14T22:00:03.2484498Z 2025-08-14T22:00:03.2484609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2484803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2484875Z return mod(**inputs) 2025-08-14T22:00:03.2485134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2485220Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2485497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2485568Z outputs = layer_module( 2025-08-14T22:00:03.2485828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2485908Z outputs = self.rel_attn( 2025-08-14T22:00:03.2486171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2486284Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2486288Z 2025-08-14T22:00:03.2486395Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2486600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2486704Z return mod(**inputs) 2025-08-14T22:00:03.2486970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2487065Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2487355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2487426Z outputs = layer_module( 2025-08-14T22:00:03.2487694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2487765Z outputs = self.rel_attn( 2025-08-14T22:00:03.2488088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2488206Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2488211Z 2025-08-14T22:00:03.2488320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2488569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2488641Z return mod(**inputs) 2025-08-14T22:00:03.2488950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2489048Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2489319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2489390Z outputs = layer_module( 2025-08-14T22:00:03.2489667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2489739Z outputs = self.rel_attn( 2025-08-14T22:00:03.2490014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2490093Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2490383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2490532Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2490537Z 2025-08-14T22:00:03.2490649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2490868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2490938Z return mod(**inputs) 2025-08-14T22:00:03.2491211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2491306Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2491576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2491649Z outputs = layer_module( 2025-08-14T22:00:03.2491925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2491998Z outputs = self.rel_attn( 2025-08-14T22:00:03.2492275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2492418Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2492421Z 2025-08-14T22:00:03.2492530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2492751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2492821Z return mod(**inputs) 2025-08-14T22:00:03.2493101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2493218Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2493492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2493572Z outputs = layer_module( 2025-08-14T22:00:03.2493840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2493932Z outputs = self.rel_attn( 2025-08-14T22:00:03.2494219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2494296Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2494601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2494742Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2494748Z 2025-08-14T22:00:03.2494858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2495094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2495183Z return mod(**inputs) 2025-08-14T22:00:03.2495475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2495567Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2495834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2495917Z outputs = layer_module( 2025-08-14T22:00:03.2496182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2496255Z outputs = self.rel_attn( 2025-08-14T22:00:03.2496526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2496637Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2496641Z 2025-08-14T22:00:03.2496760Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2496973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2497048Z return mod(**inputs) 2025-08-14T22:00:03.2497321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2497410Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2497682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2497755Z outputs = layer_module( 2025-08-14T22:00:03.2498030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2498106Z outputs = self.rel_attn( 2025-08-14T22:00:03.2498347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2498420Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2498694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2498816Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2498820Z 2025-08-14T22:00:03.2498930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2499122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2499188Z return mod(**inputs) 2025-08-14T22:00:03.2499442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2499549Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2499791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2499865Z outputs = layer_module( 2025-08-14T22:00:03.2500106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2500195Z outputs = self.rel_attn( 2025-08-14T22:00:03.2500436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2500522Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2500788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2500896Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2500900Z 2025-08-14T22:00:03.2501007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2501195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2501259Z return mod(**inputs) 2025-08-14T22:00:03.2501569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2501651Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2501902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2501975Z outputs = layer_module( 2025-08-14T22:00:03.2502222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2502300Z outputs = self.rel_attn( 2025-08-14T22:00:03.2502561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2502656Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2502948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2503064Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2503069Z 2025-08-14T22:00:03.2503184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2503390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2503458Z return mod(**inputs) 2025-08-14T22:00:03.2503731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2503818Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2504078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2504155Z outputs = layer_module( 2025-08-14T22:00:03.2504417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2504642Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2504922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2505003Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2505275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2505352Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2505623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2505695Z output = self.layer_1(output) 2025-08-14T22:00:03.2505720Z 2025-08-14T22:00:03.2505821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2506021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2506088Z return mod(**inputs) 2025-08-14T22:00:03.2506335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2506448Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2506693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2506770Z outputs = layer_module( 2025-08-14T22:00:03.2507030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2507242Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2507523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2507598Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2507892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2507968Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2508215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2508308Z output = self.activation_function(output) 2025-08-14T22:00:03.2508518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2508585Z return self.act(input) 2025-08-14T22:00:03.2508596Z 2025-08-14T22:00:03.2508878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2509087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2509164Z return mod(**inputs) 2025-08-14T22:00:03.2509433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2509521Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2509799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2509868Z outputs = layer_module( 2025-08-14T22:00:03.2510148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2510363Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2510642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2510733Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2511000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2511076Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2511353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2511433Z output = self.layer_2(output) 2025-08-14T22:00:03.2511437Z 2025-08-14T22:00:03.2511557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2511767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2511837Z return mod(**inputs) 2025-08-14T22:00:03.2512111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2512199Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2512526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2512597Z outputs = layer_module( 2025-08-14T22:00:03.2512860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2512971Z outputs = self.rel_attn( 2025-08-14T22:00:03.2513235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2513337Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2513341Z 2025-08-14T22:00:03.2513456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2513667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2513746Z return mod(**inputs) 2025-08-14T22:00:03.2514021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2514112Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2514427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2514531Z outputs = layer_module( 2025-08-14T22:00:03.2514807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2514888Z outputs = self.rel_attn( 2025-08-14T22:00:03.2515159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2515274Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2515278Z 2025-08-14T22:00:03.2515387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2515599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2515677Z return mod(**inputs) 2025-08-14T22:00:03.2516021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2516124Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2516397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2516470Z outputs = layer_module( 2025-08-14T22:00:03.2516746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2516819Z outputs = self.rel_attn( 2025-08-14T22:00:03.2517086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2517171Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2517459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2517606Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2517611Z 2025-08-14T22:00:03.2517721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2517937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2518016Z return mod(**inputs) 2025-08-14T22:00:03.2518285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2518382Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2518651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2518722Z outputs = layer_module( 2025-08-14T22:00:03.2518996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2519108Z outputs = self.rel_attn( 2025-08-14T22:00:03.2519383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2519535Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2519555Z 2025-08-14T22:00:03.2519666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2519884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2519954Z return mod(**inputs) 2025-08-14T22:00:03.2520224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2520319Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2520593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2520671Z outputs = layer_module( 2025-08-14T22:00:03.2520955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2521029Z outputs = self.rel_attn( 2025-08-14T22:00:03.2521326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2521397Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2521656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2521787Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2521790Z 2025-08-14T22:00:03.2521886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2522080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2522143Z return mod(**inputs) 2025-08-14T22:00:03.2522451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2522537Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2522779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2522850Z outputs = layer_module( 2025-08-14T22:00:03.2523089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2523153Z outputs = self.rel_attn( 2025-08-14T22:00:03.2523402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2523498Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2523503Z 2025-08-14T22:00:03.2523600Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2523801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2523866Z return mod(**inputs) 2025-08-14T22:00:03.2524125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2524207Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2524455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2524527Z outputs = layer_module( 2025-08-14T22:00:03.2524775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2524842Z outputs = self.rel_attn( 2025-08-14T22:00:03.2525097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2525190Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2525469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2525598Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2525635Z 2025-08-14T22:00:03.2525737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2525939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2526006Z return mod(**inputs) 2025-08-14T22:00:03.2526264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2526347Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2526602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2526681Z outputs = layer_module( 2025-08-14T22:00:03.2526935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2527025Z outputs = self.rel_attn( 2025-08-14T22:00:03.2527299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2527390Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2527668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2527779Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2527783Z 2025-08-14T22:00:03.2527884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2528089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2528156Z return mod(**inputs) 2025-08-14T22:00:03.2528414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2528498Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2528752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2528829Z outputs = layer_module( 2025-08-14T22:00:03.2529078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2529148Z outputs = self.rel_attn( 2025-08-14T22:00:03.2529418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2529511Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2529801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2529926Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2529930Z 2025-08-14T22:00:03.2530032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2530238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2530304Z return mod(**inputs) 2025-08-14T22:00:03.2530559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2530645Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2530912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2530987Z outputs = layer_module( 2025-08-14T22:00:03.2531269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2531505Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2531774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2531882Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2532132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2532202Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2532443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2532520Z output = self.layer_1(output) 2025-08-14T22:00:03.2532524Z 2025-08-14T22:00:03.2532623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2532821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2532885Z return mod(**inputs) 2025-08-14T22:00:03.2533145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2533236Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2533502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2533571Z outputs = layer_module( 2025-08-14T22:00:03.2533825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2534027Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2534293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2534371Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2534622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2534702Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2534952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2535048Z output = self.activation_function(output) 2025-08-14T22:00:03.2535258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2535326Z return self.act(input) 2025-08-14T22:00:03.2535329Z 2025-08-14T22:00:03.2535440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2535635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2535699Z return mod(**inputs) 2025-08-14T22:00:03.2535957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2536040Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2536296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2536365Z outputs = layer_module( 2025-08-14T22:00:03.2536610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2536818Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2537078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2537158Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2537407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2537498Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2537756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2537827Z output = self.layer_2(output) 2025-08-14T22:00:03.2537851Z 2025-08-14T22:00:03.2537952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2538150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2538214Z return mod(**inputs) 2025-08-14T22:00:03.2538468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2538550Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2538797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2538873Z outputs = layer_module( 2025-08-14T22:00:03.2539120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2539212Z outputs = self.rel_attn( 2025-08-14T22:00:03.2539480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2539580Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2539584Z 2025-08-14T22:00:03.2539692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2539887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2539950Z return mod(**inputs) 2025-08-14T22:00:03.2540212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2540289Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2540540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2540607Z outputs = layer_module( 2025-08-14T22:00:03.2540855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2540934Z outputs = self.rel_attn( 2025-08-14T22:00:03.2541180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2541283Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2541294Z 2025-08-14T22:00:03.2541394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2541588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2541659Z return mod(**inputs) 2025-08-14T22:00:03.2541905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2541990Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2542245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2542314Z outputs = layer_module( 2025-08-14T22:00:03.2542570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2542636Z outputs = self.rel_attn( 2025-08-14T22:00:03.2542882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2542962Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2543227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2543357Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2543392Z 2025-08-14T22:00:03.2543494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2543690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2543762Z return mod(**inputs) 2025-08-14T22:00:03.2544034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2544117Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2544370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2544436Z outputs = layer_module( 2025-08-14T22:00:03.2544697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2544764Z outputs = self.rel_attn( 2025-08-14T22:00:03.2545011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2545147Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2545166Z 2025-08-14T22:00:03.2545269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2545481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2545553Z return mod(**inputs) 2025-08-14T22:00:03.2545800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2545888Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2546133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2546197Z outputs = layer_module( 2025-08-14T22:00:03.2546452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2546521Z outputs = self.rel_attn( 2025-08-14T22:00:03.2546775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2546849Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2547113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2547245Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2547248Z 2025-08-14T22:00:03.2547347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2547540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2547612Z return mod(**inputs) 2025-08-14T22:00:03.2547858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2547948Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2548195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2548263Z outputs = layer_module( 2025-08-14T22:00:03.2548517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2548583Z outputs = self.rel_attn( 2025-08-14T22:00:03.2548838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2548937Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2548941Z 2025-08-14T22:00:03.2549042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2549240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2549327Z return mod(**inputs) 2025-08-14T22:00:03.2549583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2549674Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2549932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2550023Z outputs = layer_module( 2025-08-14T22:00:03.2550271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2550336Z outputs = self.rel_attn( 2025-08-14T22:00:03.2550587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2550656Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2550922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2551054Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2551058Z 2025-08-14T22:00:03.2551181Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2551424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2551496Z return mod(**inputs) 2025-08-14T22:00:03.2551761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2551855Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2552121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2552196Z outputs = layer_module( 2025-08-14T22:00:03.2552457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2552529Z outputs = self.rel_attn( 2025-08-14T22:00:03.2552798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2552891Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2553178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2553302Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2553305Z 2025-08-14T22:00:03.2553411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2553626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2553695Z return mod(**inputs) 2025-08-14T22:00:03.2553957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2554053Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2554317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2554395Z outputs = layer_module( 2025-08-14T22:00:03.2554658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2554730Z outputs = self.rel_attn( 2025-08-14T22:00:03.2555000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2555093Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2555382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2555504Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2555532Z 2025-08-14T22:00:03.2555637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2555930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2556009Z return mod(**inputs) 2025-08-14T22:00:03.2556276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2556398Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2556662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2556744Z outputs = layer_module( 2025-08-14T22:00:03.2557013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2557238Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2557528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2557620Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2557925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2558018Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2558285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2558370Z output = self.layer_1(output) 2025-08-14T22:00:03.2558374Z 2025-08-14T22:00:03.2558483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2558689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2558766Z return mod(**inputs) 2025-08-14T22:00:03.2559032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2559128Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2559400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2559473Z outputs = layer_module( 2025-08-14T22:00:03.2559746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2559962Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2560240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2560327Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2560595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2560681Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2560943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2561037Z output = self.activation_function(output) 2025-08-14T22:00:03.2561270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2561344Z return self.act(input) 2025-08-14T22:00:03.2561347Z 2025-08-14T22:00:03.2561462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2561672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2561741Z return mod(**inputs) 2025-08-14T22:00:03.2562011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2562096Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2562390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2562466Z outputs = layer_module( 2025-08-14T22:00:03.2562728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2562967Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2563237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2563320Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2563592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2563666Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2563935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2564013Z output = self.layer_2(output) 2025-08-14T22:00:03.2564017Z 2025-08-14T22:00:03.2564145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2564380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2564453Z return mod(**inputs) 2025-08-14T22:00:03.2564719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2564811Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2565078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2565154Z outputs = layer_module( 2025-08-14T22:00:03.2565419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2565492Z outputs = self.rel_attn( 2025-08-14T22:00:03.2565776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2565873Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2565876Z 2025-08-14T22:00:03.2565984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2566175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2566236Z return mod(**inputs) 2025-08-14T22:00:03.2566488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2566568Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2566814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2566889Z outputs = layer_module( 2025-08-14T22:00:03.2567133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2567204Z outputs = self.rel_attn( 2025-08-14T22:00:03.2567450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2567548Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2567552Z 2025-08-14T22:00:03.2567656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2567847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2567910Z return mod(**inputs) 2025-08-14T22:00:03.2568160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2568240Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2568508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2568572Z outputs = layer_module( 2025-08-14T22:00:03.2568812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2568903Z outputs = self.rel_attn( 2025-08-14T22:00:03.2569142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2569219Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2569474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2569600Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2569603Z 2025-08-14T22:00:03.2569708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2569900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2569963Z return mod(**inputs) 2025-08-14T22:00:03.2570253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2570352Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2570604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2570671Z outputs = layer_module( 2025-08-14T22:00:03.2570916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2570991Z outputs = self.rel_attn( 2025-08-14T22:00:03.2571236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2571373Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2571378Z 2025-08-14T22:00:03.2571478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2571672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2571746Z return mod(**inputs) 2025-08-14T22:00:03.2571995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2572076Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2572333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2572400Z outputs = layer_module( 2025-08-14T22:00:03.2572655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2572721Z outputs = self.rel_attn( 2025-08-14T22:00:03.2572968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2573049Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2573317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2573454Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2573458Z 2025-08-14T22:00:03.2573561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2573761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2573832Z return mod(**inputs) 2025-08-14T22:00:03.2574080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2574160Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2574415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2574504Z outputs = layer_module( 2025-08-14T22:00:03.2574758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2574827Z outputs = self.rel_attn( 2025-08-14T22:00:03.2575088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2575196Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2575200Z 2025-08-14T22:00:03.2575299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2575500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2575564Z return mod(**inputs) 2025-08-14T22:00:03.2575811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2575901Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2576174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2576241Z outputs = layer_module( 2025-08-14T22:00:03.2576514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2576586Z outputs = self.rel_attn( 2025-08-14T22:00:03.2576843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2576915Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2577182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2577310Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2577315Z 2025-08-14T22:00:03.2577416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2577615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2577689Z return mod(**inputs) 2025-08-14T22:00:03.2577944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2578034Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2578287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2578353Z outputs = layer_module( 2025-08-14T22:00:03.2578610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2578677Z outputs = self.rel_attn( 2025-08-14T22:00:03.2578934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2579026Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2579301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2579422Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2579427Z 2025-08-14T22:00:03.2579527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2579725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2579797Z return mod(**inputs) 2025-08-14T22:00:03.2580049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2580141Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2580394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2580481Z outputs = layer_module( 2025-08-14T22:00:03.2580738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2580806Z outputs = self.rel_attn( 2025-08-14T22:00:03.2581095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2581181Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2581443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2581558Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2581562Z 2025-08-14T22:00:03.2581657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2581847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2581916Z return mod(**inputs) 2025-08-14T22:00:03.2582173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2582262Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2582524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2582590Z outputs = layer_module( 2025-08-14T22:00:03.2582839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2583038Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2583296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2583374Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2583615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2583695Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2583938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2584011Z output = self.layer_1(output) 2025-08-14T22:00:03.2584020Z 2025-08-14T22:00:03.2584119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2584315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2584388Z return mod(**inputs) 2025-08-14T22:00:03.2584639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2584720Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2584984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2585050Z outputs = layer_module( 2025-08-14T22:00:03.2585306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2585512Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2585770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2585854Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2586106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2586179Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2586434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2586572Z output = self.activation_function(output) 2025-08-14T22:00:03.2586789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2586858Z return self.act(input) 2025-08-14T22:00:03.2586862Z 2025-08-14T22:00:03.2586981Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2587184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2587249Z return mod(**inputs) 2025-08-14T22:00:03.2587514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2587594Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2587840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2587914Z outputs = layer_module( 2025-08-14T22:00:03.2588156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2588372Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2588650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2588726Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2588979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2589049Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2589292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2589372Z output = self.layer_2(output) 2025-08-14T22:00:03.2589377Z 2025-08-14T22:00:03.2589475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2589671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2589736Z return mod(**inputs) 2025-08-14T22:00:03.2589980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2590067Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2590315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2590381Z outputs = layer_module( 2025-08-14T22:00:03.2590634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2590705Z outputs = self.rel_attn( 2025-08-14T22:00:03.2590959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2591060Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2591063Z 2025-08-14T22:00:03.2591162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2591367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2591435Z return mod(**inputs) 2025-08-14T22:00:03.2591687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2591781Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2592043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2592120Z outputs = layer_module( 2025-08-14T22:00:03.2592383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2592471Z outputs = self.rel_attn( 2025-08-14T22:00:03.2592740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2592846Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2592849Z 2025-08-14T22:00:03.2592982Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2593188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2593256Z return mod(**inputs) 2025-08-14T22:00:03.2593528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2593614Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2593874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2593951Z outputs = layer_module( 2025-08-14T22:00:03.2594214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2594292Z outputs = self.rel_attn( 2025-08-14T22:00:03.2594583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2594663Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2594957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2595095Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2595099Z 2025-08-14T22:00:03.2595214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2595423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2595490Z return mod(**inputs) 2025-08-14T22:00:03.2595763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2595936Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2596211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2596291Z outputs = layer_module( 2025-08-14T22:00:03.2596554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2596633Z outputs = self.rel_attn( 2025-08-14T22:00:03.2596895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2597032Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2597037Z 2025-08-14T22:00:03.2597151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2597360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2597437Z return mod(**inputs) 2025-08-14T22:00:03.2597701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2597790Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2598061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2598130Z outputs = layer_module( 2025-08-14T22:00:03.2598396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2598477Z outputs = self.rel_attn( 2025-08-14T22:00:03.2598741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2598823Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2599127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2599265Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2599269Z 2025-08-14T22:00:03.2599386Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2599622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2599703Z return mod(**inputs) 2025-08-14T22:00:03.2599969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2600059Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2600330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2600401Z outputs = layer_module( 2025-08-14T22:00:03.2600668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2600750Z outputs = self.rel_attn( 2025-08-14T22:00:03.2601052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2601167Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2601171Z 2025-08-14T22:00:03.2601278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2601485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2601563Z return mod(**inputs) 2025-08-14T22:00:03.2601822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2601909Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2602180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2602253Z outputs = layer_module( 2025-08-14T22:00:03.2602521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2602594Z outputs = self.rel_attn( 2025-08-14T22:00:03.2602854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2602938Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2603214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2603351Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2603355Z 2025-08-14T22:00:03.2603461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2603668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2603746Z return mod(**inputs) 2025-08-14T22:00:03.2604012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2604098Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2604374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2604444Z outputs = layer_module( 2025-08-14T22:00:03.2604711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2604782Z outputs = self.rel_attn( 2025-08-14T22:00:03.2605043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2605146Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2605454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2605581Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2605586Z 2025-08-14T22:00:03.2605692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2605919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2605994Z return mod(**inputs) 2025-08-14T22:00:03.2606278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2606365Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2606639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2606708Z outputs = layer_module( 2025-08-14T22:00:03.2606980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2607054Z outputs = self.rel_attn( 2025-08-14T22:00:03.2607341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2607461Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2607750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2607872Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2607876Z 2025-08-14T22:00:03.2607981Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2608194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2608268Z return mod(**inputs) 2025-08-14T22:00:03.2608517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2608601Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2609012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2609085Z outputs = layer_module( 2025-08-14T22:00:03.2609341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2609551Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2609809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2609893Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2610141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2610225Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2610473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2610546Z output = self.layer_1(output) 2025-08-14T22:00:03.2610550Z 2025-08-14T22:00:03.2610663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2610862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2610928Z return mod(**inputs) 2025-08-14T22:00:03.2611185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2611269Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2611525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2611591Z outputs = layer_module( 2025-08-14T22:00:03.2611915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2612139Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2612422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2612534Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2612786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2612859Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2613124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2613215Z output = self.activation_function(output) 2025-08-14T22:00:03.2613443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2613526Z return self.act(input) 2025-08-14T22:00:03.2613530Z 2025-08-14T22:00:03.2613671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2613914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2613987Z return mod(**inputs) 2025-08-14T22:00:03.2614256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2614347Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2632737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2632953Z outputs = layer_module( 2025-08-14T22:00:03.2633281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2633531Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2633815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2633909Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2634178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2634256Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2634518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2634593Z output = self.layer_2(output) 2025-08-14T22:00:03.2634601Z 2025-08-14T22:00:03.2634724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2634931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2635003Z return mod(**inputs) 2025-08-14T22:00:03.2635265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2635356Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2635612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2635698Z outputs = layer_module( 2025-08-14T22:00:03.2636075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2636168Z outputs = self.rel_attn( 2025-08-14T22:00:03.2636453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2636566Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2636572Z 2025-08-14T22:00:03.2636808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2637035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2637119Z return mod(**inputs) 2025-08-14T22:00:03.2637412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2637548Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2637811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2637882Z outputs = layer_module( 2025-08-14T22:00:03.2638134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2638215Z outputs = self.rel_attn( 2025-08-14T22:00:03.2638464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2638579Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2638582Z 2025-08-14T22:00:03.2638697Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2638942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2639019Z return mod(**inputs) 2025-08-14T22:00:03.2639266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2639351Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2639608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2639675Z outputs = layer_module( 2025-08-14T22:00:03.2639931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2640001Z outputs = self.rel_attn( 2025-08-14T22:00:03.2640249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2640334Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2640622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2640777Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2640781Z 2025-08-14T22:00:03.2640890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2641103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2641178Z return mod(**inputs) 2025-08-14T22:00:03.2641447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2641536Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2641816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2641886Z outputs = layer_module( 2025-08-14T22:00:03.2642166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2642242Z outputs = self.rel_attn( 2025-08-14T22:00:03.2642511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2642654Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2642658Z 2025-08-14T22:00:03.2642758Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2642963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2643029Z return mod(**inputs) 2025-08-14T22:00:03.2643300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2643392Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2643643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2643729Z outputs = layer_module( 2025-08-14T22:00:03.2643988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2644055Z outputs = self.rel_attn( 2025-08-14T22:00:03.2644311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2644384Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2644650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2644789Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2644793Z 2025-08-14T22:00:03.2644893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2645115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2645202Z return mod(**inputs) 2025-08-14T22:00:03.2645459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2645550Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2645806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2645872Z outputs = layer_module( 2025-08-14T22:00:03.2646131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2646199Z outputs = self.rel_attn( 2025-08-14T22:00:03.2646455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2646558Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2646562Z 2025-08-14T22:00:03.2646663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2646872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2646939Z return mod(**inputs) 2025-08-14T22:00:03.2647192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2647284Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2647535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2647609Z outputs = layer_module( 2025-08-14T22:00:03.2647860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2647927Z outputs = self.rel_attn( 2025-08-14T22:00:03.2648187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2648262Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2648533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2648657Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2648661Z 2025-08-14T22:00:03.2648761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2648968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2649033Z return mod(**inputs) 2025-08-14T22:00:03.2649285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2649395Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2649647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2649724Z outputs = layer_module( 2025-08-14T22:00:03.2650330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2650397Z outputs = self.rel_attn( 2025-08-14T22:00:03.2650658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2650750Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2651030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2651146Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2651151Z 2025-08-14T22:00:03.2651252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2651477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2651545Z return mod(**inputs) 2025-08-14T22:00:03.2651817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2651910Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2652160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2652235Z outputs = layer_module( 2025-08-14T22:00:03.2652485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2652552Z outputs = self.rel_attn( 2025-08-14T22:00:03.2652812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2652903Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2653184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2653296Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2653302Z 2025-08-14T22:00:03.2653401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2653608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2653674Z return mod(**inputs) 2025-08-14T22:00:03.2653976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2654066Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2654317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2654392Z outputs = layer_module( 2025-08-14T22:00:03.2654644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2654853Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2655135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2655215Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2655466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2655538Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2655779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2655875Z output = self.layer_1(output) 2025-08-14T22:00:03.2655878Z 2025-08-14T22:00:03.2655979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2656172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2656261Z return mod(**inputs) 2025-08-14T22:00:03.2656500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2656590Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2656829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2656894Z outputs = layer_module( 2025-08-14T22:00:03.2657142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2657344Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2657626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2657704Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2657971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2658053Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2658304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2658393Z output = self.activation_function(output) 2025-08-14T22:00:03.2658611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2658680Z return self.act(input) 2025-08-14T22:00:03.2658683Z 2025-08-14T22:00:03.2658795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2658992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2659056Z return mod(**inputs) 2025-08-14T22:00:03.2659318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2659403Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2659672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2659738Z outputs = layer_module( 2025-08-14T22:00:03.2659979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2660184Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2660436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2660513Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2660768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2660840Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2661094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2661165Z output = self.layer_2(output) 2025-08-14T22:00:03.2661169Z 2025-08-14T22:00:03.2661269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2661467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2661532Z return mod(**inputs) 2025-08-14T22:00:03.2661786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2661886Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2662130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2662203Z outputs = layer_module( 2025-08-14T22:00:03.2662468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2662537Z outputs = self.rel_attn( 2025-08-14T22:00:03.2662786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2662883Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2662887Z 2025-08-14T22:00:03.2662991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2663182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2663248Z return mod(**inputs) 2025-08-14T22:00:03.2663499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2663594Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2663853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2663931Z outputs = layer_module( 2025-08-14T22:00:03.2664170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2664244Z outputs = self.rel_attn( 2025-08-14T22:00:03.2664488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2664589Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2664593Z 2025-08-14T22:00:03.2664701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2664896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2664967Z return mod(**inputs) 2025-08-14T22:00:03.2665219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2665304Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2665560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2665627Z outputs = layer_module( 2025-08-14T22:00:03.2665875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2665952Z outputs = self.rel_attn( 2025-08-14T22:00:03.2666199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2666283Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2666545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2666684Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2666690Z 2025-08-14T22:00:03.2666801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2666995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2667066Z return mod(**inputs) 2025-08-14T22:00:03.2667315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2667395Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2667652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2667737Z outputs = layer_module( 2025-08-14T22:00:03.2668044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2668124Z outputs = self.rel_attn( 2025-08-14T22:00:03.2668377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2668529Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2668533Z 2025-08-14T22:00:03.2668629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2668826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2668903Z return mod(**inputs) 2025-08-14T22:00:03.2669168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2669261Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2669530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2669600Z outputs = layer_module( 2025-08-14T22:00:03.2669902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2669977Z outputs = self.rel_attn( 2025-08-14T22:00:03.2670242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2670326Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2670608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2670754Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2670757Z 2025-08-14T22:00:03.2670864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2671071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2671148Z return mod(**inputs) 2025-08-14T22:00:03.2671419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2671512Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2671779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2671847Z outputs = layer_module( 2025-08-14T22:00:03.2672121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2672194Z outputs = self.rel_attn( 2025-08-14T22:00:03.2672504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2672621Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2672625Z 2025-08-14T22:00:03.2672738Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2672951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2673023Z return mod(**inputs) 2025-08-14T22:00:03.2673317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2673406Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2673691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2673771Z outputs = layer_module( 2025-08-14T22:00:03.2674063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2674146Z outputs = self.rel_attn( 2025-08-14T22:00:03.2674449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2674526Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2674827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2674983Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2674987Z 2025-08-14T22:00:03.2675104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2675316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2675385Z return mod(**inputs) 2025-08-14T22:00:03.2675673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2675760Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2676137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2676223Z outputs = layer_module( 2025-08-14T22:00:03.2676535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2676640Z outputs = self.rel_attn( 2025-08-14T22:00:03.2676926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2677025Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2677333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2677451Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2677455Z 2025-08-14T22:00:03.2677569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2677776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2677845Z return mod(**inputs) 2025-08-14T22:00:03.2678118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2678207Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2678471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2678547Z outputs = layer_module( 2025-08-14T22:00:03.2678811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2678888Z outputs = self.rel_attn( 2025-08-14T22:00:03.2679147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2679240Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2679530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2679647Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2679651Z 2025-08-14T22:00:03.2679766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2679974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2680042Z return mod(**inputs) 2025-08-14T22:00:03.2680312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2680397Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2680659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2680736Z outputs = layer_module( 2025-08-14T22:00:03.2681051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2681277Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2681552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2681659Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2681935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2682010Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2682282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2682359Z output = self.layer_1(output) 2025-08-14T22:00:03.2682362Z 2025-08-14T22:00:03.2682469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2682686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2682753Z return mod(**inputs) 2025-08-14T22:00:03.2683051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2683148Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2683411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2683488Z outputs = layer_module( 2025-08-14T22:00:03.2683747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2683960Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2684239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2684321Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2684593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2684669Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2684933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2685035Z output = self.activation_function(output) 2025-08-14T22:00:03.2685257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2685329Z return self.act(input) 2025-08-14T22:00:03.2685333Z 2025-08-14T22:00:03.2685449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2685658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2685729Z return mod(**inputs) 2025-08-14T22:00:03.2685969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2686050Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2686298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2686364Z outputs = layer_module( 2025-08-14T22:00:03.2686610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2686805Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2687054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2687135Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2687401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2687474Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2687734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2687824Z output = self.layer_2(output) 2025-08-14T22:00:03.2687828Z 2025-08-14T22:00:03.2687939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2688133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2688198Z return mod(**inputs) 2025-08-14T22:00:03.2688456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2688538Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2688795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2688865Z outputs = layer_module( 2025-08-14T22:00:03.2689131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2689226Z outputs = self.rel_attn( 2025-08-14T22:00:03.2689482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2689581Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2689584Z 2025-08-14T22:00:03.2689693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2689891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2689965Z return mod(**inputs) 2025-08-14T22:00:03.2690225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2690307Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2690560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2690624Z outputs = layer_module( 2025-08-14T22:00:03.2690870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2690946Z outputs = self.rel_attn( 2025-08-14T22:00:03.2691194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2691298Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2691301Z 2025-08-14T22:00:03.2691398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2691590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2691663Z return mod(**inputs) 2025-08-14T22:00:03.2691908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2691999Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2692251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2692317Z outputs = layer_module( 2025-08-14T22:00:03.2692574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2692640Z outputs = self.rel_attn( 2025-08-14T22:00:03.2692890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2692969Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2693236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2693396Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2693400Z 2025-08-14T22:00:03.2693501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2693696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2693793Z return mod(**inputs) 2025-08-14T22:00:03.2694039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2694124Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2694370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2694433Z outputs = layer_module( 2025-08-14T22:00:03.2694685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2694751Z outputs = self.rel_attn( 2025-08-14T22:00:03.2694998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2695153Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2695173Z 2025-08-14T22:00:03.2695283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2695491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2695555Z return mod(**inputs) 2025-08-14T22:00:03.2695805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2695893Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2696141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2696216Z outputs = layer_module( 2025-08-14T22:00:03.2696462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2696530Z outputs = self.rel_attn( 2025-08-14T22:00:03.2696786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2696859Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2697133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2697257Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2697261Z 2025-08-14T22:00:03.2697359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2697552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2697614Z return mod(**inputs) 2025-08-14T22:00:03.2697862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2697949Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2698188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2698261Z outputs = layer_module( 2025-08-14T22:00:03.2698500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2698564Z outputs = self.rel_attn( 2025-08-14T22:00:03.2698811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2698907Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2698910Z 2025-08-14T22:00:03.2699016Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2699229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2699291Z return mod(**inputs) 2025-08-14T22:00:03.2699539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2699640Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2699880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2699951Z outputs = layer_module( 2025-08-14T22:00:03.2700190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2700262Z outputs = self.rel_attn( 2025-08-14T22:00:03.2700501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2700570Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2700837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2700973Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2700976Z 2025-08-14T22:00:03.2701099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2701289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2701351Z return mod(**inputs) 2025-08-14T22:00:03.2701597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2701679Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2701916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2701986Z outputs = layer_module( 2025-08-14T22:00:03.2702229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2702300Z outputs = self.rel_attn( 2025-08-14T22:00:03.2702545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2702632Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2702897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2703006Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2703009Z 2025-08-14T22:00:03.2703111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2703297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2703360Z return mod(**inputs) 2025-08-14T22:00:03.2703608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2703688Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2703929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2704005Z outputs = layer_module( 2025-08-14T22:00:03.2704244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2704316Z outputs = self.rel_attn( 2025-08-14T22:00:03.2704556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2704642Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2704908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2705045Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2705048Z 2025-08-14T22:00:03.2705143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2705340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2705404Z return mod(**inputs) 2025-08-14T22:00:03.2705672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2705752Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2705996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2706069Z outputs = layer_module( 2025-08-14T22:00:03.2706309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2706519Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2706793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2706870Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2707140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2707215Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2707459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2707538Z output = self.layer_1(output) 2025-08-14T22:00:03.2707541Z 2025-08-14T22:00:03.2707640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2707841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2707909Z return mod(**inputs) 2025-08-14T22:00:03.2708156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2708245Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2708494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2708568Z outputs = layer_module( 2025-08-14T22:00:03.2709049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2709262Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2709539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2709616Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2709884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2709970Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2710227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2710325Z output = self.activation_function(output) 2025-08-14T22:00:03.2710530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2710598Z return self.act(input) 2025-08-14T22:00:03.2710602Z 2025-08-14T22:00:03.2710709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2710903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2710976Z return mod(**inputs) 2025-08-14T22:00:03.2711225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2711380Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2711636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2711703Z outputs = layer_module( 2025-08-14T22:00:03.2711954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2712193Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2712451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2712537Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2712790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2712863Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2713124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2713222Z output = self.layer_2(output) 2025-08-14T22:00:03.2713226Z 2025-08-14T22:00:03.2713368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2713569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2713635Z return mod(**inputs) 2025-08-14T22:00:03.2713893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2713976Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2714228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2714301Z outputs = layer_module( 2025-08-14T22:00:03.2714550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2714626Z outputs = self.rel_attn( 2025-08-14T22:00:03.2714877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2714978Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2714981Z 2025-08-14T22:00:03.2715089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2715288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2715360Z return mod(**inputs) 2025-08-14T22:00:03.2715612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2715694Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2716013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2716092Z outputs = layer_module( 2025-08-14T22:00:03.2716358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2716439Z outputs = self.rel_attn( 2025-08-14T22:00:03.2716732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2716848Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2716853Z 2025-08-14T22:00:03.2716960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2717173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2717244Z return mod(**inputs) 2025-08-14T22:00:03.2717498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2717603Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2717860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2717927Z outputs = layer_module( 2025-08-14T22:00:03.2718230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2718332Z outputs = self.rel_attn( 2025-08-14T22:00:03.2718574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2718652Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2718908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2719045Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2719051Z 2025-08-14T22:00:03.2719149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2719338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2719425Z return mod(**inputs) 2025-08-14T22:00:03.2719695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2719776Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2720022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2720085Z outputs = layer_module( 2025-08-14T22:00:03.2720329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2720394Z outputs = self.rel_attn( 2025-08-14T22:00:03.2720632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2720769Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2720772Z 2025-08-14T22:00:03.2720872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2721071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2721137Z return mod(**inputs) 2025-08-14T22:00:03.2721376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2721462Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2721701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2721764Z outputs = layer_module( 2025-08-14T22:00:03.2722010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2722075Z outputs = self.rel_attn( 2025-08-14T22:00:03.2722323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2722393Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2722650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2722783Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2722786Z 2025-08-14T22:00:03.2722885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2723081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2723143Z return mod(**inputs) 2025-08-14T22:00:03.2723388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2723495Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2723741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2723807Z outputs = layer_module( 2025-08-14T22:00:03.2724063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2724147Z outputs = self.rel_attn( 2025-08-14T22:00:03.2724394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2724491Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2724494Z 2025-08-14T22:00:03.2724591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2724788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2724849Z return mod(**inputs) 2025-08-14T22:00:03.2725104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2725184Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2725464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2725538Z outputs = layer_module( 2025-08-14T22:00:03.2725782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2725846Z outputs = self.rel_attn( 2025-08-14T22:00:03.2726095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2726165Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2726435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2726560Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2726564Z 2025-08-14T22:00:03.2726664Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2726870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2726938Z return mod(**inputs) 2025-08-14T22:00:03.2727188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2727280Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2727534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2727605Z outputs = layer_module( 2025-08-14T22:00:03.2727849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2727917Z outputs = self.rel_attn( 2025-08-14T22:00:03.2728166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2728255Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2728533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2728640Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2728643Z 2025-08-14T22:00:03.2728736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2728931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2728992Z return mod(**inputs) 2025-08-14T22:00:03.2729236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2729323Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2729594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2729666Z outputs = layer_module( 2025-08-14T22:00:03.2729917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2730000Z outputs = self.rel_attn( 2025-08-14T22:00:03.2730255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2730343Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2730614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2730723Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2730726Z 2025-08-14T22:00:03.2730825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2731028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2731091Z return mod(**inputs) 2025-08-14T22:00:03.2731361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2731467Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2731712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2731780Z outputs = layer_module( 2025-08-14T22:00:03.2732020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2732216Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2732480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2732555Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2732797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2732865Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2733102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2733176Z output = self.layer_1(output) 2025-08-14T22:00:03.2733179Z 2025-08-14T22:00:03.2733277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2733459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2733527Z return mod(**inputs) 2025-08-14T22:00:03.2733763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2733850Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2734084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2734151Z outputs = layer_module( 2025-08-14T22:00:03.2734397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2734593Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2734847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2734921Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2735163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2735239Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2735503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2735588Z output = self.activation_function(output) 2025-08-14T22:00:03.2735800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2735885Z return self.act(input) 2025-08-14T22:00:03.2735888Z 2025-08-14T22:00:03.2736003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2736190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2736251Z return mod(**inputs) 2025-08-14T22:00:03.2736499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2736579Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2736825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2736892Z outputs = layer_module( 2025-08-14T22:00:03.2737151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2737374Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2737627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2737701Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2737953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2738022Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2738273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2738344Z output = self.layer_2(output) 2025-08-14T22:00:03.2738348Z 2025-08-14T22:00:03.2738446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2738645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2738708Z return mod(**inputs) 2025-08-14T22:00:03.2738961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2739041Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2739286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2739357Z outputs = layer_module( 2025-08-14T22:00:03.2739602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2739669Z outputs = self.rel_attn( 2025-08-14T22:00:03.2739920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2740014Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2740019Z 2025-08-14T22:00:03.2740122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2740321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2740386Z return mod(**inputs) 2025-08-14T22:00:03.2740643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2740726Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2740983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2741048Z outputs = layer_module( 2025-08-14T22:00:03.2741296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2741393Z outputs = self.rel_attn( 2025-08-14T22:00:03.2741645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2741748Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2741767Z 2025-08-14T22:00:03.2741873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2742073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2742143Z return mod(**inputs) 2025-08-14T22:00:03.2742383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2742462Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2742710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2742777Z outputs = layer_module( 2025-08-14T22:00:03.2743034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2743110Z outputs = self.rel_attn( 2025-08-14T22:00:03.2743365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2743446Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2743703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2743830Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2743833Z 2025-08-14T22:00:03.2743938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2744128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2744199Z return mod(**inputs) 2025-08-14T22:00:03.2744443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2744523Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2744774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2744840Z outputs = layer_module( 2025-08-14T22:00:03.2745080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2745152Z outputs = self.rel_attn( 2025-08-14T22:00:03.2745394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2745527Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2745532Z 2025-08-14T22:00:03.2745631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2745819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2745887Z return mod(**inputs) 2025-08-14T22:00:03.2746135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2746220Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2746468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2746531Z outputs = layer_module( 2025-08-14T22:00:03.2746777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2746841Z outputs = self.rel_attn( 2025-08-14T22:00:03.2747081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2747178Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2747437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2747568Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2747590Z 2025-08-14T22:00:03.2747690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2747880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2747948Z return mod(**inputs) 2025-08-14T22:00:03.2748191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2748276Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2748518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2748583Z outputs = layer_module( 2025-08-14T22:00:03.2748831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2748918Z outputs = self.rel_attn( 2025-08-14T22:00:03.2749178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2749285Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2749289Z 2025-08-14T22:00:03.2749386Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2749581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2749643Z return mod(**inputs) 2025-08-14T22:00:03.2749890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2749978Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2750225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2750291Z outputs = layer_module( 2025-08-14T22:00:03.2750546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2750614Z outputs = self.rel_attn( 2025-08-14T22:00:03.2750867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2750938Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2751197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2751327Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2751330Z 2025-08-14T22:00:03.2751430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2751634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2751697Z return mod(**inputs) 2025-08-14T22:00:03.2751951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2752041Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2752288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2752355Z outputs = layer_module( 2025-08-14T22:00:03.2752610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2752678Z outputs = self.rel_attn( 2025-08-14T22:00:03.2752930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2753039Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2753309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2753431Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2753452Z 2025-08-14T22:00:03.2753553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2753755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2753819Z return mod(**inputs) 2025-08-14T22:00:03.2754068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2754156Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2754404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2754477Z outputs = layer_module( 2025-08-14T22:00:03.2754745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2754850Z outputs = self.rel_attn( 2025-08-14T22:00:03.2755138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2755235Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2755515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2755638Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2755642Z 2025-08-14T22:00:03.2755748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2756050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2756119Z return mod(**inputs) 2025-08-14T22:00:03.2756368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2756461Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2756721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2756792Z outputs = layer_module( 2025-08-14T22:00:03.2757062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2757276Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2757560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2757642Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2757912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2757995Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2758242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2758325Z output = self.layer_1(output) 2025-08-14T22:00:03.2758331Z 2025-08-14T22:00:03.2758432Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2758630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2758702Z return mod(**inputs) 2025-08-14T22:00:03.2758949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2759030Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2759296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2759383Z outputs = layer_module( 2025-08-14T22:00:03.2759635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2759833Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2760100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2760182Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2760426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2760503Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2760748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2760837Z output = self.activation_function(output) 2025-08-14T22:00:03.2761063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2761129Z return self.act(input) 2025-08-14T22:00:03.2761147Z 2025-08-14T22:00:03.2761246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2761459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2761524Z return mod(**inputs) 2025-08-14T22:00:03.2761771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2761850Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2762090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2762159Z outputs = layer_module( 2025-08-14T22:00:03.2762398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2762603Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2762856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2762931Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2763178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2763248Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2763490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2763568Z output = self.layer_2(output) 2025-08-14T22:00:03.2763571Z 2025-08-14T22:00:03.2763667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2763866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2763928Z return mod(**inputs) 2025-08-14T22:00:03.2764173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2764263Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2764503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2764574Z outputs = layer_module( 2025-08-14T22:00:03.2764814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2764881Z outputs = self.rel_attn( 2025-08-14T22:00:03.2765127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2765222Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2765241Z 2025-08-14T22:00:03.2765339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2765537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2765601Z return mod(**inputs) 2025-08-14T22:00:03.2765873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2765952Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2766196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2766268Z outputs = layer_module( 2025-08-14T22:00:03.2766510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2766576Z outputs = self.rel_attn( 2025-08-14T22:00:03.2766826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2766922Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2766926Z 2025-08-14T22:00:03.2767049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2767278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2767347Z return mod(**inputs) 2025-08-14T22:00:03.2767625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2767712Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2767981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2768049Z outputs = layer_module( 2025-08-14T22:00:03.2768314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2768404Z outputs = self.rel_attn( 2025-08-14T22:00:03.2768653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2768728Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2769010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2769139Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2769143Z 2025-08-14T22:00:03.2769247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2769435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2769503Z return mod(**inputs) 2025-08-14T22:00:03.2769777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2769864Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2770138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2770207Z outputs = layer_module( 2025-08-14T22:00:03.2770477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2770555Z outputs = self.rel_attn( 2025-08-14T22:00:03.2770821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2770959Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2770970Z 2025-08-14T22:00:03.2771077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2771288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2771382Z return mod(**inputs) 2025-08-14T22:00:03.2771652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2771739Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2772016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2772116Z outputs = layer_module( 2025-08-14T22:00:03.2772394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2772467Z outputs = self.rel_attn( 2025-08-14T22:00:03.2772747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2772834Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2773123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2773266Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2773276Z 2025-08-14T22:00:03.2773401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2773667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2773747Z return mod(**inputs) 2025-08-14T22:00:03.2774027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2774113Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2774397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2774469Z outputs = layer_module( 2025-08-14T22:00:03.2774762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2774837Z outputs = self.rel_attn( 2025-08-14T22:00:03.2775118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2775233Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2775238Z 2025-08-14T22:00:03.2775347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2775558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2775637Z return mod(**inputs) 2025-08-14T22:00:03.2775916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2776012Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2776288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2776360Z outputs = layer_module( 2025-08-14T22:00:03.2776675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2776748Z outputs = self.rel_attn( 2025-08-14T22:00:03.2777016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2777099Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2777380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2777517Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2777521Z 2025-08-14T22:00:03.2777625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2777834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2777934Z return mod(**inputs) 2025-08-14T22:00:03.2778200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2778294Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2778563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2778650Z outputs = layer_module( 2025-08-14T22:00:03.2778924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2778993Z outputs = self.rel_attn( 2025-08-14T22:00:03.2779262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2779362Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2779652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2779779Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2779783Z 2025-08-14T22:00:03.2779906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2780133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2780212Z return mod(**inputs) 2025-08-14T22:00:03.2780476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2780567Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2780839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2780908Z outputs = layer_module( 2025-08-14T22:00:03.2781180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2781252Z outputs = self.rel_attn( 2025-08-14T22:00:03.2781512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2781611Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2781897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2782022Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2782025Z 2025-08-14T22:00:03.2782130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2782339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2782415Z return mod(**inputs) 2025-08-14T22:00:03.2782677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2782772Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2783034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2783105Z outputs = layer_module( 2025-08-14T22:00:03.2783380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2783602Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2783882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2783976Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2784254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2784340Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2784633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2784711Z output = self.layer_1(output) 2025-08-14T22:00:03.2784715Z 2025-08-14T22:00:03.2784834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2785056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2785149Z return mod(**inputs) 2025-08-14T22:00:03.2785414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2785499Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2785770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2785842Z outputs = layer_module( 2025-08-14T22:00:03.2786104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2786328Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2786636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2786727Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2786990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2787066Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2787336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2787430Z output = self.activation_function(output) 2025-08-14T22:00:03.2787659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2787733Z return self.act(input) 2025-08-14T22:00:03.2787736Z 2025-08-14T22:00:03.2787842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2788059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2788128Z return mod(**inputs) 2025-08-14T22:00:03.2788398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2788490Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2788755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2788830Z outputs = layer_module( 2025-08-14T22:00:03.2789092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2789311Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2789592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2789673Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2789946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2790024Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2790287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2790372Z output = self.layer_2(output) 2025-08-14T22:00:03.2790376Z 2025-08-14T22:00:03.2790483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2790689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2790763Z return mod(**inputs) 2025-08-14T22:00:03.2791047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2791140Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2791410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2791497Z outputs = layer_module( 2025-08-14T22:00:03.2791773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2791846Z outputs = self.rel_attn( 2025-08-14T22:00:03.2792116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T22:00:03.2792227Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T22:00:03.2792231Z 2025-08-14T22:00:03.2792336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2792553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2792622Z return mod(**inputs) 2025-08-14T22:00:03.2792912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2793022Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2793290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2793367Z outputs = layer_module( 2025-08-14T22:00:03.2793629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2793700Z outputs = self.rel_attn( 2025-08-14T22:00:03.2793982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T22:00:03.2794091Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T22:00:03.2794096Z 2025-08-14T22:00:03.2794215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2794440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2794510Z return mod(**inputs) 2025-08-14T22:00:03.2794796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2794887Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2795166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2795245Z outputs = layer_module( 2025-08-14T22:00:03.2795525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2795604Z outputs = self.rel_attn( 2025-08-14T22:00:03.2795981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2796069Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2796367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T22:00:03.2796512Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T22:00:03.2796518Z 2025-08-14T22:00:03.2796627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2796845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2796915Z return mod(**inputs) 2025-08-14T22:00:03.2797201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2797290Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2797571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2797702Z outputs = layer_module( 2025-08-14T22:00:03.2797971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2798053Z outputs = self.rel_attn( 2025-08-14T22:00:03.2798341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T22:00:03.2798479Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T22:00:03.2798482Z 2025-08-14T22:00:03.2798594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2798801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2798870Z return mod(**inputs) 2025-08-14T22:00:03.2799161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2799251Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2799548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2799623Z outputs = layer_module( 2025-08-14T22:00:03.2799928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2800012Z outputs = self.rel_attn( 2025-08-14T22:00:03.2800296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2800379Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2800672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T22:00:03.2800811Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T22:00:03.2800817Z 2025-08-14T22:00:03.2800932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2801156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2801227Z return mod(**inputs) 2025-08-14T22:00:03.2801521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2801609Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2801901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2801973Z outputs = layer_module( 2025-08-14T22:00:03.2802243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2802324Z outputs = self.rel_attn( 2025-08-14T22:00:03.2802597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T22:00:03.2802705Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T22:00:03.2802715Z 2025-08-14T22:00:03.2802825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2803038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2803117Z return mod(**inputs) 2025-08-14T22:00:03.2803387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2803473Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2803750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2803820Z outputs = layer_module( 2025-08-14T22:00:03.2804102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2804195Z outputs = self.rel_attn( 2025-08-14T22:00:03.2804468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T22:00:03.2804552Z attn_vec = self.rel_attn_core( 2025-08-14T22:00:03.2804840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T22:00:03.2804993Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T22:00:03.2805004Z 2025-08-14T22:00:03.2805113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2805323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2805399Z return mod(**inputs) 2025-08-14T22:00:03.2805669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2805759Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2806037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2806124Z outputs = layer_module( 2025-08-14T22:00:03.2806424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2806500Z outputs = self.rel_attn( 2025-08-14T22:00:03.2806771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2806876Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2807174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2807294Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2807306Z 2025-08-14T22:00:03.2807419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2807630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2807709Z return mod(**inputs) 2025-08-14T22:00:03.2807984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2808073Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2808354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2808424Z outputs = layer_module( 2025-08-14T22:00:03.2808879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T22:00:03.2808960Z outputs = self.rel_attn( 2025-08-14T22:00:03.2809232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T22:00:03.2809339Z output_h = self.post_attention(h, attn_vec) 2025-08-14T22:00:03.2809631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T22:00:03.2809751Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T22:00:03.2809765Z 2025-08-14T22:00:03.2809874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2810087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2810167Z return mod(**inputs) 2025-08-14T22:00:03.2810438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2810527Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2810805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2810943Z outputs = layer_module( 2025-08-14T22:00:03.2811223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2811453Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2811758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2811849Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2812124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2812202Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2812481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T22:00:03.2812560Z output = self.layer_1(output) 2025-08-14T22:00:03.2812566Z 2025-08-14T22:00:03.2812688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2812900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2812968Z return mod(**inputs) 2025-08-14T22:00:03.2813246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2813329Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2813583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2813649Z outputs = layer_module( 2025-08-14T22:00:03.2813894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2814112Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2814361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2814437Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2814689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2814760Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2815007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T22:00:03.2815092Z output = self.activation_function(output) 2025-08-14T22:00:03.2815295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:03.2815371Z return self.act(input) 2025-08-14T22:00:03.2815374Z 2025-08-14T22:00:03.2815473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2815669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2815732Z return mod(**inputs) 2025-08-14T22:00:03.2815975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T22:00:03.2816064Z transformer_outputs = self.transformer( 2025-08-14T22:00:03.2816307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T22:00:03.2816371Z outputs = layer_module( 2025-08-14T22:00:03.2816616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T22:00:03.2816811Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T22:00:03.2817066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:03.2817160Z return forward_fn(*input_tensors) 2025-08-14T22:00:03.2817404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T22:00:03.2817483Z output_x = self.ff(output_x) 2025-08-14T22:00:03.2817727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T22:00:03.2817829Z output = self.layer_2(output) 2025-08-14T22:00:03.2817833Z 2025-08-14T22:00:03.2817932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2818122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2818199Z return mod(**inputs) 2025-08-14T22:00:03.2818440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1624, in forward 2025-08-14T22:00:03.2818531Z logits = self.lm_loss(transformer_outputs[0]) 2025-08-14T22:00:03.2818536Z 2025-08-14T22:00:03.2818640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:03.2818844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:03.2818915Z return mod(**inputs) 2025-08-14T22:00:03.2819206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1630, in forward 2025-08-14T22:00:03.2819336Z loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1)) 2025-08-14T22:00:03.2819340Z 2025-08-14T22:00:16.1091951Z Compilation time (from dynamo_timed): 32.04389677 2025-08-14T22:00:16.1131847Z pass 2025-08-14T22:00:16.1132522Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T22:00:16.1133531Z TIMING: _recursive_pre_grad_passes:0.01385 _recursive_joint_graph_passes:1.37134 _recursive_post_grad_passes:0.24679 async_compile.wait:0.77932 code_gen:11.50752 inductor_compile:16.22072 backend_compile:25.94754 gc:0.00277 entire_frame_compile:32.0439 total_wall_time:32.0439 2025-08-14T22:00:16.1134580Z STATS: call_* op count: 818 | FakeTensorMode.__torch_dispatch__:56665 | FakeTensor.__torch_dispatch__:16773 | ProxyTorchDispatchMode.__torch_dispatch__:18623 2025-08-14T22:00:16.1135124Z Dynamo produced 1 graphs covering 818 ops with 0 graph breaks (0 unique) 2025-08-14T22:00:22.1884551Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T22:00:22.1885507Z from pkg_resources import resource_filename 2025-08-14T22:00:22.7901309Z 2025-08-14T22:00:24.2832939Z loading model: 0it [00:00, ?it/s] 2025-08-14T22:00:24.2833258Z loading model: 0it [00:01, ?it/s] 2025-08-14T22:00:24.2862302Z cpu eval YituTechConvBert 2025-08-14T22:00:25.1807804Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T22:00:25.4757640Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T22:00:25.7786637Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T22:00:38.6987169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.6987660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.6989324Z return mod(**inputs) 2025-08-14T22:00:38.6989795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.6990280Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.6990748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.6993569Z hidden_states = self.encoder( 2025-08-14T22:00:38.6994233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.6994727Z layer_outputs = layer_module( 2025-08-14T22:00:38.6996369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.6996795Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.6997262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.6997727Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.6998185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.6999555Z self_outputs = self.self( 2025-08-14T22:00:38.7000031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7000653Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7006304Z 2025-08-14T22:00:38.7012518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7017340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7021782Z return mod(**inputs) 2025-08-14T22:00:38.7027014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7027607Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7028068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7028503Z hidden_states = self.encoder( 2025-08-14T22:00:38.7028956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7029381Z layer_outputs = layer_module( 2025-08-14T22:00:38.7029759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7030165Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7030641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7031114Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7031582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7032048Z self_outputs = self.self( 2025-08-14T22:00:38.7032489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7032966Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7033135Z 2025-08-14T22:00:38.7033256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7033662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7034034Z return mod(**inputs) 2025-08-14T22:00:38.7034447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7034900Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7035345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7035783Z hidden_states = self.encoder( 2025-08-14T22:00:38.7036466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7037108Z layer_outputs = layer_module( 2025-08-14T22:00:38.7037488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7037893Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7038332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7038822Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7039243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7039671Z self_outputs = self.self( 2025-08-14T22:00:38.7040082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7040539Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7040699Z 2025-08-14T22:00:38.7040792Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7041023Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7041274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7041706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7042099Z return mod(**inputs) 2025-08-14T22:00:38.7042514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7042955Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7043390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7043832Z hidden_states = self.encoder( 2025-08-14T22:00:38.7044266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7044692Z layer_outputs = layer_module( 2025-08-14T22:00:38.7045068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7045461Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7045891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7046315Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7046751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7047239Z self_outputs = self.self( 2025-08-14T22:00:38.7047638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7048101Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7048274Z 2025-08-14T22:00:38.7048357Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7048612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7048982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7049328Z return mod(**inputs) 2025-08-14T22:00:38.7049752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7050192Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7050633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7051061Z hidden_states = self.encoder( 2025-08-14T22:00:38.7051470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7051888Z layer_outputs = layer_module( 2025-08-14T22:00:38.7052253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7052665Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7053106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7053548Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7054015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7054445Z self_outputs = self.self( 2025-08-14T22:00:38.7054869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7055387Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7055912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7056349Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7056489Z 2025-08-14T22:00:38.7056604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7057019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7057391Z return mod(**inputs) 2025-08-14T22:00:38.7057805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7058241Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7058682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7059133Z hidden_states = self.encoder( 2025-08-14T22:00:38.7059547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7059979Z layer_outputs = layer_module( 2025-08-14T22:00:38.7060351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7060744Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7061176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7061621Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7062066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7062497Z self_outputs = self.self( 2025-08-14T22:00:38.7062907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7063435Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7063968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7064401Z x = self.pointwise(x) 2025-08-14T22:00:38.7064521Z 2025-08-14T22:00:38.7064638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7065031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7065387Z return mod(**inputs) 2025-08-14T22:00:38.7065788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7066230Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7066672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7067103Z hidden_states = self.encoder( 2025-08-14T22:00:38.7067519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7067974Z layer_outputs = layer_module( 2025-08-14T22:00:38.7068352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7068752Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7069216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7069674Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7070131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7070554Z self_outputs = self.self( 2025-08-14T22:00:38.7070979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7071505Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7071726Z 2025-08-14T22:00:38.7071845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7072238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7073248Z return mod(**inputs) 2025-08-14T22:00:38.7073662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7074093Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7074527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7074965Z hidden_states = self.encoder( 2025-08-14T22:00:38.7075405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7075930Z layer_outputs = layer_module( 2025-08-14T22:00:38.7076325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7076725Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7077175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7077626Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7078078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7078522Z self_outputs = self.self( 2025-08-14T22:00:38.7078924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7079404Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7079601Z 2025-08-14T22:00:38.7079714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7080098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7080434Z return mod(**inputs) 2025-08-14T22:00:38.7080835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7081276Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7081704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7082129Z hidden_states = self.encoder( 2025-08-14T22:00:38.7082546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7082964Z layer_outputs = layer_module( 2025-08-14T22:00:38.7083322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7083757Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7084190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7084621Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7085041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7085482Z self_outputs = self.self( 2025-08-14T22:00:38.7085887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7086364Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7086563Z 2025-08-14T22:00:38.7086650Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7086882Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7087141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7087522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7087870Z return mod(**inputs) 2025-08-14T22:00:38.7088315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7088746Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7089196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7089631Z hidden_states = self.encoder( 2025-08-14T22:00:38.7090058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7090481Z layer_outputs = layer_module( 2025-08-14T22:00:38.7090849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7091433Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7091860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7092315Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7092770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7093751Z self_outputs = self.self( 2025-08-14T22:00:38.7094176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7094659Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7094847Z 2025-08-14T22:00:38.7094959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7095343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7095687Z return mod(**inputs) 2025-08-14T22:00:38.7096104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7096555Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7097009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7097425Z hidden_states = self.encoder( 2025-08-14T22:00:38.7097861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7098289Z layer_outputs = layer_module( 2025-08-14T22:00:38.7098649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7099039Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7099473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7099934Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7100359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7100860Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7101340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7101776Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7101932Z 2025-08-14T22:00:38.7102038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7102402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7102731Z return mod(**inputs) 2025-08-14T22:00:38.7103099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7103513Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7103962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7104407Z hidden_states = self.encoder( 2025-08-14T22:00:38.7104821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7105244Z layer_outputs = layer_module( 2025-08-14T22:00:38.7105605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7105986Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7106420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7106866Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7107299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7107716Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7108180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7108884Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7109396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7109843Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7109998Z 2025-08-14T22:00:38.7110111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7110498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7110838Z return mod(**inputs) 2025-08-14T22:00:38.7111245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7111683Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7112134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7112555Z hidden_states = self.encoder( 2025-08-14T22:00:38.7112981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7113429Z layer_outputs = layer_module( 2025-08-14T22:00:38.7113809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7114206Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7114659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7115180Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7115617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7116129Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7116670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7117199Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7117686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7118172Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7118571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7118933Z return self.act(input) 2025-08-14T22:00:38.7119053Z 2025-08-14T22:00:38.7119166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7119595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7119944Z return mod(**inputs) 2025-08-14T22:00:38.7120372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7120810Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7121243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7121674Z hidden_states = self.encoder( 2025-08-14T22:00:38.7122084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7122508Z layer_outputs = layer_module( 2025-08-14T22:00:38.7122879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7123263Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7123698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7124137Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7124561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7124975Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7125431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7125953Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7126437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7126864Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7127017Z 2025-08-14T22:00:38.7127131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7127510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7127854Z return mod(**inputs) 2025-08-14T22:00:38.7128255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7128691Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7129120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7129535Z hidden_states = self.encoder( 2025-08-14T22:00:38.7129948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7130404Z layer_outputs = layer_module( 2025-08-14T22:00:38.7130779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7131158Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7131613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7132049Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7132494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7132920Z self_outputs = self.self( 2025-08-14T22:00:38.7133341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7133794Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7133953Z 2025-08-14T22:00:38.7134064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7134473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7134824Z return mod(**inputs) 2025-08-14T22:00:38.7135248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7135688Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7136117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7136552Z hidden_states = self.encoder( 2025-08-14T22:00:38.7136981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7137420Z layer_outputs = layer_module( 2025-08-14T22:00:38.7137785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7138178Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7138615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7139050Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7139480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7139894Z self_outputs = self.self( 2025-08-14T22:00:38.7140307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7140753Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7140899Z 2025-08-14T22:00:38.7141018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7141397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7141750Z return mod(**inputs) 2025-08-14T22:00:38.7142151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7142589Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7143021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7143442Z hidden_states = self.encoder( 2025-08-14T22:00:38.7143855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7144272Z layer_outputs = layer_module( 2025-08-14T22:00:38.7144639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7145087Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7145560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7146002Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7146441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7146895Z self_outputs = self.self( 2025-08-14T22:00:38.7147302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7147742Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7147904Z 2025-08-14T22:00:38.7147993Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7148226Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7148478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7148860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7149206Z return mod(**inputs) 2025-08-14T22:00:38.7149620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7150057Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7150507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7150932Z hidden_states = self.encoder( 2025-08-14T22:00:38.7151342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7151760Z layer_outputs = layer_module( 2025-08-14T22:00:38.7152128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7152523Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7152956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7153411Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7153848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7154270Z self_outputs = self.self( 2025-08-14T22:00:38.7154689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7155160Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7155331Z 2025-08-14T22:00:38.7155425Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7155680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7156147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7156510Z return mod(**inputs) 2025-08-14T22:00:38.7156911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7157366Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7157777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7158179Z hidden_states = self.encoder( 2025-08-14T22:00:38.7158572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7158976Z layer_outputs = layer_module( 2025-08-14T22:00:38.7159331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7159694Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7160108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7160590Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7161009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7161404Z self_outputs = self.self( 2025-08-14T22:00:38.7161821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7162308Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7162795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7163209Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7163358Z 2025-08-14T22:00:38.7163471Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7163855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7164183Z return mod(**inputs) 2025-08-14T22:00:38.7164580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7165002Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7165427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7165847Z hidden_states = self.encoder( 2025-08-14T22:00:38.7166262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7166685Z layer_outputs = layer_module( 2025-08-14T22:00:38.7167050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7167429Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7167847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7168275Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7168700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7169124Z self_outputs = self.self( 2025-08-14T22:00:38.7169533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7170045Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7170553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7170971Z x = self.pointwise(x) 2025-08-14T22:00:38.7171093Z 2025-08-14T22:00:38.7171204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7171583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7171947Z return mod(**inputs) 2025-08-14T22:00:38.7172343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7172783Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7173208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7173641Z hidden_states = self.encoder( 2025-08-14T22:00:38.7174072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7174509Z layer_outputs = layer_module( 2025-08-14T22:00:38.7174867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7175287Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7175718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7176149Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7176596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7177013Z self_outputs = self.self( 2025-08-14T22:00:38.7177433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7177941Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7178171Z 2025-08-14T22:00:38.7178280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7178659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7179001Z return mod(**inputs) 2025-08-14T22:00:38.7179414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7179849Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7180297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7180717Z hidden_states = self.encoder( 2025-08-14T22:00:38.7181121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7181546Z layer_outputs = layer_module( 2025-08-14T22:00:38.7181892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7182253Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7182678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7183108Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7183539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7183953Z self_outputs = self.self( 2025-08-14T22:00:38.7184362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7184834Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7185015Z 2025-08-14T22:00:38.7185126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7185503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7185846Z return mod(**inputs) 2025-08-14T22:00:38.7186244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7186683Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7187114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7187541Z hidden_states = self.encoder( 2025-08-14T22:00:38.7187957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7188369Z layer_outputs = layer_module( 2025-08-14T22:00:38.7188733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7189112Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7189530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7189999Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7190426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7190841Z self_outputs = self.self( 2025-08-14T22:00:38.7191238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7191737Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7191927Z 2025-08-14T22:00:38.7192023Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7192249Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7192492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7192872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7193219Z return mod(**inputs) 2025-08-14T22:00:38.7193610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7194039Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7194501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7194940Z hidden_states = self.encoder( 2025-08-14T22:00:38.7195352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7195783Z layer_outputs = layer_module( 2025-08-14T22:00:38.7196248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7196638Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7197079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7197529Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7197973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7198374Z self_outputs = self.self( 2025-08-14T22:00:38.7198803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7199302Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7199485Z 2025-08-14T22:00:38.7199608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7199994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7200347Z return mod(**inputs) 2025-08-14T22:00:38.7200765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7201206Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7201652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7202085Z hidden_states = self.encoder( 2025-08-14T22:00:38.7202508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7202932Z layer_outputs = layer_module( 2025-08-14T22:00:38.7203305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7203699Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7204129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7204577Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7205022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7205557Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7206046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7206514Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7206666Z 2025-08-14T22:00:38.7206786Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7207179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7207527Z return mod(**inputs) 2025-08-14T22:00:38.7207943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7208380Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7208930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7209341Z hidden_states = self.encoder( 2025-08-14T22:00:38.7209858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7210296Z layer_outputs = layer_module( 2025-08-14T22:00:38.7210644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7211020Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7211444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7211872Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7212306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7212735Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7213207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7213713Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7214187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7214618Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7214768Z 2025-08-14T22:00:38.7214890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7215258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7215600Z return mod(**inputs) 2025-08-14T22:00:38.7215996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7216427Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7216855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7217257Z hidden_states = self.encoder( 2025-08-14T22:00:38.7217662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7218060Z layer_outputs = layer_module( 2025-08-14T22:00:38.7218418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7218785Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7219195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7219602Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7220008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7220429Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7220852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7221329Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7221802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7222241Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7222616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7222960Z return self.act(input) 2025-08-14T22:00:38.7223076Z 2025-08-14T22:00:38.7223190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7223551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7223880Z return mod(**inputs) 2025-08-14T22:00:38.7224287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7224717Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7225148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7225563Z hidden_states = self.encoder( 2025-08-14T22:00:38.7225982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7226403Z layer_outputs = layer_module( 2025-08-14T22:00:38.7226771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7227154Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7227587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7228019Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7228449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7228880Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7229312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7229793Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7230252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7230663Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7230802Z 2025-08-14T22:00:38.7230916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7231271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7231601Z return mod(**inputs) 2025-08-14T22:00:38.7231985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7232395Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7232823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7233246Z hidden_states = self.encoder( 2025-08-14T22:00:38.7233676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7234104Z layer_outputs = layer_module( 2025-08-14T22:00:38.7234481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7234908Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7235365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7235876Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7236346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7236804Z self_outputs = self.self( 2025-08-14T22:00:38.7237221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7237644Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7237800Z 2025-08-14T22:00:38.7237905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7238267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7238588Z return mod(**inputs) 2025-08-14T22:00:38.7238969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7239403Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7239832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7240228Z hidden_states = self.encoder( 2025-08-14T22:00:38.7240618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7241017Z layer_outputs = layer_module( 2025-08-14T22:00:38.7241363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7241757Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7242200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7242636Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7243076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7243502Z self_outputs = self.self( 2025-08-14T22:00:38.7243915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7244346Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7244504Z 2025-08-14T22:00:38.7244614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7245000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7245343Z return mod(**inputs) 2025-08-14T22:00:38.7245734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7246175Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7246605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7247030Z hidden_states = self.encoder( 2025-08-14T22:00:38.7247439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7247858Z layer_outputs = layer_module( 2025-08-14T22:00:38.7248224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7248599Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7249027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7249463Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7249923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7250332Z self_outputs = self.self( 2025-08-14T22:00:38.7250743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7251228Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7251386Z 2025-08-14T22:00:38.7251478Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7251701Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7251955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7252334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7252671Z return mod(**inputs) 2025-08-14T22:00:38.7253067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7253497Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7253968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7254387Z hidden_states = self.encoder( 2025-08-14T22:00:38.7254858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7255282Z layer_outputs = layer_module( 2025-08-14T22:00:38.7255641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7256034Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7256480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7256927Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7257349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7257741Z self_outputs = self.self( 2025-08-14T22:00:38.7258125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7258562Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7258740Z 2025-08-14T22:00:38.7258826Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7259083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7259466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7259811Z return mod(**inputs) 2025-08-14T22:00:38.7260191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7260624Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7261049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7261480Z hidden_states = self.encoder( 2025-08-14T22:00:38.7261890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7262301Z layer_outputs = layer_module( 2025-08-14T22:00:38.7262657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7263052Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7263497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7263938Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7264381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7264833Z self_outputs = self.self( 2025-08-14T22:00:38.7265221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7265703Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7266210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7266608Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7266737Z 2025-08-14T22:00:38.7266849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7267193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7267515Z return mod(**inputs) 2025-08-14T22:00:38.7267890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7268303Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7268764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7269184Z hidden_states = self.encoder( 2025-08-14T22:00:38.7269613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7270028Z layer_outputs = layer_module( 2025-08-14T22:00:38.7270394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7270776Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7271198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7271624Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7272059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7272482Z self_outputs = self.self( 2025-08-14T22:00:38.7272896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7273424Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7273952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7274387Z x = self.pointwise(x) 2025-08-14T22:00:38.7274509Z 2025-08-14T22:00:38.7274620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7275019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7275387Z return mod(**inputs) 2025-08-14T22:00:38.7275876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7276327Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7276769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7277207Z hidden_states = self.encoder( 2025-08-14T22:00:38.7277641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7278074Z layer_outputs = layer_module( 2025-08-14T22:00:38.7278446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7278850Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7279295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7279763Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7280218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7280663Z self_outputs = self.self( 2025-08-14T22:00:38.7281075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7281618Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7281839Z 2025-08-14T22:00:38.7281962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7282345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7282694Z return mod(**inputs) 2025-08-14T22:00:38.7283097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7283543Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7283973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7284515Z hidden_states = self.encoder( 2025-08-14T22:00:38.7284976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7285413Z layer_outputs = layer_module( 2025-08-14T22:00:38.7285778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7286169Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7286610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7287049Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7287486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7287918Z self_outputs = self.self( 2025-08-14T22:00:38.7288337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7288820Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7289013Z 2025-08-14T22:00:38.7289126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7289517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7289881Z return mod(**inputs) 2025-08-14T22:00:38.7290269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7290704Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7291131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7291546Z hidden_states = self.encoder( 2025-08-14T22:00:38.7291963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7292385Z layer_outputs = layer_module( 2025-08-14T22:00:38.7292754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7293128Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7293552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7293985Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7294405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7294852Z self_outputs = self.self( 2025-08-14T22:00:38.7295254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7295730Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7295922Z 2025-08-14T22:00:38.7296008Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7296255Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7296506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7296877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7297220Z return mod(**inputs) 2025-08-14T22:00:38.7297633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7298064Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7298485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7298906Z hidden_states = self.encoder( 2025-08-14T22:00:38.7299353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7299787Z layer_outputs = layer_module( 2025-08-14T22:00:38.7300152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7300542Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7300955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7301358Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7301768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7302176Z self_outputs = self.self( 2025-08-14T22:00:38.7302571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7303036Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7303211Z 2025-08-14T22:00:38.7303320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7303689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7304018Z return mod(**inputs) 2025-08-14T22:00:38.7304395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7304823Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7305241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7305657Z hidden_states = self.encoder( 2025-08-14T22:00:38.7306057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7306485Z layer_outputs = layer_module( 2025-08-14T22:00:38.7306839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7307204Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7307618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7308037Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7308448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7309075Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7309530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7310044Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7310193Z 2025-08-14T22:00:38.7310306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7310690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7311113Z return mod(**inputs) 2025-08-14T22:00:38.7311512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7311937Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7312365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7312787Z hidden_states = self.encoder( 2025-08-14T22:00:38.7313192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7313620Z layer_outputs = layer_module( 2025-08-14T22:00:38.7313998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7314416Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7314866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7315304Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7315733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7316231Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7316699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7317234Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7317716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7318146Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7318298Z 2025-08-14T22:00:38.7318407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7318789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7319135Z return mod(**inputs) 2025-08-14T22:00:38.7319526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7319961Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7320400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7320825Z hidden_states = self.encoder( 2025-08-14T22:00:38.7321233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7321656Z layer_outputs = layer_module( 2025-08-14T22:00:38.7322021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7322403Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7322829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7323264Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7323685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7324095Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7324549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7325077Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7325544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7325998Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7326420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7326784Z return self.act(input) 2025-08-14T22:00:38.7326902Z 2025-08-14T22:00:38.7327011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7327395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7327743Z return mod(**inputs) 2025-08-14T22:00:38.7328140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7328580Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7328986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7329403Z hidden_states = self.encoder( 2025-08-14T22:00:38.7329807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7330230Z layer_outputs = layer_module( 2025-08-14T22:00:38.7330597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7330985Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7331403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7331840Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7332262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7332681Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7333131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7333660Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7334129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7334544Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7334693Z 2025-08-14T22:00:38.7334800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7335169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7335501Z return mod(**inputs) 2025-08-14T22:00:38.7335881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7336304Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7336723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7337141Z hidden_states = self.encoder( 2025-08-14T22:00:38.7337525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7337921Z layer_outputs = layer_module( 2025-08-14T22:00:38.7338266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7338619Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7339019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7339446Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7339852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7340242Z self_outputs = self.self( 2025-08-14T22:00:38.7340630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7341058Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7341203Z 2025-08-14T22:00:38.7341314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7341666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7341991Z return mod(**inputs) 2025-08-14T22:00:38.7342366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7342764Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7343171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7343580Z hidden_states = self.encoder( 2025-08-14T22:00:38.7343989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7344379Z layer_outputs = layer_module( 2025-08-14T22:00:38.7344725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7345116Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7345532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7345946Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7346371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7346785Z self_outputs = self.self( 2025-08-14T22:00:38.7347181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7347619Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7347766Z 2025-08-14T22:00:38.7347885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7348262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7348616Z return mod(**inputs) 2025-08-14T22:00:38.7349016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7349445Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7349861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7350290Z hidden_states = self.encoder( 2025-08-14T22:00:38.7350710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7351139Z layer_outputs = layer_module( 2025-08-14T22:00:38.7351519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7351905Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7352330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7352768Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7353214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7353634Z self_outputs = self.self( 2025-08-14T22:00:38.7354050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7354549Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7354710Z 2025-08-14T22:00:38.7354798Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7355027Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7355290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7355688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7356110Z return mod(**inputs) 2025-08-14T22:00:38.7356524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7356960Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7357408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7357855Z hidden_states = self.encoder( 2025-08-14T22:00:38.7358331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7358764Z layer_outputs = layer_module( 2025-08-14T22:00:38.7359158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7359548Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7359985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7360434Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7360886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7361312Z self_outputs = self.self( 2025-08-14T22:00:38.7361731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7362185Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7362339Z 2025-08-14T22:00:38.7362427Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7362661Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7363018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7363350Z return mod(**inputs) 2025-08-14T22:00:38.7363750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7364175Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7364606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7365028Z hidden_states = self.encoder( 2025-08-14T22:00:38.7365440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7365833Z layer_outputs = layer_module( 2025-08-14T22:00:38.7366194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7366582Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7367000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7367436Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7367882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7368312Z self_outputs = self.self( 2025-08-14T22:00:38.7368709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7369239Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7369727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7370135Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7370291Z 2025-08-14T22:00:38.7370403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7370782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7371122Z return mod(**inputs) 2025-08-14T22:00:38.7371491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7371904Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7372344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7372790Z hidden_states = self.encoder( 2025-08-14T22:00:38.7373194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7373638Z layer_outputs = layer_module( 2025-08-14T22:00:38.7374021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7374398Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7374820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7375226Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7375627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7376025Z self_outputs = self.self( 2025-08-14T22:00:38.7376443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7376926Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7377407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7377792Z x = self.pointwise(x) 2025-08-14T22:00:38.7377911Z 2025-08-14T22:00:38.7378014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7378377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7378693Z return mod(**inputs) 2025-08-14T22:00:38.7379065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7379472Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7379872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7380261Z hidden_states = self.encoder( 2025-08-14T22:00:38.7380649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7381049Z layer_outputs = layer_module( 2025-08-14T22:00:38.7381392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7381746Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7382146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7382552Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7382945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7383372Z self_outputs = self.self( 2025-08-14T22:00:38.7383780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7384284Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7384503Z 2025-08-14T22:00:38.7384634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7385017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7385360Z return mod(**inputs) 2025-08-14T22:00:38.7385747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7386167Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7386583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7387007Z hidden_states = self.encoder( 2025-08-14T22:00:38.7387407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7387840Z layer_outputs = layer_module( 2025-08-14T22:00:38.7388223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7388609Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7389033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7389473Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7389911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7390327Z self_outputs = self.self( 2025-08-14T22:00:38.7390737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7391212Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7391394Z 2025-08-14T22:00:38.7391513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7391893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7392243Z return mod(**inputs) 2025-08-14T22:00:38.7392646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7393086Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7393523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7393968Z hidden_states = self.encoder( 2025-08-14T22:00:38.7394396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7394834Z layer_outputs = layer_module( 2025-08-14T22:00:38.7395217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7395617Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7396153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7396597Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7397033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7397462Z self_outputs = self.self( 2025-08-14T22:00:38.7397868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7398363Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7398592Z 2025-08-14T22:00:38.7398682Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7398915Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7399170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7399564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7399940Z return mod(**inputs) 2025-08-14T22:00:38.7400350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7400782Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7401220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7401654Z hidden_states = self.encoder( 2025-08-14T22:00:38.7402067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7402498Z layer_outputs = layer_module( 2025-08-14T22:00:38.7402884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7403280Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7403734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7404181Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7404619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7405040Z self_outputs = self.self( 2025-08-14T22:00:38.7405459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7405937Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7406118Z 2025-08-14T22:00:38.7406240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7406628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7406984Z return mod(**inputs) 2025-08-14T22:00:38.7407393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7407842Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7408275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7408841Z hidden_states = self.encoder( 2025-08-14T22:00:38.7409280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7409695Z layer_outputs = layer_module( 2025-08-14T22:00:38.7410065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7410454Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7410883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7411311Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7411736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7412214Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7412686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7413115Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7413269Z 2025-08-14T22:00:38.7413380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7413805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7414138Z return mod(**inputs) 2025-08-14T22:00:38.7414533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7414996Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7415424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7415839Z hidden_states = self.encoder( 2025-08-14T22:00:38.7416253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7416675Z layer_outputs = layer_module( 2025-08-14T22:00:38.7417029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7417413Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7417838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7418307Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7418756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7419174Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7419627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7420129Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7420593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7421030Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7421179Z 2025-08-14T22:00:38.7421297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7421674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7422010Z return mod(**inputs) 2025-08-14T22:00:38.7422414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7422846Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7423265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7423695Z hidden_states = self.encoder( 2025-08-14T22:00:38.7424125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7424553Z layer_outputs = layer_module( 2025-08-14T22:00:38.7424910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7425293Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7425736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7426168Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7426594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7427019Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7427482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7427981Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7428475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7428963Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7429365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7429720Z return self.act(input) 2025-08-14T22:00:38.7429845Z 2025-08-14T22:00:38.7429973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7430353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7430686Z return mod(**inputs) 2025-08-14T22:00:38.7431081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7431511Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7431937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7432345Z hidden_states = self.encoder( 2025-08-14T22:00:38.7432767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7433215Z layer_outputs = layer_module( 2025-08-14T22:00:38.7433606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7433997Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7434441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7434898Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7435333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7435770Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7436303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7436839Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7437319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7437726Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7437863Z 2025-08-14T22:00:38.7437974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7438331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7438651Z return mod(**inputs) 2025-08-14T22:00:38.7439030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7439440Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7439705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7439779Z hidden_states = self.encoder( 2025-08-14T22:00:38.7440056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7440131Z layer_outputs = layer_module( 2025-08-14T22:00:38.7440362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7440441Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7440708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7440798Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7441059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7441152Z self_outputs = self.self( 2025-08-14T22:00:38.7441424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7441518Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7441522Z 2025-08-14T22:00:38.7441634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7441887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7441954Z return mod(**inputs) 2025-08-14T22:00:38.7442222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7442304Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7442576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7442647Z hidden_states = self.encoder( 2025-08-14T22:00:38.7442911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7442989Z layer_outputs = layer_module( 2025-08-14T22:00:38.7443237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7443317Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7443594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7443674Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7443951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7444021Z self_outputs = self.self( 2025-08-14T22:00:38.7444282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7444376Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7444380Z 2025-08-14T22:00:38.7444484Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7444689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7444757Z return mod(**inputs) 2025-08-14T22:00:38.7445022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7445111Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7445373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7445443Z hidden_states = self.encoder( 2025-08-14T22:00:38.7445715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7445787Z layer_outputs = layer_module( 2025-08-14T22:00:38.7446018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7446098Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7446400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7446494Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7446778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7446857Z self_outputs = self.self( 2025-08-14T22:00:38.7447132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7447230Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7447234Z 2025-08-14T22:00:38.7447364Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7447449Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7447558Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7447780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7447851Z return mod(**inputs) 2025-08-14T22:00:38.7448156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7448233Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7448492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7448570Z hidden_states = self.encoder( 2025-08-14T22:00:38.7448824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7448894Z layer_outputs = layer_module( 2025-08-14T22:00:38.7449113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7449201Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7449483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7449564Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7449814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7449891Z self_outputs = self.self( 2025-08-14T22:00:38.7450144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7450253Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7450257Z 2025-08-14T22:00:38.7450332Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7450435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7450637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7450704Z return mod(**inputs) 2025-08-14T22:00:38.7450963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7451053Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7451315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7451392Z hidden_states = self.encoder( 2025-08-14T22:00:38.7451649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7451718Z layer_outputs = layer_module( 2025-08-14T22:00:38.7451941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7452018Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7452277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7452377Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7452631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7452706Z self_outputs = self.self( 2025-08-14T22:00:38.7452957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7453125Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7453429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7453529Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7453533Z 2025-08-14T22:00:38.7453659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7453855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7453944Z return mod(**inputs) 2025-08-14T22:00:38.7454217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7454296Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7454567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7454638Z hidden_states = self.encoder( 2025-08-14T22:00:38.7454903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7454988Z layer_outputs = layer_module( 2025-08-14T22:00:38.7455201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7455291Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7455572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7455658Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7455922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7455992Z self_outputs = self.self( 2025-08-14T22:00:38.7456255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7456417Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7456679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7456758Z x = self.pointwise(x) 2025-08-14T22:00:38.7456770Z 2025-08-14T22:00:38.7456871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7457061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7457132Z return mod(**inputs) 2025-08-14T22:00:38.7457386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7457463Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7457726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7457795Z hidden_states = self.encoder( 2025-08-14T22:00:38.7458059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7458131Z layer_outputs = layer_module( 2025-08-14T22:00:38.7458349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7458435Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7458699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7458778Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7459044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7459113Z self_outputs = self.self( 2025-08-14T22:00:38.7459382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7459533Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7459558Z 2025-08-14T22:00:38.7459662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7459865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7459931Z return mod(**inputs) 2025-08-14T22:00:38.7460219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7460300Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7460571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7460648Z hidden_states = self.encoder( 2025-08-14T22:00:38.7460903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7460971Z layer_outputs = layer_module( 2025-08-14T22:00:38.7461194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7461268Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7461577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7461659Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7461914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7461989Z self_outputs = self.self( 2025-08-14T22:00:38.7462245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7462372Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7462376Z 2025-08-14T22:00:38.7462475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7462672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7462745Z return mod(**inputs) 2025-08-14T22:00:38.7463017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7463105Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7463391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7463467Z hidden_states = self.encoder( 2025-08-14T22:00:38.7463752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7463825Z layer_outputs = layer_module( 2025-08-14T22:00:38.7464054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7464148Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7464429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7464520Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7464802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7464877Z self_outputs = self.self( 2025-08-14T22:00:38.7465166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7465299Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7465303Z 2025-08-14T22:00:38.7465387Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7465475Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7465580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7465820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7465888Z return mod(**inputs) 2025-08-14T22:00:38.7466195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7466318Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7466596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7466669Z hidden_states = self.encoder( 2025-08-14T22:00:38.7466954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7467027Z layer_outputs = layer_module( 2025-08-14T22:00:38.7467263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7467344Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7467621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7467729Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7468025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7468109Z self_outputs = self.self( 2025-08-14T22:00:38.7468383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7468502Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7468506Z 2025-08-14T22:00:38.7468620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7468825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7468896Z return mod(**inputs) 2025-08-14T22:00:38.7469177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7469262Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7469548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7469623Z hidden_states = self.encoder( 2025-08-14T22:00:38.7469897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7469978Z layer_outputs = layer_module( 2025-08-14T22:00:38.7470207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7470293Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7470590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7470676Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7470963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7471101Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7471378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7471474Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7471477Z 2025-08-14T22:00:38.7471588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7471800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7471868Z return mod(**inputs) 2025-08-14T22:00:38.7472143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7472254Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7472529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7472629Z hidden_states = self.encoder( 2025-08-14T22:00:38.7472907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7472981Z layer_outputs = layer_module( 2025-08-14T22:00:38.7473218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7473298Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7473597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7473698Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7473983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7474091Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7474437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7474574Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7474866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7474959Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7474962Z 2025-08-14T22:00:38.7475081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7475291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7475362Z return mod(**inputs) 2025-08-14T22:00:38.7475663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7475753Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7476128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7476223Z hidden_states = self.encoder( 2025-08-14T22:00:38.7476507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7476593Z layer_outputs = layer_module( 2025-08-14T22:00:38.7476829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7476914Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7477210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7477306Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7477606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7477690Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7478011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7478151Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7478430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7478551Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7478787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7478904Z return self.act(input) 2025-08-14T22:00:38.7478908Z 2025-08-14T22:00:38.7479028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7479238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7479307Z return mod(**inputs) 2025-08-14T22:00:38.7479616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7479704Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7479992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7480078Z hidden_states = self.encoder( 2025-08-14T22:00:38.7480344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7480420Z layer_outputs = layer_module( 2025-08-14T22:00:38.7480642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7480718Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7481029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7481116Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7481377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7481453Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7481747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7481884Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7482146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7482237Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7482241Z 2025-08-14T22:00:38.7482344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7482539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7482614Z return mod(**inputs) 2025-08-14T22:00:38.7482875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7482955Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7483225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7483296Z hidden_states = self.encoder( 2025-08-14T22:00:38.7483568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7483644Z layer_outputs = layer_module( 2025-08-14T22:00:38.7483875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7483964Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7484243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7484327Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7484611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7484685Z self_outputs = self.self( 2025-08-14T22:00:38.7484979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7485072Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7485095Z 2025-08-14T22:00:38.7485197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7485402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7485467Z return mod(**inputs) 2025-08-14T22:00:38.7485732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7485829Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7486091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7486167Z hidden_states = self.encoder( 2025-08-14T22:00:38.7486429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7486498Z layer_outputs = layer_module( 2025-08-14T22:00:38.7486724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7486799Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7487084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7487185Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7487447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7487527Z self_outputs = self.self( 2025-08-14T22:00:38.7487800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7487892Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7487896Z 2025-08-14T22:00:38.7488004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7488209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7488285Z return mod(**inputs) 2025-08-14T22:00:38.7488562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7488649Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7488937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7489011Z hidden_states = self.encoder( 2025-08-14T22:00:38.7489295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7489377Z layer_outputs = layer_module( 2025-08-14T22:00:38.7489596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7489682Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7489948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7490045Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7490329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7490404Z self_outputs = self.self( 2025-08-14T22:00:38.7490694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7490786Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7490790Z 2025-08-14T22:00:38.7490871Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7490959Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7491060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7491263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7491349Z return mod(**inputs) 2025-08-14T22:00:38.7491618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7491709Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7491990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7492062Z hidden_states = self.encoder( 2025-08-14T22:00:38.7492351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7492422Z layer_outputs = layer_module( 2025-08-14T22:00:38.7492656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7492733Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7493008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7493102Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7493407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7493488Z self_outputs = self.self( 2025-08-14T22:00:38.7493759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7493863Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7493867Z 2025-08-14T22:00:38.7493953Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7494057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7494256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7494330Z return mod(**inputs) 2025-08-14T22:00:38.7494600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7494691Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7494963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7495038Z hidden_states = self.encoder( 2025-08-14T22:00:38.7495317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7495388Z layer_outputs = layer_module( 2025-08-14T22:00:38.7495609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7495695Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7495965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7496057Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7496331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7496405Z self_outputs = self.self( 2025-08-14T22:00:38.7496694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7496864Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7497152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7497233Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7497237Z 2025-08-14T22:00:38.7497343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7497558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7497647Z return mod(**inputs) 2025-08-14T22:00:38.7497930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7498016Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7498311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7498390Z hidden_states = self.encoder( 2025-08-14T22:00:38.7498652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7498723Z layer_outputs = layer_module( 2025-08-14T22:00:38.7498945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7499020Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7499290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7499383Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7499677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7499756Z self_outputs = self.self( 2025-08-14T22:00:38.7500031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7500196Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7500491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7500562Z x = self.pointwise(x) 2025-08-14T22:00:38.7500565Z 2025-08-14T22:00:38.7500676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7500868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7500933Z return mod(**inputs) 2025-08-14T22:00:38.7501207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7501293Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7501574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7501648Z hidden_states = self.encoder( 2025-08-14T22:00:38.7501929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7502007Z layer_outputs = layer_module( 2025-08-14T22:00:38.7502222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7502300Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7502572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7502655Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7502925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7502994Z self_outputs = self.self( 2025-08-14T22:00:38.7503254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7503411Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7503414Z 2025-08-14T22:00:38.7503515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7503717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7503802Z return mod(**inputs) 2025-08-14T22:00:38.7504069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7504158Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7504436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7504507Z hidden_states = self.encoder( 2025-08-14T22:00:38.7504774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7504843Z layer_outputs = layer_module( 2025-08-14T22:00:38.7505065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7505141Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7505406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7505498Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7505808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7505892Z self_outputs = self.self( 2025-08-14T22:00:38.7506180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7506308Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7506312Z 2025-08-14T22:00:38.7506427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7506639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7506707Z return mod(**inputs) 2025-08-14T22:00:38.7507016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7507103Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7507403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7507475Z hidden_states = self.encoder( 2025-08-14T22:00:38.7507740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7507821Z layer_outputs = layer_module( 2025-08-14T22:00:38.7508044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7508128Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7508395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7508476Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7508918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7508998Z self_outputs = self.self( 2025-08-14T22:00:38.7509268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7509413Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7509417Z 2025-08-14T22:00:38.7509498Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7509587Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7509691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7509891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7509971Z return mod(**inputs) 2025-08-14T22:00:38.7510247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7510392Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7510678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7510781Z hidden_states = self.encoder( 2025-08-14T22:00:38.7511070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7511144Z layer_outputs = layer_module( 2025-08-14T22:00:38.7511373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7511465Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7511748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7511845Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7512131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7512232Z self_outputs = self.self( 2025-08-14T22:00:38.7512582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7512707Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7512711Z 2025-08-14T22:00:38.7512822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7513042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7513114Z return mod(**inputs) 2025-08-14T22:00:38.7513407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7513496Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7513777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7513861Z hidden_states = self.encoder( 2025-08-14T22:00:38.7514164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7514248Z layer_outputs = layer_module( 2025-08-14T22:00:38.7514483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7514565Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7514871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7514958Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7515258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7515405Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7515691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7515845Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7515851Z 2025-08-14T22:00:38.7515967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7516175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7516252Z return mod(**inputs) 2025-08-14T22:00:38.7516534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7516628Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7516921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7517020Z hidden_states = self.encoder( 2025-08-14T22:00:38.7517312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7517387Z layer_outputs = layer_module( 2025-08-14T22:00:38.7517634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7517723Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7517999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7518096Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7518368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7518448Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7518767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7518910Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7519209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7519300Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7519304Z 2025-08-14T22:00:38.7519411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7519622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7519689Z return mod(**inputs) 2025-08-14T22:00:38.7519969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7520060Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7520339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7520422Z hidden_states = self.encoder( 2025-08-14T22:00:38.7520698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7520772Z layer_outputs = layer_module( 2025-08-14T22:00:38.7521009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7521087Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7521370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7521458Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7521729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7521815Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7522125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7522255Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7522537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7522657Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7522886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7522960Z return self.act(input) 2025-08-14T22:00:38.7522964Z 2025-08-14T22:00:38.7523072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7523285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7523377Z return mod(**inputs) 2025-08-14T22:00:38.7523667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7523751Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7524049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7524132Z hidden_states = self.encoder( 2025-08-14T22:00:38.7524406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7524479Z layer_outputs = layer_module( 2025-08-14T22:00:38.7524719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7524799Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7525085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7525172Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7525488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7525573Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7525867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7526008Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7526275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7526362Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7526366Z 2025-08-14T22:00:38.7526479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7526686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7526754Z return mod(**inputs) 2025-08-14T22:00:38.7527043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7527129Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7527414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7527488Z hidden_states = self.encoder( 2025-08-14T22:00:38.7527764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7527844Z layer_outputs = layer_module( 2025-08-14T22:00:38.7528077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7528170Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7528452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7528535Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7528813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7528886Z self_outputs = self.self( 2025-08-14T22:00:38.7529158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7529262Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7529266Z 2025-08-14T22:00:38.7529369Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7529577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7529663Z return mod(**inputs) 2025-08-14T22:00:38.7529932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7530022Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7530294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7530385Z hidden_states = self.encoder( 2025-08-14T22:00:38.7530671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7530744Z layer_outputs = layer_module( 2025-08-14T22:00:38.7530986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7531066Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7531350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7531447Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7531755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7531850Z self_outputs = self.self( 2025-08-14T22:00:38.7532136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7532221Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7532224Z 2025-08-14T22:00:38.7532334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7532535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7532601Z return mod(**inputs) 2025-08-14T22:00:38.7532873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7532955Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7533230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7533301Z hidden_states = self.encoder( 2025-08-14T22:00:38.7533569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7533648Z layer_outputs = layer_module( 2025-08-14T22:00:38.7533870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7533955Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7534224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7534306Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7534583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7534653Z self_outputs = self.self( 2025-08-14T22:00:38.7534923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7535024Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7535028Z 2025-08-14T22:00:38.7535109Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7535196Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7535300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7535499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7535572Z return mod(**inputs) 2025-08-14T22:00:38.7535840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7535939Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7536219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7536295Z hidden_states = self.encoder( 2025-08-14T22:00:38.7536576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7536667Z layer_outputs = layer_module( 2025-08-14T22:00:38.7536890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7536975Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7537243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7537332Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7537600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7537671Z self_outputs = self.self( 2025-08-14T22:00:38.7537977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7538084Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7538088Z 2025-08-14T22:00:38.7538167Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7538280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7538478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7538554Z return mod(**inputs) 2025-08-14T22:00:38.7538821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7538903Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7539183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7539256Z hidden_states = self.encoder( 2025-08-14T22:00:38.7539532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7539605Z layer_outputs = layer_module( 2025-08-14T22:00:38.7539829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7539910Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7540163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7540241Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7540507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7540578Z self_outputs = self.self( 2025-08-14T22:00:38.7540844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7541003Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7541263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7541347Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7541351Z 2025-08-14T22:00:38.7541451Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7541658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7541724Z return mod(**inputs) 2025-08-14T22:00:38.7541975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7542113Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7542370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7542436Z hidden_states = self.encoder( 2025-08-14T22:00:38.7542720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7542791Z layer_outputs = layer_module( 2025-08-14T22:00:38.7543015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7543092Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7543353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7543443Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7543705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7543776Z self_outputs = self.self( 2025-08-14T22:00:38.7544072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7544230Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7544497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7544568Z x = self.pointwise(x) 2025-08-14T22:00:38.7544571Z 2025-08-14T22:00:38.7544672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7544874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7544940Z return mod(**inputs) 2025-08-14T22:00:38.7545211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7545290Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7545553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7545635Z hidden_states = self.encoder( 2025-08-14T22:00:38.7545903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7545977Z layer_outputs = layer_module( 2025-08-14T22:00:38.7546214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7546296Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7546584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7546667Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7546931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7547009Z self_outputs = self.self( 2025-08-14T22:00:38.7547275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7547436Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7547439Z 2025-08-14T22:00:38.7547542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7547748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7547820Z return mod(**inputs) 2025-08-14T22:00:38.7548081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7548179Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7548438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7548507Z hidden_states = self.encoder( 2025-08-14T22:00:38.7548767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7548856Z layer_outputs = layer_module( 2025-08-14T22:00:38.7549070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7549151Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7549410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7549498Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7549761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7549830Z self_outputs = self.self( 2025-08-14T22:00:38.7550111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7550245Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7550249Z 2025-08-14T22:00:38.7550348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7550546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7550610Z return mod(**inputs) 2025-08-14T22:00:38.7550882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7550963Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7551232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7551318Z hidden_states = self.encoder( 2025-08-14T22:00:38.7551605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7551686Z layer_outputs = layer_module( 2025-08-14T22:00:38.7551922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7552003Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7552307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7552394Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7552679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7552761Z self_outputs = self.self( 2025-08-14T22:00:38.7553042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7553185Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7553189Z 2025-08-14T22:00:38.7553274Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7553357Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7553473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7553685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7553756Z return mod(**inputs) 2025-08-14T22:00:38.7554048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7554135Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7554427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7554523Z hidden_states = self.encoder( 2025-08-14T22:00:38.7554802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7554886Z layer_outputs = layer_module( 2025-08-14T22:00:38.7555137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7555226Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7555524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7555621Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7556002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7556086Z self_outputs = self.self( 2025-08-14T22:00:38.7556392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7556546Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7556551Z 2025-08-14T22:00:38.7556681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7556908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7556980Z return mod(**inputs) 2025-08-14T22:00:38.7557274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7557371Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7557650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7557734Z hidden_states = self.encoder( 2025-08-14T22:00:38.7558019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7558093Z layer_outputs = layer_module( 2025-08-14T22:00:38.7558335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7558418Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7558700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7558792Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7559073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7559217Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7559494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7559585Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7559589Z 2025-08-14T22:00:38.7559705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7559915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7559994Z return mod(**inputs) 2025-08-14T22:00:38.7560271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7560355Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7560643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7560718Z hidden_states = self.encoder( 2025-08-14T22:00:38.7560996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7561097Z layer_outputs = layer_module( 2025-08-14T22:00:38.7561330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7561418Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7561701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7561807Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7562094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7562175Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7562495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7562624Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7562906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7563016Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7563020Z 2025-08-14T22:00:38.7563156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7563365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7563442Z return mod(**inputs) 2025-08-14T22:00:38.7563716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7563809Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7564083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7564156Z hidden_states = self.encoder( 2025-08-14T22:00:38.7564445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7564515Z layer_outputs = layer_module( 2025-08-14T22:00:38.7564740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7564819Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7565092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7565188Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7565457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7565538Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7565860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7565987Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7566274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7566395Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7566618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7566699Z return self.act(input) 2025-08-14T22:00:38.7566703Z 2025-08-14T22:00:38.7566810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7567024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7567089Z return mod(**inputs) 2025-08-14T22:00:38.7567351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7567461Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7567723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7567794Z hidden_states = self.encoder( 2025-08-14T22:00:38.7568067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7568159Z layer_outputs = layer_module( 2025-08-14T22:00:38.7568389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7568465Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7568734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7568826Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7569087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7569165Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7569508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7569663Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7569947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7570035Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7570039Z 2025-08-14T22:00:38.7570148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7570362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7570430Z return mod(**inputs) 2025-08-14T22:00:38.7570717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7570800Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7571066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7571148Z hidden_states = self.encoder( 2025-08-14T22:00:38.7571410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7571487Z layer_outputs = layer_module( 2025-08-14T22:00:38.7571705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7571782Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7572050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7572133Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7572396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7572476Z self_outputs = self.self( 2025-08-14T22:00:38.7572740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7572843Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7572846Z 2025-08-14T22:00:38.7572949Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7573142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7573214Z return mod(**inputs) 2025-08-14T22:00:38.7573477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7573557Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7573867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7573942Z hidden_states = self.encoder( 2025-08-14T22:00:38.7574232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7574321Z layer_outputs = layer_module( 2025-08-14T22:00:38.7574554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7574643Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7574939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7575030Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7575329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7575402Z self_outputs = self.self( 2025-08-14T22:00:38.7575704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7575809Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7575814Z 2025-08-14T22:00:38.7575922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7576145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7576211Z return mod(**inputs) 2025-08-14T22:00:38.7576484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7576565Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7576831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7576913Z hidden_states = self.encoder( 2025-08-14T22:00:38.7577179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7577258Z layer_outputs = layer_module( 2025-08-14T22:00:38.7577478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7577558Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7577828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7577908Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7578171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7578252Z self_outputs = self.self( 2025-08-14T22:00:38.7578518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7578621Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7578624Z 2025-08-14T22:00:38.7578706Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7578788Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7578900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7579096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7579160Z return mod(**inputs) 2025-08-14T22:00:38.7579434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7579518Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7579790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7579879Z hidden_states = self.encoder( 2025-08-14T22:00:38.7580146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7580222Z layer_outputs = layer_module( 2025-08-14T22:00:38.7580445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7580543Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7580804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7580881Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7581147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7581216Z self_outputs = self.self( 2025-08-14T22:00:38.7581476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7581588Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7581592Z 2025-08-14T22:00:38.7581684Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7581809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7582004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7582071Z return mod(**inputs) 2025-08-14T22:00:38.7582340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7582423Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7582694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7582767Z hidden_states = self.encoder( 2025-08-14T22:00:38.7583031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7583112Z layer_outputs = layer_module( 2025-08-14T22:00:38.7583334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7583420Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7583710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7583799Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7584085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7584161Z self_outputs = self.self( 2025-08-14T22:00:38.7584439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7584621Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7584911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7585006Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7585011Z 2025-08-14T22:00:38.7585124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7585334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7585420Z return mod(**inputs) 2025-08-14T22:00:38.7585684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7585768Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7586047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7586141Z hidden_states = self.encoder( 2025-08-14T22:00:38.7586424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7586498Z layer_outputs = layer_module( 2025-08-14T22:00:38.7586729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7586832Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7587108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7587190Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7587478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7587551Z self_outputs = self.self( 2025-08-14T22:00:38.7587842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7588011Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7588333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7588420Z x = self.pointwise(x) 2025-08-14T22:00:38.7588425Z 2025-08-14T22:00:38.7588537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7588761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7588831Z return mod(**inputs) 2025-08-14T22:00:38.7589126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7589220Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7589499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7589575Z hidden_states = self.encoder( 2025-08-14T22:00:38.7589864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7589940Z layer_outputs = layer_module( 2025-08-14T22:00:38.7590181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7590263Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7590542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7590638Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7590924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7591009Z self_outputs = self.self( 2025-08-14T22:00:38.7591296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7591460Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7591467Z 2025-08-14T22:00:38.7591586Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7591799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7591870Z return mod(**inputs) 2025-08-14T22:00:38.7592164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7592252Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7592545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7592642Z hidden_states = self.encoder( 2025-08-14T22:00:38.7592928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7593013Z layer_outputs = layer_module( 2025-08-14T22:00:38.7593252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7593360Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7593645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7593732Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7594028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7594103Z self_outputs = self.self( 2025-08-14T22:00:38.7594387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7594523Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7594527Z 2025-08-14T22:00:38.7594655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7594900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7594973Z return mod(**inputs) 2025-08-14T22:00:38.7595262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7595355Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7595647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7595729Z hidden_states = self.encoder( 2025-08-14T22:00:38.7596317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7596403Z layer_outputs = layer_module( 2025-08-14T22:00:38.7596654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7596738Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7597039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7597131Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7597411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7597491Z self_outputs = self.self( 2025-08-14T22:00:38.7597770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7597906Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7597911Z 2025-08-14T22:00:38.7598002Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7598085Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7598201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7598412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7598483Z return mod(**inputs) 2025-08-14T22:00:38.7598770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7598855Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7599135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7599217Z hidden_states = self.encoder( 2025-08-14T22:00:38.7599493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7599596Z layer_outputs = layer_module( 2025-08-14T22:00:38.7599826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7599905Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7600212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7600296Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7600569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7600649Z self_outputs = self.self( 2025-08-14T22:00:38.7600923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7601049Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7601054Z 2025-08-14T22:00:38.7601160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7601394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7601473Z return mod(**inputs) 2025-08-14T22:00:38.7601774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7601867Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7602148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7602221Z hidden_states = self.encoder( 2025-08-14T22:00:38.7602512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7602585Z layer_outputs = layer_module( 2025-08-14T22:00:38.7602820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7602910Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7603191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7603285Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7603567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7603704Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7603996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7604084Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7604088Z 2025-08-14T22:00:38.7604202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7604411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7604479Z return mod(**inputs) 2025-08-14T22:00:38.7604766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7604854Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7605133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7605215Z hidden_states = self.encoder( 2025-08-14T22:00:38.7605492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7605571Z layer_outputs = layer_module( 2025-08-14T22:00:38.7605803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7605901Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7606182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7606275Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7606562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7606661Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7606980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7607117Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7607401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7607488Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7607501Z 2025-08-14T22:00:38.7607610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7607826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7607917Z return mod(**inputs) 2025-08-14T22:00:38.7608213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7608301Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7608584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7608803Z hidden_states = self.encoder( 2025-08-14T22:00:38.7609105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7609181Z layer_outputs = layer_module( 2025-08-14T22:00:38.7609411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7609504Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7609788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7609879Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7610170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7610251Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7610578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7610704Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7610986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7611116Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7611347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7611424Z return self.act(input) 2025-08-14T22:00:38.7611428Z 2025-08-14T22:00:38.7611536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7611733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7611804Z return mod(**inputs) 2025-08-14T22:00:38.7612069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7612149Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7612425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7612545Z hidden_states = self.encoder( 2025-08-14T22:00:38.7612842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7612917Z layer_outputs = layer_module( 2025-08-14T22:00:38.7613157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7613298Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7613559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7613642Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7613911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7613986Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7614292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7614426Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7614747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7614841Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7614845Z 2025-08-14T22:00:38.7614946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7615145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7615210Z return mod(**inputs) 2025-08-14T22:00:38.7615470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7615560Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7615820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7615901Z hidden_states = self.encoder( 2025-08-14T22:00:38.7616164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7616235Z layer_outputs = layer_module( 2025-08-14T22:00:38.7616460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7616537Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7616797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7616886Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7617147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7617227Z self_outputs = self.self( 2025-08-14T22:00:38.7617489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7617582Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7617586Z 2025-08-14T22:00:38.7617699Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7617896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7617960Z return mod(**inputs) 2025-08-14T22:00:38.7618237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7618322Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7618588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7618658Z hidden_states = self.encoder( 2025-08-14T22:00:38.7618937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7619015Z layer_outputs = layer_module( 2025-08-14T22:00:38.7619237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7619337Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7619606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7619688Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7619961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7620029Z self_outputs = self.self( 2025-08-14T22:00:38.7620296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7620394Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7620398Z 2025-08-14T22:00:38.7620508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7620732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7620814Z return mod(**inputs) 2025-08-14T22:00:38.7621071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7621159Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7621428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7621505Z hidden_states = self.encoder( 2025-08-14T22:00:38.7621766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7621838Z layer_outputs = layer_module( 2025-08-14T22:00:38.7622064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7622142Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7622409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7622498Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7622765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7622841Z self_outputs = self.self( 2025-08-14T22:00:38.7623101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7623193Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7623196Z 2025-08-14T22:00:38.7623287Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7623364Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7623473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7623672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7623738Z return mod(**inputs) 2025-08-14T22:00:38.7624010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7624089Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7624360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7624436Z hidden_states = self.encoder( 2025-08-14T22:00:38.7624696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7624788Z layer_outputs = layer_module( 2025-08-14T22:00:38.7625010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7625086Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7625356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7625450Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7625708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7625785Z self_outputs = self.self( 2025-08-14T22:00:38.7626094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7626206Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7626209Z 2025-08-14T22:00:38.7626287Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7626389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7626588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7626667Z return mod(**inputs) 2025-08-14T22:00:38.7626958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7627041Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7627299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7627375Z hidden_states = self.encoder( 2025-08-14T22:00:38.7627634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7627705Z layer_outputs = layer_module( 2025-08-14T22:00:38.7627970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7628051Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7628340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7628429Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7628715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7628797Z self_outputs = self.self( 2025-08-14T22:00:38.7629081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7629254Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7629529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7629611Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7629615Z 2025-08-14T22:00:38.7629730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7629939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7630010Z return mod(**inputs) 2025-08-14T22:00:38.7630292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7630376Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7630657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7630731Z hidden_states = self.encoder( 2025-08-14T22:00:38.7631006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7631108Z layer_outputs = layer_module( 2025-08-14T22:00:38.7631339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7631427Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7631706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7631808Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7632089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7632163Z self_outputs = self.self( 2025-08-14T22:00:38.7632439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7632611Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7632892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7632975Z x = self.pointwise(x) 2025-08-14T22:00:38.7632983Z 2025-08-14T22:00:38.7633110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7633336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7633417Z return mod(**inputs) 2025-08-14T22:00:38.7633701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7633799Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7634082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7634158Z hidden_states = self.encoder( 2025-08-14T22:00:38.7634448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7634524Z layer_outputs = layer_module( 2025-08-14T22:00:38.7634763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7634856Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7635142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7635235Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7635538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7635613Z self_outputs = self.self( 2025-08-14T22:00:38.7635981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7636149Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7636157Z 2025-08-14T22:00:38.7636274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7636490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7636562Z return mod(**inputs) 2025-08-14T22:00:38.7636862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7636950Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7637253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7637344Z hidden_states = self.encoder( 2025-08-14T22:00:38.7637648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7637736Z layer_outputs = layer_module( 2025-08-14T22:00:38.7637996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7638079Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7638377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7638483Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7638767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7638851Z self_outputs = self.self( 2025-08-14T22:00:38.7639153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7639290Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7639293Z 2025-08-14T22:00:38.7639404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7639619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7639696Z return mod(**inputs) 2025-08-14T22:00:38.7640059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7640157Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7640443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7640519Z hidden_states = self.encoder( 2025-08-14T22:00:38.7640813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7640888Z layer_outputs = layer_module( 2025-08-14T22:00:38.7641125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7641216Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7641502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7641599Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7641889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7641963Z self_outputs = self.self( 2025-08-14T22:00:38.7642254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7642393Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7642397Z 2025-08-14T22:00:38.7642489Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7642572Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7642682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7642903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7642974Z return mod(**inputs) 2025-08-14T22:00:38.7643264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7643359Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7643646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7643731Z hidden_states = self.encoder( 2025-08-14T22:00:38.7644016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7644093Z layer_outputs = layer_module( 2025-08-14T22:00:38.7644341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7644441Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7644727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7644822Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7645110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7645217Z self_outputs = self.self( 2025-08-14T22:00:38.7645506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7645638Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7645642Z 2025-08-14T22:00:38.7645756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7645962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7646041Z return mod(**inputs) 2025-08-14T22:00:38.7646330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7646425Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7646707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7646780Z hidden_states = self.encoder( 2025-08-14T22:00:38.7647043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7647120Z layer_outputs = layer_module( 2025-08-14T22:00:38.7647336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7647419Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7647687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7647769Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7648045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7648175Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7648445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7648527Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7648531Z 2025-08-14T22:00:38.7648632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7648834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7648900Z return mod(**inputs) 2025-08-14T22:00:38.7649166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7649256Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7649521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7649606Z hidden_states = self.encoder( 2025-08-14T22:00:38.7649886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7649959Z layer_outputs = layer_module( 2025-08-14T22:00:38.7650203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7650283Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7650579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7650684Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7650955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7651040Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7651333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7651469Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7651738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7651822Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7651825Z 2025-08-14T22:00:38.7651933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7652129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7652197Z return mod(**inputs) 2025-08-14T22:00:38.7652466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7652559Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7652841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7652914Z hidden_states = self.encoder( 2025-08-14T22:00:38.7653174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7653274Z layer_outputs = layer_module( 2025-08-14T22:00:38.7653492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7653568Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7653842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7653926Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7654195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7654271Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7654565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7654692Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7654959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7655083Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7655298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7655373Z return self.act(input) 2025-08-14T22:00:38.7655376Z 2025-08-14T22:00:38.7655487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7655690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7655760Z return mod(**inputs) 2025-08-14T22:00:38.7656049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7656133Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7656426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7656500Z hidden_states = self.encoder( 2025-08-14T22:00:38.7656769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7656850Z layer_outputs = layer_module( 2025-08-14T22:00:38.7657088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7657174Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7657445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7657547Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7657817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7657892Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7658193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7658335Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7658602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7658695Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7658698Z 2025-08-14T22:00:38.7658815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7659038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7659116Z return mod(**inputs) 2025-08-14T22:00:38.7659389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7659480Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7659748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7659818Z hidden_states = self.encoder( 2025-08-14T22:00:38.7660101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7660172Z layer_outputs = layer_module( 2025-08-14T22:00:38.7660389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7660475Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7660741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7660828Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7661087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7661156Z self_outputs = self.self( 2025-08-14T22:00:38.7661423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7661515Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7661520Z 2025-08-14T22:00:38.7661629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7661824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7661887Z return mod(**inputs) 2025-08-14T22:00:38.7662162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7662243Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7662506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7662584Z hidden_states = self.encoder( 2025-08-14T22:00:38.7662846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7662920Z layer_outputs = layer_module( 2025-08-14T22:00:38.7663156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7663232Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7663503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7663603Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7663871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7663947Z self_outputs = self.self( 2025-08-14T22:00:38.7664228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7664316Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7664319Z 2025-08-14T22:00:38.7664418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7664615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7664690Z return mod(**inputs) 2025-08-14T22:00:38.7664978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7665107Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7665376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7665446Z hidden_states = self.encoder( 2025-08-14T22:00:38.7665715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7665784Z layer_outputs = layer_module( 2025-08-14T22:00:38.7666002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7666085Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7666351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7666440Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7666708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7666779Z self_outputs = self.self( 2025-08-14T22:00:38.7667055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7667147Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7667151Z 2025-08-14T22:00:38.7667237Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7667315Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7667416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7667621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7667687Z return mod(**inputs) 2025-08-14T22:00:38.7667950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7668039Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7668303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7668380Z hidden_states = self.encoder( 2025-08-14T22:00:38.7668660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7668732Z layer_outputs = layer_module( 2025-08-14T22:00:38.7668973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7669053Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7669370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7669465Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7669761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7669863Z self_outputs = self.self( 2025-08-14T22:00:38.7670148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7670255Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7670259Z 2025-08-14T22:00:38.7670348Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7670454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7670666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7670738Z return mod(**inputs) 2025-08-14T22:00:38.7671030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7671139Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7671434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7671511Z hidden_states = self.encoder( 2025-08-14T22:00:38.7671800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7671874Z layer_outputs = layer_module( 2025-08-14T22:00:38.7672107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7672186Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7672484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7672576Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7672853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7672935Z self_outputs = self.self( 2025-08-14T22:00:38.7673213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7673379Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7673685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7673764Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7673768Z 2025-08-14T22:00:38.7673873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7674089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7674159Z return mod(**inputs) 2025-08-14T22:00:38.7674441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7674528Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7674806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7674891Z hidden_states = self.encoder( 2025-08-14T22:00:38.7675164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7675244Z layer_outputs = layer_module( 2025-08-14T22:00:38.7675472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7675554Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7675942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7676040Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7676336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7676443Z self_outputs = self.self( 2025-08-14T22:00:38.7676732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7676911Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7677218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7677291Z x = self.pointwise(x) 2025-08-14T22:00:38.7677295Z 2025-08-14T22:00:38.7677415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7677625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7677705Z return mod(**inputs) 2025-08-14T22:00:38.7678041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7678133Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7678423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7678498Z hidden_states = self.encoder( 2025-08-14T22:00:38.7678800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7678885Z layer_outputs = layer_module( 2025-08-14T22:00:38.7679122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7679218Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7679502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7679590Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7679881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7679956Z self_outputs = self.self( 2025-08-14T22:00:38.7680237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7680406Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7680410Z 2025-08-14T22:00:38.7680519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7680740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7680814Z return mod(**inputs) 2025-08-14T22:00:38.7681098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7681194Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7681479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7681563Z hidden_states = self.encoder( 2025-08-14T22:00:38.7681849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7681925Z layer_outputs = layer_module( 2025-08-14T22:00:38.7682171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7682255Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7682563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7682658Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7682950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7683049Z self_outputs = self.self( 2025-08-14T22:00:38.7683335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7683465Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7683469Z 2025-08-14T22:00:38.7683589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7683802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7683879Z return mod(**inputs) 2025-08-14T22:00:38.7684176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7684266Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7684595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7684676Z hidden_states = self.encoder( 2025-08-14T22:00:38.7684971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7685055Z layer_outputs = layer_module( 2025-08-14T22:00:38.7685292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7685384Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7685674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7685762Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7686054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7686130Z self_outputs = self.self( 2025-08-14T22:00:38.7686426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7686566Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7686570Z 2025-08-14T22:00:38.7686657Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7686749Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7686862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7687072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7687150Z return mod(**inputs) 2025-08-14T22:00:38.7687436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7687532Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7687821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7687901Z hidden_states = self.encoder( 2025-08-14T22:00:38.7688196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7688271Z layer_outputs = layer_module( 2025-08-14T22:00:38.7688507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7688598Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7688881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7688993Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7689278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7689353Z self_outputs = self.self( 2025-08-14T22:00:38.7689646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7689785Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7689789Z 2025-08-14T22:00:38.7689904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7690118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7690197Z return mod(**inputs) 2025-08-14T22:00:38.7690485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7690571Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7690850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7690945Z hidden_states = self.encoder( 2025-08-14T22:00:38.7691251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7691334Z layer_outputs = layer_module( 2025-08-14T22:00:38.7691565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7691645Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7691931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7692013Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7692308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7712224Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7712794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7712905Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7712914Z 2025-08-14T22:00:38.7713053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7713282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7713362Z return mod(**inputs) 2025-08-14T22:00:38.7713680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7713779Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7714094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7714188Z hidden_states = self.encoder( 2025-08-14T22:00:38.7714487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7714580Z layer_outputs = layer_module( 2025-08-14T22:00:38.7714837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7714929Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7715233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7715330Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7715631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7715717Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7716310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7716460Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7716752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7716906Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7716912Z 2025-08-14T22:00:38.7717037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7717267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7717353Z return mod(**inputs) 2025-08-14T22:00:38.7717646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7717741Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7718040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7718123Z hidden_states = self.encoder( 2025-08-14T22:00:38.7718493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7718575Z layer_outputs = layer_module( 2025-08-14T22:00:38.7718820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7718916Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7719207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7719306Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7719593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7719678Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7720011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7720148Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7720439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7720570Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7720800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7720887Z return self.act(input) 2025-08-14T22:00:38.7720892Z 2025-08-14T22:00:38.7721006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7721227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7721310Z return mod(**inputs) 2025-08-14T22:00:38.7721599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7721699Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7721990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7722067Z hidden_states = self.encoder( 2025-08-14T22:00:38.7722362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7722432Z layer_outputs = layer_module( 2025-08-14T22:00:38.7722651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7722735Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7723014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7723111Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7723385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7723483Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7723809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7723951Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7724240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7724326Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7724330Z 2025-08-14T22:00:38.7724441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7724659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7724728Z return mod(**inputs) 2025-08-14T22:00:38.7725039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7725134Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7725409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7725491Z hidden_states = self.encoder( 2025-08-14T22:00:38.7725777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7725845Z layer_outputs = layer_module( 2025-08-14T22:00:38.7726067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7726145Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7726412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7726493Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7726758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7726837Z self_outputs = self.self( 2025-08-14T22:00:38.7727100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7727193Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7727197Z 2025-08-14T22:00:38.7727307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7727503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7727577Z return mod(**inputs) 2025-08-14T22:00:38.7727837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7727918Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7728188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7728259Z hidden_states = self.encoder( 2025-08-14T22:00:38.7728530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7728598Z layer_outputs = layer_module( 2025-08-14T22:00:38.7728814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7728900Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7729159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7729259Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7729537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7729624Z self_outputs = self.self( 2025-08-14T22:00:38.7729895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7729976Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7729979Z 2025-08-14T22:00:38.7730082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7730288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7730354Z return mod(**inputs) 2025-08-14T22:00:38.7730618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7730708Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7730986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7731066Z hidden_states = self.encoder( 2025-08-14T22:00:38.7731349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7731422Z layer_outputs = layer_module( 2025-08-14T22:00:38.7731653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7731730Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7732013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7732095Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7732375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7732454Z self_outputs = self.self( 2025-08-14T22:00:38.7732733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7732831Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7732842Z 2025-08-14T22:00:38.7732929Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7733010Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7733124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7733331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7733397Z return mod(**inputs) 2025-08-14T22:00:38.7733681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7733766Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7734053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7734127Z hidden_states = self.encoder( 2025-08-14T22:00:38.7734408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7734488Z layer_outputs = layer_module( 2025-08-14T22:00:38.7734717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7734797Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7735082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7735164Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7735457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7735528Z self_outputs = self.self( 2025-08-14T22:00:38.7735798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7735928Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7735932Z 2025-08-14T22:00:38.7736023Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7736125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7736331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7736397Z return mod(**inputs) 2025-08-14T22:00:38.7736666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7736745Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7737022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7737104Z hidden_states = self.encoder( 2025-08-14T22:00:38.7737424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7737517Z layer_outputs = layer_module( 2025-08-14T22:00:38.7737738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7737815Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7738090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7738171Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7738449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7738532Z self_outputs = self.self( 2025-08-14T22:00:38.7738813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7739000Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7739267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7739345Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7739349Z 2025-08-14T22:00:38.7739459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7739657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7739742Z return mod(**inputs) 2025-08-14T22:00:38.7740005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7740095Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7740368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7740439Z hidden_states = self.encoder( 2025-08-14T22:00:38.7740702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7740785Z layer_outputs = layer_module( 2025-08-14T22:00:38.7741005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7741081Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7741354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7741436Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7741725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7741793Z self_outputs = self.self( 2025-08-14T22:00:38.7742065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7742257Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7742534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7742625Z x = self.pointwise(x) 2025-08-14T22:00:38.7742629Z 2025-08-14T22:00:38.7742732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7742925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7742999Z return mod(**inputs) 2025-08-14T22:00:38.7743259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7743347Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7743638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7743711Z hidden_states = self.encoder( 2025-08-14T22:00:38.7743980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7744048Z layer_outputs = layer_module( 2025-08-14T22:00:38.7744264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7744349Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7744612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7744701Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7744964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7745030Z self_outputs = self.self( 2025-08-14T22:00:38.7745300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7745454Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7745458Z 2025-08-14T22:00:38.7745566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7745760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7745825Z return mod(**inputs) 2025-08-14T22:00:38.7746093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7746176Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7746444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7746522Z hidden_states = self.encoder( 2025-08-14T22:00:38.7746789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7746869Z layer_outputs = layer_module( 2025-08-14T22:00:38.7747086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7747161Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7747431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7747510Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7747785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7747861Z self_outputs = self.self( 2025-08-14T22:00:38.7748122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7748267Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7748271Z 2025-08-14T22:00:38.7748372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7748564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7748637Z return mod(**inputs) 2025-08-14T22:00:38.7748899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7748992Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7749264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7749339Z hidden_states = self.encoder( 2025-08-14T22:00:38.7749640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7749736Z layer_outputs = layer_module( 2025-08-14T22:00:38.7749968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7750057Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7750336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7750423Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7750684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7750752Z self_outputs = self.self( 2025-08-14T22:00:38.7751022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7751150Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7751153Z 2025-08-14T22:00:38.7751242Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7751319Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7751419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7751622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7751686Z return mod(**inputs) 2025-08-14T22:00:38.7751951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7752038Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7752301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7752379Z hidden_states = self.encoder( 2025-08-14T22:00:38.7752643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7752713Z layer_outputs = layer_module( 2025-08-14T22:00:38.7752939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7753014Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7753276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7753363Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7753625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7753717Z self_outputs = self.self( 2025-08-14T22:00:38.7753998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7754121Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7754126Z 2025-08-14T22:00:38.7754243Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7754465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7754541Z return mod(**inputs) 2025-08-14T22:00:38.7754839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7754928Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7755211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7755288Z hidden_states = self.encoder( 2025-08-14T22:00:38.7755571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7755657Z layer_outputs = layer_module( 2025-08-14T22:00:38.7756006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7756105Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7756386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7756474Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7756772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7756916Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7757228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7757320Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7757324Z 2025-08-14T22:00:38.7757433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7757646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7757717Z return mod(**inputs) 2025-08-14T22:00:38.7757997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7758090Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7758387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7758473Z hidden_states = self.encoder( 2025-08-14T22:00:38.7758747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7758824Z layer_outputs = layer_module( 2025-08-14T22:00:38.7759062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7759143Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7759432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7759524Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7759796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7759883Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7760194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7760323Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7760626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7760715Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7760719Z 2025-08-14T22:00:38.7760834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7761063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7761131Z return mod(**inputs) 2025-08-14T22:00:38.7761416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7761500Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7761798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7761873Z hidden_states = self.encoder( 2025-08-14T22:00:38.7762154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7762234Z layer_outputs = layer_module( 2025-08-14T22:00:38.7762521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7762604Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7762892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7762978Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7763257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7763337Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7763645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7763791Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7764069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7764197Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7764420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7764493Z return self.act(input) 2025-08-14T22:00:38.7764497Z 2025-08-14T22:00:38.7764622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7764819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7764893Z return mod(**inputs) 2025-08-14T22:00:38.7765154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7765238Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7765505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7765577Z hidden_states = self.encoder( 2025-08-14T22:00:38.7765840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7765927Z layer_outputs = layer_module( 2025-08-14T22:00:38.7766139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7766221Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7766474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7766554Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7766812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7766905Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7767200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7767345Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7767604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7767691Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7767695Z 2025-08-14T22:00:38.7767795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7767990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7768052Z return mod(**inputs) 2025-08-14T22:00:38.7768309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7768398Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7768683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7768754Z hidden_states = self.encoder( 2025-08-14T22:00:38.7769025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7769095Z layer_outputs = layer_module( 2025-08-14T22:00:38.7769318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7769393Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7769653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7769742Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7770006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7770076Z self_outputs = self.self( 2025-08-14T22:00:38.7770355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T22:00:38.7770455Z mixed_query_layer = self.query(hidden_states) 2025-08-14T22:00:38.7770459Z 2025-08-14T22:00:38.7770573Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7770777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7770845Z return mod(**inputs) 2025-08-14T22:00:38.7771125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7771207Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7771477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7771548Z hidden_states = self.encoder( 2025-08-14T22:00:38.7771809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7771886Z layer_outputs = layer_module( 2025-08-14T22:00:38.7772102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7772177Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7772444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7772524Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7772789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7772875Z self_outputs = self.self( 2025-08-14T22:00:38.7773143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T22:00:38.7773234Z mixed_key_layer = self.key(hidden_states) 2025-08-14T22:00:38.7773252Z 2025-08-14T22:00:38.7773353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7773552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7773617Z return mod(**inputs) 2025-08-14T22:00:38.7773875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7773962Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7774221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7774292Z hidden_states = self.encoder( 2025-08-14T22:00:38.7774575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7774649Z layer_outputs = layer_module( 2025-08-14T22:00:38.7774900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7774982Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7775259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7775350Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7775651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7775723Z self_outputs = self.self( 2025-08-14T22:00:38.7776012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T22:00:38.7776112Z mixed_value_layer = self.value(hidden_states) 2025-08-14T22:00:38.7776116Z 2025-08-14T22:00:38.7776208Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7776292Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7776401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7776623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7776688Z return mod(**inputs) 2025-08-14T22:00:38.7776961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7777041Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7777307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7777388Z hidden_states = self.encoder( 2025-08-14T22:00:38.7777663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7777738Z layer_outputs = layer_module( 2025-08-14T22:00:38.7777977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7778057Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7778340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7778423Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7778721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7778801Z self_outputs = self.self( 2025-08-14T22:00:38.7779088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T22:00:38.7779215Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T22:00:38.7779218Z 2025-08-14T22:00:38.7779298Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7779399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7779624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7779686Z return mod(**inputs) 2025-08-14T22:00:38.7779940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7780025Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7780278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7780353Z hidden_states = self.encoder( 2025-08-14T22:00:38.7780610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7780676Z layer_outputs = layer_module( 2025-08-14T22:00:38.7780905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7780993Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7781249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7781335Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7781588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7781661Z self_outputs = self.self( 2025-08-14T22:00:38.7781914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7782073Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7782343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T22:00:38.7782419Z x = self.depthwise(hidden_states) 2025-08-14T22:00:38.7782424Z 2025-08-14T22:00:38.7782533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7782738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7782801Z return mod(**inputs) 2025-08-14T22:00:38.7783059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7783138Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7783397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7783475Z hidden_states = self.encoder( 2025-08-14T22:00:38.7783734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7783811Z layer_outputs = layer_module( 2025-08-14T22:00:38.7784030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7784105Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7784374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7784452Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7784717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7784784Z self_outputs = self.self( 2025-08-14T22:00:38.7785045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T22:00:38.7785234Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T22:00:38.7785501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T22:00:38.7785587Z x = self.pointwise(x) 2025-08-14T22:00:38.7785598Z 2025-08-14T22:00:38.7785699Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7785895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7785965Z return mod(**inputs) 2025-08-14T22:00:38.7786237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7786315Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7786592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7786664Z hidden_states = self.encoder( 2025-08-14T22:00:38.7786953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7787040Z layer_outputs = layer_module( 2025-08-14T22:00:38.7787258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7787342Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7787604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7787683Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7787953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7788023Z self_outputs = self.self( 2025-08-14T22:00:38.7788293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T22:00:38.7788442Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T22:00:38.7788448Z 2025-08-14T22:00:38.7788548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7788751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7788817Z return mod(**inputs) 2025-08-14T22:00:38.7789119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7789203Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7789491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7789574Z hidden_states = self.encoder( 2025-08-14T22:00:38.7789864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7789939Z layer_outputs = layer_module( 2025-08-14T22:00:38.7790187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7790268Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7790569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7790652Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7790941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7791022Z self_outputs = self.self( 2025-08-14T22:00:38.7791313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T22:00:38.7791466Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T22:00:38.7791471Z 2025-08-14T22:00:38.7791579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7791790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7791888Z return mod(**inputs) 2025-08-14T22:00:38.7792184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7792269Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7792584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7792659Z hidden_states = self.encoder( 2025-08-14T22:00:38.7792961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7793036Z layer_outputs = layer_module( 2025-08-14T22:00:38.7793296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7793400Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7793689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7793772Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7794070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7794141Z self_outputs = self.self( 2025-08-14T22:00:38.7794430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T22:00:38.7794567Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T22:00:38.7794573Z 2025-08-14T22:00:38.7794656Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7794744Z cudagraph partition due to non gpu ops 2025-08-14T22:00:38.7794852Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7795063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7795132Z return mod(**inputs) 2025-08-14T22:00:38.7795427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7795517Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7795897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7795984Z hidden_states = self.encoder( 2025-08-14T22:00:38.7796291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7796370Z layer_outputs = layer_module( 2025-08-14T22:00:38.7796620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7796704Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7796995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7797092Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7797378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T22:00:38.7797460Z self_outputs = self.self( 2025-08-14T22:00:38.7797748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T22:00:38.7797861Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T22:00:38.7797883Z 2025-08-14T22:00:38.7797995Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7798191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7798258Z return mod(**inputs) 2025-08-14T22:00:38.7798531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7799482Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7799753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7799824Z hidden_states = self.encoder( 2025-08-14T22:00:38.7800091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7800170Z layer_outputs = layer_module( 2025-08-14T22:00:38.7800392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7800467Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7800779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T22:00:38.7800867Z self_attention_outputs = self.attention( 2025-08-14T22:00:38.7801151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T22:00:38.7801285Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T22:00:38.7801561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T22:00:38.7801657Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7801660Z 2025-08-14T22:00:38.7801767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7801981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7802047Z return mod(**inputs) 2025-08-14T22:00:38.7802325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7802417Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7802690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7802771Z hidden_states = self.encoder( 2025-08-14T22:00:38.7803044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7803116Z layer_outputs = layer_module( 2025-08-14T22:00:38.7803351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7803431Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7803708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7803803Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7804073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7804161Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7804470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7804600Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7804891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T22:00:38.7804979Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7805001Z 2025-08-14T22:00:38.7805130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7805339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7805409Z return mod(**inputs) 2025-08-14T22:00:38.7805710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7805810Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7806085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7806166Z hidden_states = self.encoder( 2025-08-14T22:00:38.7806439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7806518Z layer_outputs = layer_module( 2025-08-14T22:00:38.7806746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7806827Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7807125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7807239Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7807515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7807601Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7807914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T22:00:38.7808046Z intermediate_output = self.intermediate(attention_output) 2025-08-14T22:00:38.7808322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T22:00:38.7808440Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T22:00:38.7808833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T22:00:38.7808917Z return self.act(input) 2025-08-14T22:00:38.7808921Z 2025-08-14T22:00:38.7809041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7809246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7809315Z return mod(**inputs) 2025-08-14T22:00:38.7809603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T22:00:38.7809688Z generator_hidden_states = self.convbert( 2025-08-14T22:00:38.7809963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T22:00:38.7810049Z hidden_states = self.encoder( 2025-08-14T22:00:38.7810325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T22:00:38.7810407Z layer_outputs = layer_module( 2025-08-14T22:00:38.7810639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T22:00:38.7810723Z return super().__call__(*args, **kwargs) 2025-08-14T22:00:38.7811010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T22:00:38.7811101Z layer_output = apply_chunking_to_forward( 2025-08-14T22:00:38.7811382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T22:00:38.7811462Z return forward_fn(*input_tensors) 2025-08-14T22:00:38.7811774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T22:00:38.7811967Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T22:00:38.7812248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T22:00:38.7812374Z hidden_states = self.dense(hidden_states) 2025-08-14T22:00:38.7812386Z 2025-08-14T22:00:38.7812494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7812700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7812778Z return mod(**inputs) 2025-08-14T22:00:38.7813058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 938, in forward 2025-08-14T22:00:38.7813219Z prediction_scores = self.generator_predictions(generator_sequence_output) 2025-08-14T22:00:38.7813508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 876, in forward 2025-08-14T22:00:38.7813618Z hidden_states = self.dense(generator_hidden_states) 2025-08-14T22:00:38.7813645Z 2025-08-14T22:00:38.7813761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7813994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7814064Z return mod(**inputs) 2025-08-14T22:00:38.7814348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 939, in forward 2025-08-14T22:00:38.7814482Z prediction_scores = self.generator_lm_head(prediction_scores) 2025-08-14T22:00:38.7814486Z 2025-08-14T22:00:38.7814597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T22:00:38.7814803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T22:00:38.7814872Z return mod(**inputs) 2025-08-14T22:00:38.7815156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 945, in forward 2025-08-14T22:00:38.7815334Z loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T22:00:38.7815338Z 2025-08-14T22:00:48.6909989Z Compilation time (from dynamo_timed): 21.506015366 2025-08-14T22:00:48.6969977Z pass 2025-08-14T22:00:48.6970368Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T22:00:48.6973964Z TIMING: _recursive_pre_grad_passes:0.01098 _recursive_joint_graph_passes:0.64661 _recursive_post_grad_passes:0.18973 async_compile.wait:0.606 code_gen:9.28438 inductor_compile:11.76021 backend_compile:16.94603 gc:0.0017 entire_frame_compile:21.50602 total_wall_time:21.50602 2025-08-14T22:00:48.6975016Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:23085 | FakeTensor.__torch_dispatch__:7564 | ProxyTorchDispatchMode.__torch_dispatch__:8630 2025-08-14T22:00:48.6975557Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2025-08-14T22:00:50.7974473Z accuracy pass_rate=95.35% 2025-08-14T22:00:50.7979092Z calls_captured gmean=0.00x mean=609.233x 2025-08-14T22:00:50.7984766Z unique_graphs gmean=0.00x mean=1.093x 2025-08-14T22:00:50.7988175Z graph_breaks gmean=0.00x mean=0.140x 2025-08-14T22:00:50.7989325Z unique_graph_breaks gmean=0.00x mean=0.047x 2025-08-14T22:00:50.7989629Z autograd_captures gmean=0.00x mean=0.000x 2025-08-14T22:00:50.7989880Z autograd_compiles gmean=0.00x mean=0.000x 2025-08-14T22:00:50.7990118Z cudagraph_skips gmean=0.00x mean=1.093x 2025-08-14T22:00:50.7990358Z compilation_latency mean=20.210 seconds 2025-08-14T22:00:51.8464279Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/cpu_inductor_huggingface_inference.csv 2025-08-14T22:00:52.1471049Z AlbertForMaskedLM PASS 2025-08-14T22:00:52.1471388Z AlbertForQuestionAnswering PASS 2025-08-14T22:00:52.1476514Z AllenaiLongformerBase PASS 2025-08-14T22:00:52.1478939Z BartForCausalLM PASS 2025-08-14T22:00:52.1484867Z BartForConditionalGeneration PASS 2025-08-14T22:00:52.1485428Z BertForMaskedLM PASS 2025-08-14T22:00:52.1488424Z BertForQuestionAnswering PASS 2025-08-14T22:00:52.1490650Z BlenderbotForCausalLM XFAIL 2025-08-14T22:00:52.1493897Z BlenderbotSmallForCausalLM PASS 2025-08-14T22:00:52.1500182Z BlenderbotSmallForConditionalGeneration PASS 2025-08-14T22:00:52.1500510Z CamemBert PASS 2025-08-14T22:00:52.1504300Z DebertaV2ForMaskedLM XFAIL 2025-08-14T22:00:52.1508273Z DebertaV2ForQuestionAnswering PASS 2025-08-14T22:00:52.1510622Z DistilBertForMaskedLM PASS 2025-08-14T22:00:52.1517360Z DistilBertForQuestionAnswering PASS 2025-08-14T22:00:52.1523771Z DistillGPT2 PASS 2025-08-14T22:00:52.1525332Z ElectraForCausalLM PASS 2025-08-14T22:00:52.1531696Z ElectraForQuestionAnswering PASS 2025-08-14T22:00:52.1532014Z GPT2ForSequenceClassification PASS 2025-08-14T22:00:52.1539503Z GoogleFnet PASS 2025-08-14T22:00:52.1539727Z LayoutLMForMaskedLM PASS 2025-08-14T22:00:52.1543125Z LayoutLMForSequenceClassification PASS 2025-08-14T22:00:52.1543494Z M2M100ForConditionalGeneration PASS 2025-08-14T22:00:52.1550133Z MBartForCausalLM PASS 2025-08-14T22:00:52.1550428Z MBartForConditionalGeneration PASS 2025-08-14T22:00:52.1553732Z MT5ForConditionalGeneration PASS 2025-08-14T22:00:52.1554105Z MegatronBertForCausalLM PASS 2025-08-14T22:00:52.1556571Z MegatronBertForQuestionAnswering PASS 2025-08-14T22:00:52.1570174Z MobileBertForMaskedLM PASS 2025-08-14T22:00:52.1575978Z MobileBertForQuestionAnswering PASS 2025-08-14T22:00:52.1578263Z OPTForCausalLM PASS 2025-08-14T22:00:52.1578546Z PLBartForCausalLM PASS 2025-08-14T22:00:52.1578802Z PLBartForConditionalGeneration PASS 2025-08-14T22:00:52.1582676Z PegasusForCausalLM PASS 2025-08-14T22:00:52.1583025Z PegasusForConditionalGeneration PASS 2025-08-14T22:00:52.1590573Z RobertaForCausalLM PASS 2025-08-14T22:00:52.1590872Z RobertaForQuestionAnswering PASS 2025-08-14T22:00:52.1591440Z T5ForConditionalGeneration PASS 2025-08-14T22:00:52.1592317Z T5Small PASS 2025-08-14T22:00:52.1596349Z TrOCRForCausalLM PASS 2025-08-14T22:00:52.1603578Z XGLMForCausalLM PASS 2025-08-14T22:00:52.1605772Z XLNetLMHeadModel PASS 2025-08-14T22:00:52.1605998Z YituTechConvBert PASS 2025-08-14T22:00:52.2141964Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/cpu_inductor_huggingface_inference.csv 2025-08-14T22:00:52.5125495Z AlbertForMaskedLM PASS 2025-08-14T22:00:52.5130997Z AlbertForQuestionAnswering PASS 2025-08-14T22:00:52.5131512Z AllenaiLongformerBase PASS 2025-08-14T22:00:52.5131768Z BartForCausalLM PASS 2025-08-14T22:00:52.5138400Z BartForConditionalGeneration PASS 2025-08-14T22:00:52.5138739Z BertForMaskedLM PASS 2025-08-14T22:00:52.5139239Z BertForQuestionAnswering PASS 2025-08-14T22:00:52.5139478Z BlenderbotForCausalLM PASS 2025-08-14T22:00:52.5147557Z BlenderbotSmallForCausalLM PASS 2025-08-14T22:00:52.5152572Z BlenderbotSmallForConditionalGeneration PASS 2025-08-14T22:00:52.5152908Z CamemBert PASS 2025-08-14T22:00:52.5153483Z DebertaV2ForMaskedLM PASS 2025-08-14T22:00:52.5156539Z DebertaV2ForQuestionAnswering PASS 2025-08-14T22:00:52.5165200Z DistilBertForMaskedLM PASS 2025-08-14T22:00:52.5165504Z DistilBertForQuestionAnswering PASS 2025-08-14T22:00:52.5170500Z DistillGPT2 PASS 2025-08-14T22:00:52.5170809Z ElectraForCausalLM PASS 2025-08-14T22:00:52.5179578Z ElectraForQuestionAnswering PASS 2025-08-14T22:00:52.5184772Z GPT2ForSequenceClassification PASS 2025-08-14T22:00:52.5189957Z GoogleFnet PASS 2025-08-14T22:00:52.5192020Z LayoutLMForMaskedLM PASS 2025-08-14T22:00:52.5192291Z LayoutLMForSequenceClassification PASS 2025-08-14T22:00:52.5192541Z M2M100ForConditionalGeneration PASS 2025-08-14T22:00:52.5192784Z MBartForCausalLM PASS 2025-08-14T22:00:52.5196944Z MBartForConditionalGeneration PASS 2025-08-14T22:00:52.5205739Z MT5ForConditionalGeneration PASS 2025-08-14T22:00:52.5207728Z MegatronBertForCausalLM PASS 2025-08-14T22:00:52.5208166Z MegatronBertForQuestionAnswering PASS 2025-08-14T22:00:52.5214688Z MobileBertForMaskedLM PASS 2025-08-14T22:00:52.5215425Z MobileBertForQuestionAnswering PASS 2025-08-14T22:00:52.5222702Z OPTForCausalLM PASS 2025-08-14T22:00:52.5223531Z PLBartForCausalLM PASS 2025-08-14T22:00:52.5227918Z PLBartForConditionalGeneration PASS 2025-08-14T22:00:52.5228448Z PegasusForCausalLM PASS 2025-08-14T22:00:52.5228735Z PegasusForConditionalGeneration PASS 2025-08-14T22:00:52.5231048Z RobertaForCausalLM PASS 2025-08-14T22:00:52.5235524Z RobertaForQuestionAnswering PASS 2025-08-14T22:00:52.5242257Z T5ForConditionalGeneration PASS 2025-08-14T22:00:52.5242631Z T5Small PASS 2025-08-14T22:00:52.5247779Z TrOCRForCausalLM PASS 2025-08-14T22:00:52.5248132Z XGLMForCausalLM PASS_BUT_FLAKY 2025-08-14T22:00:52.5251669Z XLNetLMHeadModel PASS 2025-08-14T22:00:52.5256380Z YituTechConvBert PASS 2025-08-14T22:00:52.5762932Z + sccache_epilogue 2025-08-14T22:00:52.5767798Z + echo '::group::Sccache Compilation Log' 2025-08-14T22:00:52.5768430Z ##[group]Sccache Compilation Log 2025-08-14T22:00:52.5768724Z + echo '=================== sccache compilation log ===================' 2025-08-14T22:00:52.5769044Z =================== sccache compilation log =================== 2025-08-14T22:00:52.5769456Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-08-14T22:00:52.5995468Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-08-14T22:00:52.5996328Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-08-14T22:00:52.5996678Z + sccache --show-stats 2025-08-14T22:00:52.6037778Z Compile requests 381 2025-08-14T22:00:52.6038025Z Compile requests executed 0 2025-08-14T22:00:52.6038264Z Cache hits 0 2025-08-14T22:00:52.6038486Z Cache misses 0 2025-08-14T22:00:52.6038696Z Cache hits rate - 2025-08-14T22:00:52.6038902Z Cache timeouts 0 2025-08-14T22:00:52.6039117Z Cache read errors 0 2025-08-14T22:00:52.6039324Z Forced recaches 0 2025-08-14T22:00:52.6039530Z Cache write errors 0 2025-08-14T22:00:52.6039736Z Cache errors 0 2025-08-14T22:00:52.6039943Z Compilations 0 2025-08-14T22:00:52.6040143Z Compilation failures 0 2025-08-14T22:00:52.6040360Z Non-cacheable compilations 0 2025-08-14T22:00:52.6040579Z Non-cacheable calls 41 2025-08-14T22:00:52.6040781Z Non-compilation calls 340 2025-08-14T22:00:52.6041258Z Unsupported compiler calls 0 2025-08-14T22:00:52.6041482Z Average cache write 0.000 s 2025-08-14T22:00:52.6041710Z Average compiler 0.000 s 2025-08-14T22:00:52.6041926Z Average cache read hit 0.000 s 2025-08-14T22:00:52.6042148Z Failed distributed compilations 0 2025-08-14T22:00:52.6042351Z 2025-08-14T22:00:52.6042434Z Non-cacheable reasons: 2025-08-14T22:00:52.6042630Z -E 41 2025-08-14T22:00:52.6042772Z 2025-08-14T22:00:52.6042933Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-08-14T22:00:52.6043231Z Version (client) 0.10.0 2025-08-14T22:00:52.6043433Z + sccache --stop-server 2025-08-14T22:00:52.6058087Z Stopping sccache server... 2025-08-14T22:00:52.6068965Z Compile requests 381 2025-08-14T22:00:52.6069253Z Compile requests executed 0 2025-08-14T22:00:52.6069602Z Cache hits 0 2025-08-14T22:00:52.6069840Z Cache misses 0 2025-08-14T22:00:52.6070098Z Cache hits rate - 2025-08-14T22:00:52.6070309Z Cache timeouts 0 2025-08-14T22:00:52.6070726Z Cache read errors 0 2025-08-14T22:00:52.6070949Z Forced recaches 0 2025-08-14T22:00:52.6071232Z Cache write errors 0 2025-08-14T22:00:52.6071446Z Cache errors 0 2025-08-14T22:00:52.6071760Z Compilations 0 2025-08-14T22:00:52.6071984Z Compilation failures 0 2025-08-14T22:00:52.6072207Z Non-cacheable compilations 0 2025-08-14T22:00:52.6072462Z Non-cacheable calls 41 2025-08-14T22:00:52.6072687Z Non-compilation calls 340 2025-08-14T22:00:52.6072903Z Unsupported compiler calls 0 2025-08-14T22:00:52.6073127Z Average cache write 0.000 s 2025-08-14T22:00:52.6073355Z Average compiler 0.000 s 2025-08-14T22:00:52.6073578Z Average cache read hit 0.000 s 2025-08-14T22:00:52.6073803Z Failed distributed compilations 0 2025-08-14T22:00:52.6073964Z 2025-08-14T22:00:52.6074043Z Non-cacheable reasons: 2025-08-14T22:00:52.6074241Z -E 41 2025-08-14T22:00:52.6074374Z 2025-08-14T22:00:52.6074554Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-08-14T22:00:52.6074869Z Version (client) 0.10.0 2025-08-14T22:00:52.6075135Z + echo ::endgroup:: 2025-08-14T22:00:52.6075562Z ##[endgroup] 2025-08-14T22:00:52.6075728Z + cleanup_workspace 2025-08-14T22:00:52.6076440Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-08-14T22:00:52.6076943Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-08-14T22:00:52.6077351Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-08-14T22:00:52.6077670Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-08-14T22:00:52.6078036Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-08-14T22:00:52.6078429Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-08-14T22:00:52.6078758Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-08-14T22:00:53.0470930Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-08-14T22:00:53.0471633Z with: 2025-08-14T22:00:53.0471864Z benchmark-results-dir: test/test-reports 2025-08-14T22:00:53.0472211Z dry-run: false 2025-08-14T22:00:53.0472417Z schema-version: v3 2025-08-14T22:00:53.0472882Z github-token: *** 2025-08-14T22:00:53.0473245Z env: 2025-08-14T22:00:53.0473437Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:53.0473815Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:53.0474222Z ##[endgroup] 2025-08-14T22:00:53.0490064Z ##[group]Run set -eux 2025-08-14T22:00:53.0490374Z set -eux 2025-08-14T22:00:53.0490644Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-08-14T22:00:53.0490941Z  2025-08-14T22:00:53.0491122Z DEVICE_NAME="" 2025-08-14T22:00:53.0491308Z DEVICE_TYPE="" 2025-08-14T22:00:53.0491484Z  2025-08-14T22:00:53.0491747Z if command -v nvidia-smi; then 2025-08-14T22:00:53.0492059Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-08-14T22:00:53.0492570Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-08-14T22:00:53.0492927Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-08-14T22:00:53.0493220Z  python3 -mpip install torch==2.7.1 2025-08-14T22:00:53.0493463Z elif command -v rocminfo; then 2025-08-14T22:00:53.0493758Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-08-14T22:00:53.0494258Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-08-14T22:00:53.0494626Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-08-14T22:00:53.0494980Z  # GPU device name coming from rocminfo instead 2025-08-14T22:00:53.0495225Z  DEVICE_NAME=rocm 2025-08-14T22:00:53.0495560Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-08-14T22:00:53.0496027Z fi 2025-08-14T22:00:53.0496192Z  2025-08-14T22:00:53.0496388Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-08-14T22:00:53.0496670Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-08-14T22:00:53.0506216Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:53.0506488Z env: 2025-08-14T22:00:53.0506667Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:53.0506984Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:53.0507477Z ##[endgroup] 2025-08-14T22:00:53.0538730Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-08-14T22:00:53.2450960Z Defaulting to user installation because normal site-packages is not writeable 2025-08-14T22:00:54.0422022Z Collecting boto3==1.35.33 2025-08-14T22:00:54.0574923Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2025-08-14T22:00:54.2867118Z Collecting psutil==7.0.0 2025-08-14T22:00:54.2900535Z Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB) 2025-08-14T22:00:54.3142892Z Collecting pynvml==12.0.0 2025-08-14T22:00:54.3177045Z Downloading pynvml-12.0.0-py3-none-any.whl (26 kB) 2025-08-14T22:00:55.1864214Z Collecting botocore<1.36.0,>=1.35.33 2025-08-14T22:00:55.1899055Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-08-14T22:00:55.2881151Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-08-14T22:00:55.3184324Z Collecting s3transfer<0.11.0,>=0.10.0 2025-08-14T22:00:55.3218817Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-08-14T22:00:55.3617891Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0 2025-08-14T22:00:55.3651730Z Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB) 2025-08-14T22:00:55.3731070Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-08-14T22:00:55.3738602Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-08-14T22:00:55.5521995Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-08-14T22:00:55.6637045Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3 2025-08-14T22:00:56.0282594Z Attempting uninstall: nvidia-ml-py 2025-08-14T22:00:56.0283096Z Found existing installation: nvidia-ml-py 11.525.84 2025-08-14T22:00:56.0294125Z Uninstalling nvidia-ml-py-11.525.84: 2025-08-14T22:00:56.0434106Z Successfully uninstalled nvidia-ml-py-11.525.84 2025-08-14T22:00:56.0978876Z Attempting uninstall: psutil 2025-08-14T22:00:56.0979476Z Found existing installation: psutil 5.9.8 2025-08-14T22:00:56.1028384Z Uninstalling psutil-5.9.8: 2025-08-14T22:00:56.1031719Z Successfully uninstalled psutil-5.9.8 2025-08-14T22:00:56.2406655Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4 2025-08-14T22:00:56.3613901Z + DEVICE_NAME= 2025-08-14T22:00:56.3616662Z + DEVICE_TYPE= 2025-08-14T22:00:56.3617276Z + command -v nvidia-smi 2025-08-14T22:00:56.3617600Z + command -v rocminfo 2025-08-14T22:00:56.3617784Z + echo DEVICE_NAME= 2025-08-14T22:00:56.3617992Z + echo DEVICE_TYPE= 2025-08-14T22:00:56.3638272Z ##[group]Run set -eux 2025-08-14T22:00:56.3638462Z set -eux 2025-08-14T22:00:56.3638605Z  2025-08-14T22:00:56.3638772Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-08-14T22:00:56.3639005Z  echo "Missing github-token input" 2025-08-14T22:00:56.3639202Z  exit 1 2025-08-14T22:00:56.3639378Z fi 2025-08-14T22:00:56.3643777Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:56.3644019Z env: 2025-08-14T22:00:56.3644180Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:56.3644463Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:56.3644771Z DEVICE_NAME: 2025-08-14T22:00:56.3644932Z DEVICE_TYPE: 2025-08-14T22:00:56.3645315Z GITHUB_TOKEN: *** 2025-08-14T22:00:56.3645468Z ##[endgroup] 2025-08-14T22:00:56.3666701Z + [[ -z *** ]] 2025-08-14T22:00:56.3696877Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-08-14T22:00:56.3697207Z with: 2025-08-14T22:00:56.3697535Z github-token: *** 2025-08-14T22:00:56.3697750Z env: 2025-08-14T22:00:56.3697935Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:56.3698273Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:56.3698637Z DEVICE_NAME: 2025-08-14T22:00:56.3698825Z DEVICE_TYPE: 2025-08-14T22:00:56.3699019Z ##[endgroup] 2025-08-14T22:00:56.3710617Z ##[group]Run set -eux 2025-08-14T22:00:56.3710851Z set -eux 2025-08-14T22:00:56.3711042Z  2025-08-14T22:00:56.3711603Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-08-14T22:00:56.3716934Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:56.3717250Z env: 2025-08-14T22:00:56.3717437Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:56.3717780Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:56.3718138Z DEVICE_NAME: 2025-08-14T22:00:56.3718338Z DEVICE_TYPE: 2025-08-14T22:00:56.3718676Z GITHUB_TOKEN: *** 2025-08-14T22:00:56.3718873Z ##[endgroup] 2025-08-14T22:00:56.3744269Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 16976338999 i-0aaf71856f9399359 2025-08-14T22:00:57.7993343Z setting job-id=48128301875 2025-08-14T22:00:57.7993976Z setting job-name=linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T22:00:57.8091436Z ##[group]Run set -eux 2025-08-14T22:00:57.8091656Z set -eux 2025-08-14T22:00:57.8091822Z  2025-08-14T22:00:57.8092094Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-08-14T22:00:57.8092427Z  --schema-version "${SCHEMA_VERSION}" \ 2025-08-14T22:00:57.8092665Z  --repo "${REPO}" \ 2025-08-14T22:00:57.8092884Z  --head-branch "${HEAD_BRANCH}" \ 2025-08-14T22:00:57.8093183Z  --head-sha "${HEAD_SHA}" \ 2025-08-14T22:00:57.8093416Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-08-14T22:00:57.8093657Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-08-14T22:00:57.8093881Z  --job-id "${JOB_ID}" \ 2025-08-14T22:00:57.8094097Z  --job-name "${JOB_NAME}" 2025-08-14T22:00:57.8098905Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:57.8099158Z env: 2025-08-14T22:00:57.8099318Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:57.8099628Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:57.8099945Z DEVICE_NAME: 2025-08-14T22:00:57.8100104Z DEVICE_TYPE: 2025-08-14T22:00:57.8100272Z SCHEMA_VERSION: v3 2025-08-14T22:00:57.8100460Z REPO: pytorch/pytorch 2025-08-14T22:00:57.8100647Z HEAD_BRANCH: refs/heads/main 2025-08-14T22:00:57.8100882Z HEAD_SHA: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T22:00:57.8101132Z WORKFLOW_RUN_ID: 16976338999 2025-08-14T22:00:57.8101327Z RUN_ATTEMPT: 1 2025-08-14T22:00:57.8101494Z JOB_ID: 48128301875 2025-08-14T22:00:57.8101921Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T22:00:57.8102346Z ##[endgroup] 2025-08-14T22:00:57.8132790Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 1fc683cf17c8c673044538d10266c00f92987be2 --workflow-id 16976338999 --run-attempt 1 --job-id 48128301875 --job-name 'linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)' 2025-08-14T22:00:57.8407325Z ##[group]Run set -eux 2025-08-14T22:00:57.8407558Z set -eux 2025-08-14T22:00:57.8407705Z  2025-08-14T22:00:57.8407957Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-08-14T22:00:57.8412804Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:57.8413035Z env: 2025-08-14T22:00:57.8413196Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:57.8413493Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:57.8413795Z DEVICE_NAME: 2025-08-14T22:00:57.8413942Z DEVICE_TYPE: 2025-08-14T22:00:57.8414093Z ##[endgroup] 2025-08-14T22:00:57.8436904Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-08-14T22:00:57.8767288Z INFO:root:Fail to import torch to get the device name 2025-08-14T22:00:57.8857827Z ##[group]Run set -eux 2025-08-14T22:00:57.8858029Z set -eux 2025-08-14T22:00:57.8858186Z  2025-08-14T22:00:57.8858369Z # TODO (huydhn): Implement this part 2025-08-14T22:00:57.8858648Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-08-14T22:00:57.8863290Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:57.8863527Z env: 2025-08-14T22:00:57.8863686Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:57.8863976Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:57.8864415Z DEVICE_NAME: 2025-08-14T22:00:57.8864582Z DEVICE_TYPE: 2025-08-14T22:00:57.8864752Z ##[endgroup] 2025-08-14T22:00:57.8886924Z + echo 'dependencies={}' 2025-08-14T22:00:57.8900418Z ##[group]Run set -eux 2025-08-14T22:00:57.8900632Z set -eux 2025-08-14T22:00:57.8900802Z  2025-08-14T22:00:57.8901002Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-08-14T22:00:57.8901308Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-08-14T22:00:57.8901630Z  # We don't want the job to fail if the directory doesn't exist 2025-08-14T22:00:57.8901894Z  exit 0 2025-08-14T22:00:57.8902130Z fi 2025-08-14T22:00:57.8902278Z  2025-08-14T22:00:57.8902457Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-08-14T22:00:57.8902783Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-08-14T22:00:57.8903154Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-08-14T22:00:57.8903507Z  --metadata "${BENCHMARK_METADATA}" \ 2025-08-14T22:00:57.8903753Z  --runners "${RUNNER_INFO}" \ 2025-08-14T22:00:57.8903997Z  --dependencies "${DEPENDENCIES}" \ 2025-08-14T22:00:57.8904221Z  --dry-run 2025-08-14T22:00:57.8904401Z else 2025-08-14T22:00:57.8904662Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-08-14T22:00:57.8905017Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-08-14T22:00:57.8905303Z  --metadata "${BENCHMARK_METADATA}" \ 2025-08-14T22:00:57.8905550Z  --runners "${RUNNER_INFO}" \ 2025-08-14T22:00:57.8905791Z  --dependencies "${DEPENDENCIES}" 2025-08-14T22:00:57.8906006Z fi 2025-08-14T22:00:57.8910763Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:57.8911030Z env: 2025-08-14T22:00:57.8911198Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:57.8911529Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:57.8911863Z DEVICE_NAME: 2025-08-14T22:00:57.8912025Z DEVICE_TYPE: 2025-08-14T22:00:57.8912225Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-08-14T22:00:57.8912450Z DRY_RUN: false 2025-08-14T22:00:57.8913414Z BENCHMARK_METADATA: {"timestamp": 1755208857, "schema_version": "v3", "name": "linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976338999, "run_attempt": 1, "job_id": 48128301875} 2025-08-14T22:00:57.8914598Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-36-175.ec2.internal"}, "name": "", "type": ""}] 2025-08-14T22:00:57.8915014Z DEPENDENCIES: {} 2025-08-14T22:00:57.8915196Z ##[endgroup] 2025-08-14T22:00:57.8936663Z + [[ ! -d test/test-reports ]] 2025-08-14T22:00:57.8939929Z + [[ false == \t\r\u\e ]] 2025-08-14T22:00:57.8946629Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1755208857, "schema_version": "v3", "name": "linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976338999, "run_attempt": 1, "job_id": 48128301875}' --runners '[{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-36-175.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}' 2025-08-14T22:00:58.0151298Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/16976338999/48128301875/inference_huggingface.json 2025-08-14T22:00:58.0462545Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-08-14T22:00:58.3084193Z ##[group]Run cat test/**/*_toprint.log || true 2025-08-14T22:00:58.3084497Z cat test/**/*_toprint.log || true 2025-08-14T22:00:58.3089459Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:58.3089717Z env: 2025-08-14T22:00:58.3089888Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:58.3090188Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:58.3090600Z DEVICE_NAME: 2025-08-14T22:00:58.3090767Z DEVICE_TYPE: 2025-08-14T22:00:58.3090932Z ##[endgroup] 2025-08-14T22:00:58.3166438Z cat: 'test/**/*_toprint.log': No such file or directory 2025-08-14T22:00:58.3210097Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-08-14T22:00:58.3210365Z kill "$MONITOR_SCRIPT_PID" 2025-08-14T22:00:58.3214866Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:58.3215199Z env: 2025-08-14T22:00:58.3215366Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:58.3215669Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:58.3216000Z DEVICE_NAME: 2025-08-14T22:00:58.3216164Z DEVICE_TYPE: 2025-08-14T22:00:58.3216336Z MONITOR_SCRIPT_PID: 48163 2025-08-14T22:00:58.3216517Z ##[endgroup] 2025-08-14T22:00:58.3315479Z Prepare all required actions 2025-08-14T22:00:58.3315998Z Getting action download info 2025-08-14T22:00:58.4694316Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-08-14T22:00:58.6732595Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-08-14T22:00:59.0469318Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-08-14T22:00:59.0469584Z with: 2025-08-14T22:00:59.0469859Z file-suffix: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875 2025-08-14T22:00:59.0470184Z s3-bucket: gha-artifacts 2025-08-14T22:00:59.0470377Z env: 2025-08-14T22:00:59.0470540Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:59.0470883Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:59.0471238Z DEVICE_NAME: 2025-08-14T22:00:59.0471435Z DEVICE_TYPE: 2025-08-14T22:00:59.0471609Z ##[endgroup] 2025-08-14T22:00:59.0489904Z ##[group]Run # Remove any previous test jsons if they exist 2025-08-14T22:00:59.0490240Z # Remove any previous test jsons if they exist 2025-08-14T22:00:59.0490494Z rm -f test-jsons-*.zip 2025-08-14T22:00:59.0490804Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-08-14T22:00:59.0495623Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:59.0495873Z env: 2025-08-14T22:00:59.0496043Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:59.0496354Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:59.0496681Z DEVICE_NAME: 2025-08-14T22:00:59.0496857Z DEVICE_TYPE: 2025-08-14T22:00:59.0497130Z FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875 2025-08-14T22:00:59.0497433Z ##[endgroup] 2025-08-14T22:00:59.0725828Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2025-08-14T22:00:59.0755162Z ##[group]Run # Remove any previous test reports if they exist 2025-08-14T22:00:59.0755502Z # Remove any previous test reports if they exist 2025-08-14T22:00:59.0755768Z rm -f test-reports-*.zip 2025-08-14T22:00:59.0756502Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-08-14T22:00:59.0761289Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:59.0761541Z env: 2025-08-14T22:00:59.0761712Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:59.0762022Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:59.0762347Z DEVICE_NAME: 2025-08-14T22:00:59.0762516Z DEVICE_TYPE: 2025-08-14T22:00:59.0762779Z FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875 2025-08-14T22:00:59.0763080Z ##[endgroup] 2025-08-14T22:00:59.0817191Z adding: test/test-reports/inference_huggingface.csv (deflated 69%) 2025-08-14T22:00:59.0817670Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 85%) 2025-08-14T22:00:59.0818147Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 64%) 2025-08-14T22:00:59.0845288Z ##[group]Run # Remove any previous usage logs if they exist 2025-08-14T22:00:59.0845682Z # Remove any previous usage logs if they exist 2025-08-14T22:00:59.0845929Z rm -f logs-*.zip 2025-08-14T22:00:59.0846166Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-08-14T22:00:59.0846477Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-08-14T22:00:59.0850771Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:59.0851009Z env: 2025-08-14T22:00:59.0851163Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:59.0851455Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:59.0851753Z DEVICE_NAME: 2025-08-14T22:00:59.0851907Z DEVICE_TYPE: 2025-08-14T22:00:59.0852290Z FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875 2025-08-14T22:00:59.0852576Z ##[endgroup] 2025-08-14T22:00:59.0923192Z adding: usage_log.txt (deflated 96%) 2025-08-14T22:00:59.0933166Z 2025-08-14T22:00:59.0933613Z zip error: Nothing to do! (logs-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875.zip) 2025-08-14T22:00:59.0953672Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-08-14T22:00:59.0954038Z # Remove any previous debugging artifacts if they exist 2025-08-14T22:00:59.0954326Z rm -f debug-*.zip 2025-08-14T22:00:59.0954539Z if [ -d 'test/debug' ]; then 2025-08-14T22:00:59.0954799Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-08-14T22:00:59.0955037Z fi 2025-08-14T22:00:59.0959804Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:00:59.0960037Z env: 2025-08-14T22:00:59.0960194Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:59.0960501Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:59.0960798Z DEVICE_NAME: 2025-08-14T22:00:59.0960960Z DEVICE_TYPE: 2025-08-14T22:00:59.0961222Z FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875 2025-08-14T22:00:59.0961497Z ##[endgroup] 2025-08-14T22:00:59.1026050Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-14T22:00:59.1026288Z with: 2025-08-14T22:00:59.1026461Z s3-bucket: gha-artifacts 2025-08-14T22:00:59.1026693Z s3-prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T22:00:59.1026922Z retention-days: 14 2025-08-14T22:00:59.1027117Z if-no-files-found: warn 2025-08-14T22:00:59.1027311Z path: test-jsons-*.zip 2025-08-14T22:00:59.1027486Z name: artifact 2025-08-14T22:00:59.1027653Z region: us-east-1 2025-08-14T22:00:59.1027818Z env: 2025-08-14T22:00:59.1027971Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:59.1028282Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:59.1028653Z DEVICE_NAME: 2025-08-14T22:00:59.1028818Z DEVICE_TYPE: 2025-08-14T22:00:59.1028985Z ##[endgroup] 2025-08-14T22:00:59.4055964Z NOTE: s3-prefix specified, ignoring name parameter 2025-08-14T22:00:59.4056578Z With the provided path, there will be 1 file uploaded 2025-08-14T22:00:59.4059872Z Uploading to s3 prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T22:00:59.4087497Z Starting upload of test-jsons-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875.zip 2025-08-14T22:00:59.5230450Z Finished upload of test-jsons-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875.zip 2025-08-14T22:00:59.5398204Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-14T22:00:59.5398466Z with: 2025-08-14T22:00:59.5398644Z s3-bucket: gha-artifacts 2025-08-14T22:00:59.5398893Z s3-prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T22:00:59.5399148Z retention-days: 14 2025-08-14T22:00:59.5399332Z if-no-files-found: error 2025-08-14T22:00:59.5399541Z path: test-reports-*.zip 2025-08-14T22:00:59.5399750Z name: artifact 2025-08-14T22:00:59.5399920Z region: us-east-1 2025-08-14T22:00:59.5400093Z env: 2025-08-14T22:00:59.5400258Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:59.5400573Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:59.5400990Z DEVICE_NAME: 2025-08-14T22:00:59.5401163Z DEVICE_TYPE: 2025-08-14T22:00:59.5401336Z ##[endgroup] 2025-08-14T22:00:59.8086752Z NOTE: s3-prefix specified, ignoring name parameter 2025-08-14T22:00:59.8087104Z With the provided path, there will be 1 file uploaded 2025-08-14T22:00:59.8087732Z Uploading to s3 prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T22:00:59.8124435Z Starting upload of test-reports-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875.zip 2025-08-14T22:00:59.9380347Z Finished upload of test-reports-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875.zip 2025-08-14T22:00:59.9531772Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-14T22:00:59.9532165Z with: 2025-08-14T22:00:59.9532344Z s3-bucket: gha-artifacts 2025-08-14T22:00:59.9532580Z s3-prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T22:00:59.9532817Z retention-days: 14 2025-08-14T22:00:59.9533013Z if-no-files-found: ignore 2025-08-14T22:00:59.9533220Z path: logs-*.zip 2025-08-14T22:00:59.9533382Z name: artifact 2025-08-14T22:00:59.9533552Z region: us-east-1 2025-08-14T22:00:59.9533714Z env: 2025-08-14T22:00:59.9533860Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:00:59.9534167Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:00:59.9534484Z DEVICE_NAME: 2025-08-14T22:00:59.9534650Z DEVICE_TYPE: 2025-08-14T22:00:59.9534804Z ##[endgroup] 2025-08-14T22:01:00.2220862Z NOTE: s3-prefix specified, ignoring name parameter 2025-08-14T22:01:00.2221217Z With the provided path, there will be 1 file uploaded 2025-08-14T22:01:00.2221518Z Uploading to s3 prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T22:01:00.2257812Z Starting upload of logs-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875.zip 2025-08-14T22:01:00.4176475Z Finished upload of logs-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875.zip 2025-08-14T22:01:00.4371350Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-14T22:01:00.4371606Z with: 2025-08-14T22:01:00.4371774Z s3-bucket: gha-artifacts 2025-08-14T22:01:00.4371998Z s3-prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T22:01:00.4372218Z retention-days: 14 2025-08-14T22:01:00.4372393Z if-no-files-found: ignore 2025-08-14T22:01:00.4372590Z path: debug-*.zip 2025-08-14T22:01:00.4372742Z name: artifact 2025-08-14T22:01:00.4372900Z region: us-east-1 2025-08-14T22:01:00.4373053Z env: 2025-08-14T22:01:00.4373194Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:01:00.4373492Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:00.4373803Z DEVICE_NAME: 2025-08-14T22:01:00.4373963Z DEVICE_TYPE: 2025-08-14T22:01:00.4374165Z ##[endgroup] 2025-08-14T22:01:00.6893106Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-08-14T22:01:00.7049875Z ##[group]Run # shellcheck disable=SC2156 2025-08-14T22:01:00.7050170Z # shellcheck disable=SC2156 2025-08-14T22:01:00.7050556Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-08-14T22:01:00.7056068Z shell: /usr/bin/bash -e {0} 2025-08-14T22:01:00.7056267Z env: 2025-08-14T22:01:00.7056426Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:01:00.7056747Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:00.7057089Z DEVICE_NAME: 2025-08-14T22:01:00.7057257Z DEVICE_TYPE: 2025-08-14T22:01:00.7057423Z ##[endgroup] 2025-08-14T22:01:00.8883573Z Prepare all required actions 2025-08-14T22:01:00.8883936Z Getting action download info 2025-08-14T22:01:01.0028778Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-08-14T22:01:01.0029088Z with: 2025-08-14T22:01:01.0029281Z job_id: 48128301875 2025-08-14T22:01:01.0029763Z job_name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T22:01:01.0030372Z workflow_name: inductor-periodic 2025-08-14T22:01:01.0030624Z workflow_run_id: 16976338999 2025-08-14T22:01:01.0030843Z workflow_attempt: 1 2025-08-14T22:01:01.0031041Z env: 2025-08-14T22:01:01.0031228Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:01:01.0031585Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:01.0032050Z DEVICE_NAME: 2025-08-14T22:01:01.0032250Z DEVICE_TYPE: 2025-08-14T22:01:01.0032441Z ##[endgroup] 2025-08-14T22:01:01.0046316Z ##[group]Run echo "workflow_id: 16976338999" 2025-08-14T22:01:01.0046591Z echo "workflow_id: 16976338999" 2025-08-14T22:01:01.0046837Z echo "workflow_attempt: 1" 2025-08-14T22:01:01.0047100Z echo "workflow_Name: inductor-periodic" 2025-08-14T22:01:01.0047349Z echo "job_id: 48128301875" 2025-08-14T22:01:01.0047814Z echo "job_name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" 2025-08-14T22:01:01.0048291Z echo "artifact_prefix: " 2025-08-14T22:01:01.0048518Z python3 --version 2025-08-14T22:01:01.0053301Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:01:01.0053593Z env: 2025-08-14T22:01:01.0053765Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:01:01.0054086Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:01.0054417Z DEVICE_NAME: 2025-08-14T22:01:01.0054593Z DEVICE_TYPE: 2025-08-14T22:01:01.0054765Z ##[endgroup] 2025-08-14T22:01:01.0076524Z workflow_id: 16976338999 2025-08-14T22:01:01.0076780Z workflow_attempt: 1 2025-08-14T22:01:01.0077005Z workflow_Name: inductor-periodic 2025-08-14T22:01:01.0077216Z job_id: 48128301875 2025-08-14T22:01:01.0077673Z job_name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T22:01:01.0078169Z artifact_prefix: 2025-08-14T22:01:01.0090562Z Python 3.9.23 2025-08-14T22:01:01.0121180Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-14T22:01:01.0121412Z with: 2025-08-14T22:01:01.0121584Z shell: bash 2025-08-14T22:01:01.0121763Z timeout_minutes: 5 2025-08-14T22:01:01.0121944Z max_attempts: 5 2025-08-14T22:01:01.0122128Z retry_wait_seconds: 30 2025-08-14T22:01:01.0122527Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 dataclasses_json==0.6.7 2025-08-14T22:01:01.0122934Z polling_interval_seconds: 1 2025-08-14T22:01:01.0123140Z warning_on_retry: true 2025-08-14T22:01:01.0123351Z continue_on_error: false 2025-08-14T22:01:01.0123545Z env: 2025-08-14T22:01:01.0123702Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:01:01.0124028Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:01.0124367Z DEVICE_NAME: 2025-08-14T22:01:01.0124535Z DEVICE_TYPE: 2025-08-14T22:01:01.0124708Z ##[endgroup] 2025-08-14T22:01:01.2950662Z Defaulting to user installation because normal site-packages is not writeable 2025-08-14T22:01:01.3587602Z Collecting python-dateutil==2.8.2 2025-08-14T22:01:01.3730838Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2025-08-14T22:01:02.1080717Z Collecting boto3==1.35.42 2025-08-14T22:01:02.1136331Z Downloading boto3-1.35.42-py3-none-any.whl (139 kB) 2025-08-14T22:01:02.5025513Z Collecting pandas==2.1.3 2025-08-14T22:01:02.5066549Z Downloading pandas-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB) 2025-08-14T22:01:02.6399315Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-08-14T22:01:02.6409766Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0) 2025-08-14T22:01:02.6446771Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-08-14T22:01:02.6454361Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-08-14T22:01:02.6455000Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-08-14T22:01:02.6940423Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1) 2025-08-14T22:01:02.7208516Z Collecting tzdata>=2022.1 2025-08-14T22:01:02.7246142Z Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB) 2025-08-14T22:01:03.3444897Z Collecting numpy<2,>=1.22.4 2025-08-14T22:01:03.3482037Z Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB) 2025-08-14T22:01:03.4847291Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-08-14T22:01:03.4850326Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-08-14T22:01:03.4893720Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-08-14T22:01:03.4985033Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0) 2025-08-14T22:01:03.5057389Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.14.1) 2025-08-14T22:01:03.5063661Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.1.0) 2025-08-14T22:01:03.6290107Z Installing collected packages: python-dateutil, tzdata, numpy, pandas, boto3 2025-08-14T22:01:07.7336388Z Attempting uninstall: boto3 2025-08-14T22:01:07.7336722Z Found existing installation: boto3 1.35.33 2025-08-14T22:01:07.7405028Z Uninstalling boto3-1.35.33: 2025-08-14T22:01:07.7413759Z Successfully uninstalled boto3-1.35.33 2025-08-14T22:01:07.7855943Z Successfully installed boto3-1.35.42 numpy-1.26.4 pandas-2.1.3 python-dateutil-2.8.2 tzdata-2025.2 2025-08-14T22:01:08.0854130Z Command completed after 1 attempt(s). 2025-08-14T22:01:08.0907662Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-08-14T22:01:08.0908115Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-08-14T22:01:08.0908431Z  --workflow-run-id "16976338999" \ 2025-08-14T22:01:08.0908918Z  --workflow-name "inductor-periodic" \ 2025-08-14T22:01:08.0909170Z  --workflow-run-attempt "1" \ 2025-08-14T22:01:08.0909384Z  --job-id "48128301875" \ 2025-08-14T22:01:08.0909815Z  --job-name "linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" \ 2025-08-14T22:01:08.0910264Z  --local-path "" \ 2025-08-14T22:01:08.0910467Z  --artifact-prefix "" 2025-08-14T22:01:08.0916363Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:01:08.0916633Z env: 2025-08-14T22:01:08.0916811Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:01:08.0917124Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:08.0917433Z DEVICE_NAME: 2025-08-14T22:01:08.0917591Z DEVICE_TYPE: 2025-08-14T22:01:08.0917875Z ##[endgroup] 2025-08-14T22:01:08.9635468Z repo: pytorch/pytorch 2025-08-14T22:01:08.9635876Z Search for test log in s3 bucket: ossci-utilization 2025-08-14T22:01:08.9636289Z Downloading logs-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875.zip 2025-08-14T22:01:08.9636814Z extracting usage_log.txt from zip file logs-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301875.zip 2025-08-14T22:01:08.9637629Z Converted Log Model: UtilizationMetadata: 2025-08-14T22:01:08.9638695Z UtilizationMetadata(level='metadata', workflow_id='16976338999', job_id='48128301875', workflow_name='inductor-periodic', job_name='linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)', usage_collect_interval=1.0, data_model_version=1.5, start_at=1755207365, gpu_count=0, cpu_count=32, gpu_type=None, error=None) 2025-08-14T22:01:08.9639833Z [Db Segments] detected pytest cmd: 10, generated segments: 10 2025-08-14T22:01:08.9640142Z [db model] Peek db timeseries 2025-08-14T22:01:08.9640361Z :{ 2025-08-14T22:01:08.9640541Z "created_at": 1755208868, 2025-08-14T22:01:08.9640770Z "type": "utilization", 2025-08-14T22:01:08.9640965Z "tags": [ 2025-08-14T22:01:08.9641135Z "record" 2025-08-14T22:01:08.9641336Z ], 2025-08-14T22:01:08.9641497Z "time_stamp": 1755207365, 2025-08-14T22:01:08.9641692Z "repo": "pytorch/pytorch", 2025-08-14T22:01:08.9641885Z "workflow_id": 16976338999, 2025-08-14T22:01:08.9642067Z "run_attempt": 1, 2025-08-14T22:01:08.9642238Z "job_id": 48128301875, 2025-08-14T22:01:08.9642463Z "workflow_name": "inductor-periodic", 2025-08-14T22:01:08.9642962Z "job_name": "linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", 2025-08-14T22:01:08.9643383Z "json_data": "{}" 2025-08-14T22:01:08.9643546Z } 2025-08-14T22:01:08.9643906Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/16976338999/1/48128301875/metadata 2025-08-14T22:01:08.9644485Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/16976338999/1/48128301875/metadata 2025-08-14T22:01:08.9645063Z Writing 292 documents to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/16976338999/1/48128301875/time_series 2025-08-14T22:01:08.9645652Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/16976338999/1/48128301875/time_series 2025-08-14T22:01:09.0701153Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-08-14T22:01:09.0701520Z with: 2025-08-14T22:01:09.0701703Z env: 2025-08-14T22:01:09.0701890Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:01:09.0702273Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:09.0702680Z DEVICE_NAME: 2025-08-14T22:01:09.0702865Z DEVICE_TYPE: 2025-08-14T22:01:09.0703052Z ##[endgroup] 2025-08-14T22:01:09.0716557Z ##[group]Run set -eou pipefail 2025-08-14T22:01:09.0716854Z set -eou pipefail 2025-08-14T22:01:09.0717082Z  2025-08-14T22:01:09.0717385Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-08-14T22:01:09.0717756Z for _ in $(seq 1440); do 2025-08-14T22:01:09.0718045Z  # Break if no ssh session exists anymore 2025-08-14T22:01:09.0718336Z  if [ "$(who)" = "" ]; then 2025-08-14T22:01:09.0718565Z  break 2025-08-14T22:01:09.0718786Z  fi 2025-08-14T22:01:09.0718961Z  echo "." 2025-08-14T22:01:09.0719153Z  sleep 5 2025-08-14T22:01:09.0719336Z done 2025-08-14T22:01:09.0725061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:01:09.0725371Z env: 2025-08-14T22:01:09.0725563Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:01:09.0725925Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:09.0726301Z DEVICE_NAME: 2025-08-14T22:01:09.0726489Z DEVICE_TYPE: 2025-08-14T22:01:09.0726674Z ##[endgroup] 2025-08-14T22:01:09.0751280Z Holding runner for 2 hours until all ssh sessions have logged out 2025-08-14T22:01:09.0828389Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T22:01:09.0828922Z # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T22:01:09.0829266Z # shellcheck disable=SC2046 2025-08-14T22:01:09.0829500Z docker stop $(docker ps -q) || true 2025-08-14T22:01:09.0829714Z # Prune all of the docker images 2025-08-14T22:01:09.0829923Z docker system prune -af 2025-08-14T22:01:09.0834382Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:01:09.0834700Z env: 2025-08-14T22:01:09.0834871Z GIT_DEFAULT_BRANCH: main 2025-08-14T22:01:09.0835184Z DOCKER_CONTAINER_ID: ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:09.0835500Z DEVICE_NAME: 2025-08-14T22:01:09.0835672Z DEVICE_TYPE: 2025-08-14T22:01:09.0836134Z ##[endgroup] 2025-08-14T22:01:19.9157819Z ec43c4531511 2025-08-14T22:01:20.2097459Z Deleted Containers: 2025-08-14T22:01:20.2099456Z ec43c4531511a6b79232130fd215393146214a245a89775adb2a5fab27fa62bb 2025-08-14T22:01:20.2099874Z 2025-08-14T22:01:27.4098340Z Deleted Images: 2025-08-14T22:01:27.4099208Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T22:01:27.4100097Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:4236794baba289041d240d08fd393bbd57497c3012e5e0ccd9fd98f61ebf35c6 2025-08-14T22:01:27.4100672Z deleted: sha256:0899ae453036ee7a91795ea95b1db61000579eeb74b140edab5976919ee64bbe 2025-08-14T22:01:27.4101101Z deleted: sha256:aa7b544271e9ba3105dabd1afb12e315887018f3471e03135c1d50e64cc550c4 2025-08-14T22:01:27.4101520Z deleted: sha256:4c685831817cc2fc6dfdfda1726df1f402222d8cdccc40daad3198cf8b17e3f4 2025-08-14T22:01:27.4101950Z deleted: sha256:cedf3fb09a62e68c6d7e22cedbce12e77166a50649d0269200ee0efce8a57b88 2025-08-14T22:01:27.4102373Z deleted: sha256:1b3a9a237b4153f8f523a85cead9d36e29717eb57182e2f75069788681627d95 2025-08-14T22:01:27.4102791Z deleted: sha256:67bd313103dfbe7fe0172e6f4f7ee420fad9743a64a1cc1cd20bc22250d3602c 2025-08-14T22:01:27.4103173Z deleted: sha256:b17820137ada46a2a726c67aa08cce73d2ead7c95db08575cf5e69bedb4b600d 2025-08-14T22:01:27.4103901Z deleted: sha256:b16c9bc40cc1cf924638323aece4168d6332cfae212dad2a431a584a44fe967c 2025-08-14T22:01:27.4104307Z deleted: sha256:ab35ed781133eb4aaa1b2478aea73fb80dc71bceffbe474b55e1a60fc6c5ffbe 2025-08-14T22:01:27.4104721Z deleted: sha256:b9d0b0720dd9c0bcb4f174ae6770a7c2fe540c6983872180f3a5e18300434cdb 2025-08-14T22:01:27.4105145Z deleted: sha256:f5d1a4f32d90030cc174d73b579758d28f95c992a8cf21360e5addee99dea169 2025-08-14T22:01:27.4105559Z deleted: sha256:4af408141f8591f4b69cef9b425b6caa3c4cbc62ced38b5d08f3150f0c8ff449 2025-08-14T22:01:27.4105965Z deleted: sha256:e0019e5c461051e54a9af37ae22b49cfd2c2e5366da57a20304f6ef89171a3b3 2025-08-14T22:01:27.4106333Z deleted: sha256:542f999b2cfc965b97861645356840864e9946fa2fa40f1f5c4c45684e91c239 2025-08-14T22:01:27.4106709Z deleted: sha256:633629aa3d4ae6472e222a1c0b2ceb729b0d84ccb48e12d52ba2d2987c9063e1 2025-08-14T22:01:27.4107096Z deleted: sha256:ea645aba1ba54baac43713f3df7f1b89dd119764a747273897eb2931fea42856 2025-08-14T22:01:27.4107481Z deleted: sha256:1f50e367efff88c7182b9dc3ff618c1cf7bd34edf2f31805e268c50fac02a627 2025-08-14T22:01:27.4107908Z deleted: sha256:aff22d7ae43d842befa617e2e5f9878d09a82b67c362b0c44a40a4c88be92120 2025-08-14T22:01:27.4108361Z deleted: sha256:4275d4addb77b473ed40194e42918cf2aeb484d1d8e25cf54d374392643a095c 2025-08-14T22:01:27.4108959Z deleted: sha256:66471f6c8dc869455ff193909110d824b5d65f7383877a7d0face6331b21fff3 2025-08-14T22:01:27.4109375Z deleted: sha256:8cfd2d55570494ff2b993725f5eb13d0440a5698fa905823ca1677d2d16febb8 2025-08-14T22:01:27.4109782Z deleted: sha256:5c8cf8b9c4a76f679994decc8800bc6eefd258a8dc6293a714d5e100fea3a1bc 2025-08-14T22:01:27.4110201Z deleted: sha256:1acc162c6b9de62d13ce7fd33bb9b134458f7e7dbe996e5442e0047ec8f70c80 2025-08-14T22:01:27.4110660Z deleted: sha256:044bab98f3bceb1948c626ce6bdd19d3ec8f9c5ad42a4f635dd685a7ae9c9024 2025-08-14T22:01:27.4111079Z deleted: sha256:2acb11a9448f13c2c2d29c4d0d4013e046862bd019cf5ec9fe04bdf35299f1dd 2025-08-14T22:01:27.4111492Z deleted: sha256:8e7b56334416233f301944000dec16952e13bb69296cc80e1031bfecaf6e7f9d 2025-08-14T22:01:27.4111986Z deleted: sha256:4a4d1ec727c43389a601aefccdaeff6b3bf54c0daefb12e0c2098c3e18b383ba 2025-08-14T22:01:27.4112399Z deleted: sha256:8b9ca4276331196a2f03c2fa3a87422d2042cf06011b49368c2335be7da829c1 2025-08-14T22:01:27.4112803Z deleted: sha256:5076357fd3cc8b06ed54a0f692362a38f1ebafa4843c0b0bf8021f9021d2e583 2025-08-14T22:01:27.4113269Z deleted: sha256:f9451fa0842798e2a67c059fda5124cafb401801bb8c40d03ae736ff3ef5ed20 2025-08-14T22:01:27.4113673Z deleted: sha256:52b716f02091d6af6b79e7b2e1f5bbd7391235993d415c7a852d6752220c8b65 2025-08-14T22:01:27.4114066Z deleted: sha256:748225161c361d3779c96eb7ae5ea0c33d35311f9445c371d62616b98e3426e8 2025-08-14T22:01:27.4114474Z deleted: sha256:5eeda1478a46d8d58267e8917422eb0a182a40c8bdfb4bfe0869923f8114c770 2025-08-14T22:01:27.4114889Z deleted: sha256:66d4cebb04304f556dd191b425a876f7dbbcde8c3c647af4ef47c10804e51f5a 2025-08-14T22:01:27.4115291Z deleted: sha256:0b526447174d22890be2bc866228e40989483b1102a0430b4ab3ad16dc6c7787 2025-08-14T22:01:27.4115700Z deleted: sha256:1aa31d55f8f9bb51f1eb702ba7d46ceda8290ed90e8e8cf299bb8a9179bf2ae2 2025-08-14T22:01:27.4116374Z deleted: sha256:dd1f47c8dc7518f303a91fc8aae81a512caff53987d5a89a378bb24c1c6d7707 2025-08-14T22:01:27.4116793Z deleted: sha256:d60f9527fcb284e73795a37d4f536badd451a2eade4c9314ebe549d31efcc876 2025-08-14T22:01:27.4117192Z deleted: sha256:f23ad0355704751b0f71a8900169354e3bf23a7b3f5fa2cd9b2478a561bfbb45 2025-08-14T22:01:27.4117615Z deleted: sha256:10e7acf6460743fcad0c1fff0bbd01158fbeb88151621c1e15ae5994f1c8ef55 2025-08-14T22:01:27.4118021Z deleted: sha256:f674e3067e97f1407f4cd55202d4c0c8641f02811550e65a00a875fc19354b75 2025-08-14T22:01:27.4118423Z deleted: sha256:8a9c75c896425ccd25101f0cf39316bec7779111954f44df726842bf583e907b 2025-08-14T22:01:27.4118822Z deleted: sha256:9730d30edfcaa135287479d80f1720b39c6f728228df6d0eb7f095e917cc16b6 2025-08-14T22:01:27.4119232Z deleted: sha256:2787e13cf97e870ca65312526c3000163ebf3da20fe59e5f5d53b1aeb4fb424b 2025-08-14T22:01:27.4119728Z deleted: sha256:d61197909174795bd69f8d5f534f1b086065d36b7aa6c5a50744eca6f8d6b12b 2025-08-14T22:01:27.4120144Z deleted: sha256:ecdfbb81e95b2ae2c8e9ab4ca72ba8564095caabb0512a47da8f866923f71bff 2025-08-14T22:01:27.4120556Z deleted: sha256:cd2d7c644df243742a0c0349af0d37570c06fdd1711ddc367e79514757a6d5cc 2025-08-14T22:01:27.4120973Z deleted: sha256:6703ab1ced70b30a87660c0dd778fe95fb90b04ed8461c2a331272aa54eb3499 2025-08-14T22:01:27.4121389Z deleted: sha256:b7088ce49d7df1d6fb18eee5fc5664e637c5649c89e581d972c76a83f60d0a62 2025-08-14T22:01:27.4121837Z deleted: sha256:d0d2786658af9907d8c4ecfa84fa9e2bd07131257264395b804deef744a5c39c 2025-08-14T22:01:27.4122219Z deleted: sha256:d46baf72d8e570e6004c6f95131cea6ede27eb01c213d8c1e8b263ab95fdfe95 2025-08-14T22:01:27.4122603Z deleted: sha256:0219ea0bd0e38d169ed596ed80807b0f70b609ec5f886d671c249d10575dff2c 2025-08-14T22:01:27.4123040Z deleted: sha256:77d1a1f15cf8ae85a4c5495d800378c307967004360814810fd13b07a74aee5e 2025-08-14T22:01:27.4123397Z deleted: sha256:47c77d89ce8782a94a6f5435b1611a76b47f830153ba4b462d3e08dcbdaa40f7 2025-08-14T22:01:27.4123770Z deleted: sha256:d5120b2e61fb0ccc32a2ad02fc0b2b908bc69f1f174268bde3d26d79ce46f046 2025-08-14T22:01:27.4124141Z deleted: sha256:65626052fd7e03a8e90c72072a54f0eaa43788cfcb0835ffb98b700be89b0567 2025-08-14T22:01:27.4124500Z deleted: sha256:05c09c0832c35f0128e0258b1d3069d7bb4b94ce58239faba5d585e49c34e904 2025-08-14T22:01:27.4124872Z deleted: sha256:2d6749fb2c30585eebb1d97e99318434ec34e0f7a4414e552fd4a44175f86839 2025-08-14T22:01:27.4125242Z deleted: sha256:2d65e2932810021e5b3cfedd89cfd851dd47fce63fbe5dc6959e59f3d8a98499 2025-08-14T22:01:27.4125626Z deleted: sha256:b2e71ddacad35b6caa3a77429bab51b654f6acaccc9e9263f1cb43edb8c53ac3 2025-08-14T22:01:27.4126001Z deleted: sha256:632a43100a629c40972b4da95fbbb581f29fe8b073a96386c72931d27ffbbefa 2025-08-14T22:01:27.4126377Z deleted: sha256:11964e5f5833fdf2bcc61c52f33d5aebf9b5504c6792baf58beb96b90398d10a 2025-08-14T22:01:27.4126761Z deleted: sha256:f0c1cb4c9e4655464b9b62b6589ac5005c2392213765ab4175bd61e3f6462643 2025-08-14T22:01:27.4127159Z deleted: sha256:5113aaee4b4d5ee45b58bcee467ac314112b02e4c4e5e9c3cc7a236dd308e9de 2025-08-14T22:01:27.4127528Z deleted: sha256:9cdc88c7b7fe728e15c72d0e8eef813ace31905b4b317a0a23f1334b6a22e604 2025-08-14T22:01:27.4127891Z deleted: sha256:8056a3da01752a91095e2d0afd80b625172f0915f22f7d998b9b926b9462dc5f 2025-08-14T22:01:27.4128269Z deleted: sha256:8a99968112e0edd39c242f3452b05d167911724468fdd9b18d11a8f5fa9c3ac8 2025-08-14T22:01:27.4128631Z deleted: sha256:6f70653bcfea9c1dd39aba76713adac0ac8f6f4c202387ff86a3ffe45d2079f2 2025-08-14T22:01:27.4129004Z deleted: sha256:9a0ed45f26188ecbfcf7658f46e29922b441969b2aded64d1d6b287b6de2e49c 2025-08-14T22:01:27.4129363Z deleted: sha256:f84c75780b110e68f7593fe9592456387118761b365a954a105aee72016adeac 2025-08-14T22:01:27.4129725Z deleted: sha256:1a5a81f8cbb945eee96e25ee8b4958d7140bb6751b86bc2e4a6aa9e18a16846c 2025-08-14T22:01:27.4130089Z deleted: sha256:7e072dc6aa8c1831ddc97ba8229235081976cb8036c06ee1320b33606e03f9a4 2025-08-14T22:01:27.4130447Z deleted: sha256:369af3627df8ecb48c51ea4fd3267e561b2f6821075ddce314e9485494447f16 2025-08-14T22:01:27.4130804Z deleted: sha256:4d49b99f2eee0f82788e33a9c771f75b1411b0b70ce47771fc1b3bc160f23961 2025-08-14T22:01:27.4131162Z deleted: sha256:fe04dcb9c711f36f9ed1df5b2d0854d30dc5abaa6e6cd493b85d4c2e2d2c3e1b 2025-08-14T22:01:27.4131530Z deleted: sha256:4800771a0435c52d6e480540ffa8a65ecc51fdc82a91302c1a373e6021bc37ca 2025-08-14T22:01:27.4131886Z deleted: sha256:90a2bf02e851326fc70d05470553ed33e578342d6e06bfa0cfaf331c4079b7e4 2025-08-14T22:01:27.4132100Z 2025-08-14T22:01:27.4132197Z Total reclaimed space: 51.8GB 2025-08-14T22:01:27.4212599Z Post job cleanup. 2025-08-14T22:01:27.4241938Z Post job cleanup. 2025-08-14T22:01:27.5018321Z [command]/usr/bin/git version 2025-08-14T22:01:27.5052815Z git version 2.47.1 2025-08-14T22:01:27.5087197Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/9dfe9feb-2087-462a-aeb5-3e0c0fda9def/.gitconfig' 2025-08-14T22:01:27.5099959Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/9dfe9feb-2087-462a-aeb5-3e0c0fda9def' before making global git config changes 2025-08-14T22:01:27.5100594Z Adding repository directory to the temporary git global config as a safe directory 2025-08-14T22:01:27.5104737Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T22:01:27.5165169Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-08-14T22:01:27.5192341Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-08-14T22:01:27.5512213Z Entering 'android/libs/fbjni' 2025-08-14T22:01:27.5566652Z Entering 'third_party/FP16' 2025-08-14T22:01:27.5625988Z Entering 'third_party/FXdiv' 2025-08-14T22:01:27.5675180Z Entering 'third_party/NNPACK' 2025-08-14T22:01:27.5732036Z Entering 'third_party/NVTX' 2025-08-14T22:01:27.5782635Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T22:01:27.5836354Z Entering 'third_party/XNNPACK' 2025-08-14T22:01:27.5907016Z Entering 'third_party/aiter' 2025-08-14T22:01:27.5956382Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T22:01:27.6019890Z Entering 'third_party/benchmark' 2025-08-14T22:01:27.6082664Z Entering 'third_party/composable_kernel' 2025-08-14T22:01:27.6139106Z Entering 'third_party/cpp-httplib' 2025-08-14T22:01:27.6192716Z Entering 'third_party/cpuinfo' 2025-08-14T22:01:27.6252330Z Entering 'third_party/cudnn_frontend' 2025-08-14T22:01:27.6304355Z Entering 'third_party/cutlass' 2025-08-14T22:01:27.6368613Z Entering 'third_party/fbgemm' 2025-08-14T22:01:27.6423951Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T22:01:27.6480372Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T22:01:27.6541017Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T22:01:27.6595651Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T22:01:27.6659897Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T22:01:27.6714474Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T22:01:27.6777996Z Entering 'third_party/fbgemm/external/json' 2025-08-14T22:01:27.6840575Z Entering 'third_party/flash-attention' 2025-08-14T22:01:27.6892174Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T22:01:27.6946065Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T22:01:27.7006137Z Entering 'third_party/flatbuffers' 2025-08-14T22:01:27.7065046Z Entering 'third_party/fmt' 2025-08-14T22:01:27.7118787Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T22:01:27.7172667Z Entering 'third_party/gloo' 2025-08-14T22:01:27.7229449Z Entering 'third_party/googletest' 2025-08-14T22:01:27.7285575Z Entering 'third_party/ideep' 2025-08-14T22:01:27.7339115Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T22:01:27.7400243Z Entering 'third_party/ittapi' 2025-08-14T22:01:27.7453657Z Entering 'third_party/kineto' 2025-08-14T22:01:27.7504083Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T22:01:27.7566959Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T22:01:27.7621975Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T22:01:27.7679031Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T22:01:27.7733666Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T22:01:27.7788293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T22:01:27.7845697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T22:01:27.7897431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T22:01:27.7953126Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T22:01:27.8014123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T22:01:27.8068538Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T22:01:27.8119951Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T22:01:27.8175883Z Entering 'third_party/kleidiai' 2025-08-14T22:01:27.8229208Z Entering 'third_party/mimalloc' 2025-08-14T22:01:27.8280335Z Entering 'third_party/nlohmann' 2025-08-14T22:01:27.8339519Z Entering 'third_party/onnx' 2025-08-14T22:01:27.8403561Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T22:01:27.8465391Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T22:01:27.8514857Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T22:01:27.8568642Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T22:01:27.8629389Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T22:01:27.8673940Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T22:01:27.8730119Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T22:01:27.8784217Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T22:01:27.8835767Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T22:01:27.8888982Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T22:01:27.8949970Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T22:01:27.9004887Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T22:01:27.9076457Z Entering 'third_party/pocketfft' 2025-08-14T22:01:27.9133696Z Entering 'third_party/protobuf' 2025-08-14T22:01:27.9187455Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T22:01:27.9247074Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T22:01:27.9303206Z Entering 'third_party/psimd' 2025-08-14T22:01:27.9367840Z Entering 'third_party/pthreadpool' 2025-08-14T22:01:27.9421419Z Entering 'third_party/pybind11' 2025-08-14T22:01:27.9480897Z Entering 'third_party/python-peachpy' 2025-08-14T22:01:27.9541410Z Entering 'third_party/sleef' 2025-08-14T22:01:27.9590851Z Entering 'third_party/tensorpipe' 2025-08-14T22:01:27.9646562Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T22:01:27.9697060Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T22:01:27.9752012Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T22:01:27.9804014Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T22:01:27.9856932Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T22:01:27.9933567Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-08-14T22:01:27.9953435Z http.https://github.com/.extraheader 2025-08-14T22:01:27.9967992Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-08-14T22:01:27.9997517Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-08-14T22:01:28.0309630Z Entering 'android/libs/fbjni' 2025-08-14T22:01:28.0347394Z http.https://github.com/.extraheader 2025-08-14T22:01:28.0385292Z Entering 'third_party/FP16' 2025-08-14T22:01:28.0420529Z http.https://github.com/.extraheader 2025-08-14T22:01:28.0458082Z Entering 'third_party/FXdiv' 2025-08-14T22:01:28.0490204Z http.https://github.com/.extraheader 2025-08-14T22:01:28.0524535Z Entering 'third_party/NNPACK' 2025-08-14T22:01:28.0562229Z http.https://github.com/.extraheader 2025-08-14T22:01:28.0598453Z Entering 'third_party/NVTX' 2025-08-14T22:01:28.0633976Z http.https://github.com/.extraheader 2025-08-14T22:01:28.0669670Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T22:01:28.0703967Z http.https://github.com/.extraheader 2025-08-14T22:01:28.0744996Z Entering 'third_party/XNNPACK' 2025-08-14T22:01:28.0781467Z http.https://github.com/.extraheader 2025-08-14T22:01:28.0827679Z Entering 'third_party/aiter' 2025-08-14T22:01:28.0865793Z http.https://github.com/.extraheader 2025-08-14T22:01:28.0898736Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T22:01:28.0934172Z http.https://github.com/.extraheader 2025-08-14T22:01:28.0979410Z Entering 'third_party/benchmark' 2025-08-14T22:01:28.1010224Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1043946Z Entering 'third_party/composable_kernel' 2025-08-14T22:01:28.1081644Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1123200Z Entering 'third_party/cpp-httplib' 2025-08-14T22:01:28.1155634Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1190664Z Entering 'third_party/cpuinfo' 2025-08-14T22:01:28.1230307Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1265999Z Entering 'third_party/cudnn_frontend' 2025-08-14T22:01:28.1301593Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1346444Z Entering 'third_party/cutlass' 2025-08-14T22:01:28.1377717Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1421006Z Entering 'third_party/fbgemm' 2025-08-14T22:01:28.1460014Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1491789Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T22:01:28.1525379Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1564725Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T22:01:28.1600230Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1639230Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T22:01:28.1673240Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1705674Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T22:01:28.1747635Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1784959Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T22:01:28.1819763Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1853704Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T22:01:28.1889707Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1920673Z Entering 'third_party/fbgemm/external/json' 2025-08-14T22:01:28.1956963Z http.https://github.com/.extraheader 2025-08-14T22:01:28.1994674Z Entering 'third_party/flash-attention' 2025-08-14T22:01:28.2030150Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2061937Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T22:01:28.2100140Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2142415Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T22:01:28.2178652Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2216948Z Entering 'third_party/flatbuffers' 2025-08-14T22:01:28.2254129Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2289021Z Entering 'third_party/fmt' 2025-08-14T22:01:28.2326992Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2364908Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T22:01:28.2398824Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2433619Z Entering 'third_party/gloo' 2025-08-14T22:01:28.2469246Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2506472Z Entering 'third_party/googletest' 2025-08-14T22:01:28.2545954Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2584367Z Entering 'third_party/ideep' 2025-08-14T22:01:28.2627753Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2662875Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T22:01:28.2692236Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2731214Z Entering 'third_party/ittapi' 2025-08-14T22:01:28.2768986Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2807995Z Entering 'third_party/kineto' 2025-08-14T22:01:28.2844255Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2878290Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T22:01:28.2914254Z http.https://github.com/.extraheader 2025-08-14T22:01:28.2949960Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T22:01:28.2982537Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3025187Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T22:01:28.3054598Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3089694Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T22:01:28.3127837Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3164266Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T22:01:28.3199173Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3242893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T22:01:28.3275159Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3309235Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T22:01:28.3344296Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3382081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T22:01:28.3417175Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3455474Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T22:01:28.3493284Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3526547Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T22:01:28.3558371Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3597412Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T22:01:28.3631379Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3672186Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T22:01:28.3704302Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3746425Z Entering 'third_party/kleidiai' 2025-08-14T22:01:28.3777359Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3810071Z Entering 'third_party/mimalloc' 2025-08-14T22:01:28.3849065Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3880627Z Entering 'third_party/nlohmann' 2025-08-14T22:01:28.3920522Z http.https://github.com/.extraheader 2025-08-14T22:01:28.3965360Z Entering 'third_party/onnx' 2025-08-14T22:01:28.3995250Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4048356Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T22:01:28.4080365Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4122690Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T22:01:28.4153948Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4188623Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T22:01:28.4223751Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4269700Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T22:01:28.4307729Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4343892Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T22:01:28.4384944Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4419807Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T22:01:28.4447259Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4478877Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T22:01:28.4517367Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4554263Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T22:01:28.4590468Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4620065Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T22:01:28.4658024Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4692177Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T22:01:28.4731615Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4768078Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T22:01:28.4804288Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4850152Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T22:01:28.4883243Z http.https://github.com/.extraheader 2025-08-14T22:01:28.4938919Z Entering 'third_party/pocketfft' 2025-08-14T22:01:28.4976670Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5007449Z Entering 'third_party/protobuf' 2025-08-14T22:01:28.5047374Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5084467Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T22:01:28.5120358Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5153659Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T22:01:28.5191172Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5229472Z Entering 'third_party/psimd' 2025-08-14T22:01:28.5266600Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5298279Z Entering 'third_party/pthreadpool' 2025-08-14T22:01:28.5334073Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5376177Z Entering 'third_party/pybind11' 2025-08-14T22:01:28.5415008Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5455285Z Entering 'third_party/python-peachpy' 2025-08-14T22:01:28.5482853Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5517219Z Entering 'third_party/sleef' 2025-08-14T22:01:28.5554724Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5588367Z Entering 'third_party/tensorpipe' 2025-08-14T22:01:28.5629457Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5668446Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T22:01:28.5703765Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5740381Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T22:01:28.5776580Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5808409Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T22:01:28.5845097Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5880492Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T22:01:28.5914533Z http.https://github.com/.extraheader 2025-08-14T22:01:28.5950944Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T22:01:28.5986890Z http.https://github.com/.extraheader 2025-08-14T22:01:28.6125458Z A job completed hook has been configured by the self-hosted runner administrator 2025-08-14T22:01:28.6138547Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-08-14T22:01:28.6142014Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T22:01:28.6142361Z ##[endgroup] 2025-08-14T22:01:28.6231369Z [!ALERT!] Swap in detected! [!ALERT!] 2025-08-14T22:01:37.7120779Z [!ALERT!] Swap out detected [!ALERT!] 2025-08-14T22:01:53.0433270Z Cleaning up orphan processes